I have a WCF service and client developed using C# which handles data transfer from SQL server database on the server to the SQL server database on the client end. I am facing some issues with the current architecture and planning to modify it to an idea I have, and would like to know if it is possible to achieve it, or how best can I modify the architecture to suite my needs.
The Server side database server is SQL 2008 R2 SP1 and client side servers are SQL 2000
Before I state the idea, below is the overview and current shortcomings of the architecture design I am using.
Overview:
Client requests for a table’s data.
WCF service queries the Server database for all pending data for the requested table. This data is loaded into a dataset.
WCF Compresses the Dataset using GZIP compression and converts it to byte for the client to download.
Client receives the Byte stream, un-compresses it and replicates the data from the Dataset to the physical table on the client database. This data is inserted row by row since in need the Primary key column filed to be returned to the server so that it can be flagged of as transferred.
Once the client has finished replicating the data, it uploads the successful rows Primary key fields back to the server, and in turn the server update each field one by one.
The above procedure uses a basic http binding, with streamed transfer mode.
Shortcomings:
This works great for little data, but when it comes to bulk data, maintaining the dataset in memory as the download is ongoing and also at the client side as replication is ongoing, is becoming impossible as sometimes the dataset size goes up to 4gb. The server can hold this much data since it’s a 32gb RAM server, but at the client side I get System out of memory exception since the client machine has 2gb RAM.
There are numerous deadlocks as the select query is running and also when updating since I am using transaction mode as read committed.
For bulk data it is very slow and completely hangs the client machine when the DTS is ongoing.
Idea in mind:
Maintain the same service and logic of row by row transfer since I cannot change this due to sensitivity of the data, but rather than downloading bulk data I plan to use the sample given in http://code.msdn.microsoft.com/Custom-WCF-Streaming-436861e6.
Thus the new flow will be as:
Upon receiving the download request, the server will open a connection to the DB using snapshot isolation as the transaction level.
Build the row by row object on the server and send it to the client on the requested channel, as the client receives each row object, it gets processed and a success or failure response is sent back to the server on the same method same channel, as I need to update the data on the same snapshot transaction.
This way I will reduce bulk objects in memory, and rely on SQL for the snapshot data that will be maintained in temdb once the transaction is initiated.
Challenge:
How can I send the row object and wait for a confirmation before sending the next one, as the update to the server row has to occur on the same snapshot transaction. Since if I create another method on the service to perform the flagging off, the snapshots will be different and this will cause issues in the integrity of the data in case the data undergoes changes after the snapshot transaction was initiated.
If this is the wrong approach, then please suggest a better one, as I am open to any suggestions.
If my understanding of the snapshot isolation is wrong, then please correct me as I am new to this.
Update 1:
I would like to achieve something like this when the client is the one requesting:
//Client Invokes this method on the server
public Stream GetData(string sTableName)
{
//Open the Snapshot Transaction on the Server
SqlDataReader rdr = operations.InitSnapshotTrans("Select * from " + sTableName + " Where Isnull(ColToCheck,'N') <> 'Y'");
//Check if there are rows available
if(rdr.HasRows)
{
while rdr.read()
{
SendObj sendobj = Logic.CreateObejct(rdr);
//Here is where i am stuck
//At this point I want to write the object to the Stream
...Write sendobj to Stream
//Once the client is done processing it reverts with a true for success or false for failuer.
if (returnObj == true)
{
operations.updateMovedRecord(rdr);
}
}
}
}
For the server sending i have written the code as Such (I used Pub Sub Model for this):
public void ServerData(string sServerText)
{
List<SubList> subscribers = Filter.GetClients();
if (subscribers == null) return;
Type type = typeof(ITransfer);
MethodInfo publishMethodInfo = type.GetMethod("ServerData");
foreach (SubList subscriber in subscribers)
{
try
{
//Open the Snapshot Transaction on the Server
SqlDataReader rdr = operations.InitSnapshotTrans("Select * from " + sTableName + " Where Isnull(ColToCheck,'N') <> 'Y'");
//Check if there are rows available
if(rdr.HasRows)
{
while rdr.read()
{
SendObj sendobj = Logic.CreateObejct(rdr);
bool rtnVal = Convert.ToBoolean(publishMethodInfo.Invoke(subscriber.CallBackId, new object[] { sendobj }));
if (rtnVal == true)
{
operations.updateMovedRecord(rdr);
}
}
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
}
Just off the top of my head, this sounds like it might take longer. That may or may not be a concern.
Given the requirement in challenge 1 (that everything happen in the context of one method call), it sounds like what actually needs to happen is for the server to call a method on the client, sending a record, and then waiting for the client to return confirmation. That way, everything that needs to happen, happens in the context of a single call (server to client). I don't know if that's feasible in your situation.
Another option might be to use some kind of double-queue system (perhaps with MSMQ?) so that the server and client can maintain an ongoing conversation within a single session.
I assume there's a reason why you can't just divide the data to be downloaded into manageable chunks and repeatedly execute the original process on the chunks. That sounds the least ambitious option, but you probably would have done it already if it met all your needs.
Related
I need to do some Data Synchronization between a client and a server. I have decided to use RabbitMQ for Data Synchronization, in order for my program to be tolerant for network failure. I know how to use RabbitMQ, so no problem there.
All data needs to be stored in a local database, and then copied to an other database.
My problem is exactly which data to transfer over the network.
Let me show some example code:
This is the way I write to my database today. The body of my method is removed since it is irrelevant for the question.
public static class DatabaseHelper
{
public static void RegisterCheckout(string checkoutStationId, string employeeId, string destinationId)
{
//Insert in to database
}
}
So in short my program calls DatabaseHelper.RegisterCheckout("123", "321", "456"), the checkout is put into the local database.
But how do I serialize this method call, in order for me to reproduce it server side?
I am currently working on a system that makes calls to an external service and caches some of the data in the HttpContext.Current.Items collection for performance. The data can change quite regularly and it is user sensitive which is why we are currently storing it only for the duration of the current HttpRequest.
Example:
if (HttpContext.Current.Items[cacheKey] != null)
{
LogHelper.Debug<ExampleService>("[- CACHED RESULT -] GetUser({0})", () => email);
return (ExampleUser)HttpContext.Current.Items[cacheKey];
}
using (var client = new UserServiceClient())
{
using (new OperationContextScope(client.InnerChannel))
{
LogHelper.Debug<ExampleService>("GetUser({0})", () => email);
exampleUser = svc.GetUser(email);
HttpContext.Current.Items.Add(cacheKey, exampleUser);
}
}
In my local environment this behaves as expected and mostly also does in staging where the same thread is used for the duration of the request however in production this is not the case and there are still multiple calls to the external service in the same request. This can be seen from the logs which show that the value in HttpContext.Current.Items[cacheKey] is not returned in cases where the Thread ID does not match the original request.
This I guess means that my current understanding of HttpContext.Current.Items is wrong and that this is not a suitable solution for my needs.
My question therefore is can this be made to work across threads in the same request and if so should it, otherwise what suitable alternative is there?
One option is to use Session to store your data. Unfortunately it's not applicable for API-specific requests (e.g mobile device makes a call to server API). Besides, server session state requires all of your data serializable (DB session state doesn't).
If session does not satisfy your requirements, then you should go to next option: Using cache protected by something that represents your requests coming from the same user (a.k.a access token).
Some architecture dilemma:
I'm using WPF as my client-side, EF Code First as my Data Access Layer, and WCF to connect between those. My probelm is hou to reupdate the UI after I did some changes to the DB, for example:
User insert new "Person" on the UI (ID=0)
User save the "Person" to the DB (ID=10, for example)
When talking about one user it's very simple - I can return the ID and update my UI as well (so next change to this person will be considered as "Update"), but what about adding more than one user at once, or updating other properties that was calculated on the server? should I return the whole graph? not to mention is very hard to remap it on the client side.
Before CodeFirst we could use STE, but it has it's own problems. anyone knows about known CodeFirst approach?
Would be happy to hear your voice.
Thanks!
You can send as the request to your wcf service the dateTime of your last update in client-side. But in the server-side you take all Persons which was updated/added after that dateTime and return it as the result. In this way you will get only modified/added Persons from your server-side.
So add lastUpdate collumn to your entity Person.
EDIT 1
If you want to server update the information in client but not client ask for news from server.
You can use the way like it works in Web Programming.
(1)The client-side asks server-side - "hey, my last update was at 20:00 10.02.2013", then server looks into DB - "is news after 20:00 10.02.2013?" if yes:
a) returns the news to the client
if no news in DB:
b) He dont returns null, but he does Thread.Sleep(somevalue). He sleeps then repeats the query to db and asks is "there news in db". So it's all repeats untill the news in DB will apear. After news in db appears he return the List<data> which is in updated after the dateTime. After that client gets the data he goes back to the point - (1).
So you dont make a lot of requests to the server but making only one request and wait for the news from the server.
Notice 2 things:
1) If the client waits too long the server side will throw the exception(dont remember actually the error code but it's not important now), so you have to catch this exception on client-side and make a new request to server-side. Also you have to configure on the server-side as long as you can wait time, to minimize the amount of requests from client.
2) You have to run this data-updater in the new thread not in the main, where the application runs.
How it will looks from the code(it may not work, i just want to show you the logic):
Server side:
public List<SomeData> Updater(DateTime clientSideLastUpdate)
{
List<SomeData> news = new List<SomeData>();
while(true)
{
List<SomeData> news = dbContext.SomeData.Where(e=>e.UpdateDateTime > clientSideLastUpdate).ToList();
if(news.Count()>0)
{
return news;
}
}
}
Client-side:
public static void Updater()
{
try
{
var news = someServiceReference.Updater(DateTime clientSideLastUpdate);
RenewDataInForms(news);
Updater();
}
catch(ServerDiesOrWhatElseExcepption)
{
Updater()
}
}
And somewhere in the code you run this updater in the new thread
Thread updaterThread = new Thread(Updater());
updaterThread.Start();
Edit 2
if you want update by one request all entities but not only SomeData then you have to add Dto object which will contain the List of every entities you want to be updatable. The server-side will complete and return this Dto object.
Hope it helps.
I am building a web application that will pull event logs from multiple servers and display them into a page that I have set up. I have set it to go back 20 events for both the Application log and the System log. However, I am trying to decide if I want to save the data to a SQL database and display it from there or directly from the server into a list box. I was leaning towards directly to the list box just because the data changes so frequently. Does anyone have any suggestions or benefits from doing this any other way?
Just in case anyone is curious, here is the code I am using:
string LogType = "Application";
string serverIP = "192.168.1.5";
EventLog eventLog = new EventLog(LogType, serverIP);
int LastLog = eventLog.Entries.Count;
int i;
for (i = eventLog.Entries.Count - 1; i >= LastLog - 20; i--) {
EventLogEntry CurrentEntry = eventLog.Entries[i];
}
I actually don't see any benefits to save the logs to the database. If you're planning to serve a lot of requests from your application, you should consider caching them in your server (and I would try to do it in memory, and not in DB), and refreshing this cache from time to time. Even if your server handles few requests - it could be useful to cache the events, just for not hitting the target servers too much.
Since this data (events) seem to me a volatile in nature, and can be easily regenerated (by querying the target machines) at any time, and changes frequently - there is no need to persist it.
I have an ASP.NET-MVC application that:
opens a db transaction
updates a cart status and other things
submits this cart to another web server via an HttpRequest
register in database the transmission with its code status
send a confirmation mail, that the command has been sent
then if no error has occurs commit the transaction else rollback it.
Normally, after that the distant server sends another web request to my application to a controller action that will update the previous transmission and set an aknowledge field.
My problem is that the distant web server sometimes is very fast and sends the aknowledge status before the transmission insertion in the database is committed, so the update fails. How could I prevent this?
Thanks.
Just make your commit operation in two stages. First is too create record. Then do processing like create mail and so on. And second to make real(logical) commit.
using(var db = new Db(){
db.Insert(
} // This will commit first stage
// Send email do other stuff
using(var db = new Db(){
var t = db.getTransmission()
r.Commited = true;
db.Save();
} // This will logically commit
Can you update the database, commit and mark as inactive or invalid, then take away this mark once you get our acknowledgement status?
I may be misunderstanding what exacty it is you're doing.