Develop Reference

Row by row streaming of data while waiting for response

I have a WCF service and client developed using C# which handles data transfer from SQL server database on the server to the SQL server database on the client end. I am facing some issues with the current architecture and planning to modify it to an idea I have, and would like to know if it is possible to achieve it, or how best can I modify the architecture to suite my needs.
The Server side database server is SQL 2008 R2 SP1 and client side servers are SQL 2000
Before I state the idea, below is the overview and current shortcomings of the architecture design I am using.
Overview:
Client requests for a table’s data.
WCF service queries the Server database for all pending data for the requested table. This data is loaded into a dataset.
WCF Compresses the Dataset using GZIP compression and converts it to byte for the client to download.
Client receives the Byte stream, un-compresses it and replicates the data from the Dataset to the physical table on the client database. This data is inserted row by row since in need the Primary key column filed to be returned to the server so that it can be flagged of as transferred.
Once the client has finished replicating the data, it uploads the successful rows Primary key fields back to the server, and in turn the server update each field one by one.
The above procedure uses a basic http binding, with streamed transfer mode.
Shortcomings:
This works great for little data, but when it comes to bulk data, maintaining the dataset in memory as the download is ongoing and also at the client side as replication is ongoing, is becoming impossible as sometimes the dataset size goes up to 4gb. The server can hold this much data since it’s a 32gb RAM server, but at the client side I get System out of memory exception since the client machine has 2gb RAM.
There are numerous deadlocks as the select query is running and also when updating since I am using transaction mode as read committed.
For bulk data it is very slow and completely hangs the client machine when the DTS is ongoing.
Idea in mind:
Maintain the same service and logic of row by row transfer since I cannot change this due to sensitivity of the data, but rather than downloading bulk data I plan to use the sample given in http://code.msdn.microsoft.com/Custom-WCF-Streaming-436861e6.
Thus the new flow will be as:
Upon receiving the download request, the server will open a connection to the DB using snapshot isolation as the transaction level.
Build the row by row object on the server and send it to the client on the requested channel, as the client receives each row object, it gets processed and a success or failure response is sent back to the server on the same method same channel, as I need to update the data on the same snapshot transaction.
This way I will reduce bulk objects in memory, and rely on SQL for the snapshot data that will be maintained in temdb once the transaction is initiated.
Challenge:
How can I send the row object and wait for a confirmation before sending the next one, as the update to the server row has to occur on the same snapshot transaction. Since if I create another method on the service to perform the flagging off, the snapshots will be different and this will cause issues in the integrity of the data in case the data undergoes changes after the snapshot transaction was initiated.
If this is the wrong approach, then please suggest a better one, as I am open to any suggestions.
If my understanding of the snapshot isolation is wrong, then please correct me as I am new to this.
Update 1:
I would like to achieve something like this when the client is the one requesting:
//Client Invokes this method on the server
public Stream GetData(string sTableName)
{
//Open the Snapshot Transaction on the Server
SqlDataReader rdr = operations.InitSnapshotTrans("Select * from " + sTableName + " Where Isnull(ColToCheck,'N') <> 'Y'");
//Check if there are rows available
if(rdr.HasRows)
{
while rdr.read()
{
SendObj sendobj = Logic.CreateObejct(rdr);
//Here is where i am stuck
//At this point I want to write the object to the Stream
...Write sendobj to Stream
//Once the client is done processing it reverts with a true for success or false for failuer.
if (returnObj == true)
{
operations.updateMovedRecord(rdr);
}
}
}
}
For the server sending i have written the code as Such (I used Pub Sub Model for this):
public void ServerData(string sServerText)
{
List<SubList> subscribers = Filter.GetClients();
if (subscribers == null) return;
Type type = typeof(ITransfer);
MethodInfo publishMethodInfo = type.GetMethod("ServerData");
foreach (SubList subscriber in subscribers)
{
try
{
//Open the Snapshot Transaction on the Server
SqlDataReader rdr = operations.InitSnapshotTrans("Select * from " + sTableName + " Where Isnull(ColToCheck,'N') <> 'Y'");
//Check if there are rows available
if(rdr.HasRows)
{
while rdr.read()
{
SendObj sendobj = Logic.CreateObejct(rdr);
bool rtnVal = Convert.ToBoolean(publishMethodInfo.Invoke(subscriber.CallBackId, new object[] { sendobj }));
if (rtnVal == true)
{
operations.updateMovedRecord(rdr);
}
}
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
}

Just off the top of my head, this sounds like it might take longer. That may or may not be a concern.
Given the requirement in challenge 1 (that everything happen in the context of one method call), it sounds like what actually needs to happen is for the server to call a method on the client, sending a record, and then waiting for the client to return confirmation. That way, everything that needs to happen, happens in the context of a single call (server to client). I don't know if that's feasible in your situation.
Another option might be to use some kind of double-queue system (perhaps with MSMQ?) so that the server and client can maintain an ongoing conversation within a single session.
I assume there's a reason why you can't just divide the data to be downloaded into manageable chunks and repeatedly execute the original process on the chunks. That sounds the least ambitious option, but you probably would have done it already if it met all your needs.

Better Technique: Reading Data in a Thread

I've got a routine called GetEmployeeList that loads when my Windows Application starts.
This routine pulls in basic employee information from our Active Directory server and retains this in a list called m_adEmpList.
We have a few Windows accounts set up as Public Profiles that most of our employees on our manufacturing floor use. This m_adEmpList gives our employees the ability to log in to select features using those Public Profiles.
Once all of the Active Directory data is loaded, I attempt to "auto logon" that employee based on the System.Environment.UserName if that person is logged in under their private profile. (employees love this, by the way)
If I do not thread GetEmployeeList, the Windows Form will appear unresponsive until the routine is complete.
The problem with GetEmployeeList is that we have had times when the Active Directory server was down, the network was down, or a particular computer was not able to connect over our network.
To get around these issues, I have included a ManualResetEvent m_mre with the THREADSEARCH_TIMELIMIT timeout so that the process does not go off forever. I cannot login someone using their Private Profile with System.Environment.UserName until I have the list of employees.
I realize I am not showing ALL of the code, but hopefully it is not necessary.
public static ADUserList GetEmployeeList()
{
if ((m_adEmpList == null) ||
(((m_adEmpList.Count < 10) || !m_gotData) &&
((m_thread == null) || !m_thread.IsAlive))
)
{
m_adEmpList = new ADUserList();
m_thread = new Thread(new ThreadStart(fillThread));
m_mre = new ManualResetEvent(false);
m_thread.IsBackground = true;
m_thread.Name = FILLTHREADNAME;
try {
m_thread.Start();
m_gotData = m_mre.WaitOne(THREADSEARCH_TIMELIMIT * 1000);
} catch (Exception err) {
Global.LogError(_CODEFILE + "GetEmployeeList", err);
} finally {
if ((m_thread != null) && (m_thread.IsAlive)) {
// m_thread.Abort();
m_thread = null;
}
}
}
return m_adEmpList;
}
I would like to just put a basic lock using something like m_adEmpList, but I'm not sure if it is a good idea to lock something that I need to populate, and the actual data population is going to happen in another thread using the routine fillThread.
If the ManualResetEvent's WaitOne timer fails to collect the data I need in the time allotted, there is probably a network issue, and m_mre does not have many records (if any). So, I would need to try to pull this information again the next time.
If anyone understands what I'm trying to explain, I'd like to see a better way of doing this.
It just seems too forced, right now. I keep thinking there is a better way to do it.

I think you're going about the multithreading part the wrong way. I can't really explain it, but threads should cooperate and not compete for resources, but that's exactly what's bothering you here a bit. Another problem is that your timeout is too long (so that it annoys users) and at the same time too short (if the AD server is a bit slow, but still there and serving). Your goal should be to let the thread run in the background and when it is finished, it updates the list. In the meantime, you present some fallbacks to the user and the notification that the user list is still being populated.
A few more notes on your code above:
You have a variable m_thread that is only used locally. Further, your code contains a redundant check whether that variable is null.
If you create a user list with defaults/fallbacks first and then update it through a function (make sure you are checking the InvokeRequired flag of the displaying control!) you won't need a lock. This means that the thread does not access the list stored as member but a separate list it has exclusive access to (not a member variable). The update function then replaces (!) this list, so now it is for exclusive use by the UI.
Lastly, if the AD server is really not there, try to forward the error from the background thread to the UI in some way, so that the user knows what's broken.
If you want, you can add an event to signal the thread to stop, but in most cases that won't even be necessary.

WPF, WCF and Code First - Disconnected mode

Some architecture dilemma:
I'm using WPF as my client-side, EF Code First as my Data Access Layer, and WCF to connect between those. My probelm is hou to reupdate the UI after I did some changes to the DB, for example:
User insert new "Person" on the UI (ID=0)
User save the "Person" to the DB (ID=10, for example)
When talking about one user it's very simple - I can return the ID and update my UI as well (so next change to this person will be considered as "Update"), but what about adding more than one user at once, or updating other properties that was calculated on the server? should I return the whole graph? not to mention is very hard to remap it on the client side.
Before CodeFirst we could use STE, but it has it's own problems. anyone knows about known CodeFirst approach?
Would be happy to hear your voice.
Thanks!

You can send as the request to your wcf service the dateTime of your last update in client-side. But in the server-side you take all Persons which was updated/added after that dateTime and return it as the result. In this way you will get only modified/added Persons from your server-side.
So add lastUpdate collumn to your entity Person.
EDIT 1
If you want to server update the information in client but not client ask for news from server.
You can use the way like it works in Web Programming.
(1)The client-side asks server-side - "hey, my last update was at 20:00 10.02.2013", then server looks into DB - "is news after 20:00 10.02.2013?" if yes:
a) returns the news to the client
if no news in DB:
b) He dont returns null, but he does Thread.Sleep(somevalue). He sleeps then repeats the query to db and asks is "there news in db". So it's all repeats untill the news in DB will apear. After news in db appears he return the List<data> which is in updated after the dateTime. After that client gets the data he goes back to the point - (1).
So you dont make a lot of requests to the server but making only one request and wait for the news from the server.
Notice 2 things:
1) If the client waits too long the server side will throw the exception(dont remember actually the error code but it's not important now), so you have to catch this exception on client-side and make a new request to server-side. Also you have to configure on the server-side as long as you can wait time, to minimize the amount of requests from client.
2) You have to run this data-updater in the new thread not in the main, where the application runs.
How it will looks from the code(it may not work, i just want to show you the logic):
Server side:
public List<SomeData> Updater(DateTime clientSideLastUpdate)
{
List<SomeData> news = new List<SomeData>();
while(true)
{
List<SomeData> news = dbContext.SomeData.Where(e=>e.UpdateDateTime > clientSideLastUpdate).ToList();
if(news.Count()>0)
{
return news;
}
}
}
Client-side:
public static void Updater()
{
try
{
var news = someServiceReference.Updater(DateTime clientSideLastUpdate);
RenewDataInForms(news);
Updater();
}
catch(ServerDiesOrWhatElseExcepption)
{
Updater()
}
}
And somewhere in the code you run this updater in the new thread
Thread updaterThread = new Thread(Updater());
updaterThread.Start();
Edit 2
if you want update by one request all entities but not only SomeData then you have to add Dto object which will contain the List of every entities you want to be updatable. The server-side will complete and return this Dto object.
Hope it helps.

DataGridView live display of datatable using virtual mode

I have a DataGridView that will display records (log entries) from a database. The amount of records that can exist at a time is very large. I would like to use the virtual mode feature of the DataGridView to display a page of data, and to minimize the amount of data that has to be transferred across a network at a given time.
Polling for data is out of the question. There will be several clients running at a time, all of which are on the same network and viewing the records. If they all poll for data, the network will run very slowly.
The data is read-only to the user; they won't be able to edit any of it, just view it. I need to know when updates occur in the database, and I need to update the screen with those updates accordingly using virtual mode. If a page of data a user is viewing contains data that has change, he/she will see those updates on that page. If updates were made to data in the database, but not in the data the user is viewing, then not much really changes on the user screen (Maybe just the scroll bar if records were added or removed).
My current approach is using SQL server change tracking with the sync framework. Each client has a local SQL Server CE instance and database file that is kept in sync with the main database server. I use the information from the synchronization event to see if any changes were made to the main database and were sync'ed to the client. I need to use the DataGridView virtual mode here because I can't have thousands of records loaded into the DataGridView at once, otherwise memory usage goes through the roof.
The main challenge right now is knowing how to use virtual mode to provide a seamless experience to the user by allowing them to scroll up and down through the records, and also have records update on the fly without interfering with the user inappropriately. Has anybody dealt with this issue before, and if so, where I can see how they did it? I've gone through some of the MSDN documentation and examples on virtual mode. So far, I haven't found documentation and/or examples on their site that explains how to do what I am trying to accomplish.

Add following to form startup
dataGridView1.CellValueNeeded +=new DataGridViewCellValueEventHandler( dataGridView1_CellValueNeeded );
dataGridView1.VirtualMode = true;
Use the following code where u receive the update
dataGridView1.RowCount = (int)rowscount.TotalCount;
Add following function :
private void dataGridView1_CellValueNeeded( object sender, DataGridViewCellValueEventArgs e )
{
_cache.LoadPage( e.RowIndex );
int rowIndex = e.RowIndex % PageSize;
e.Value = datatable.rows[rowIndex][e.ColumnIndex];
}

what is the right way to spawn thread for database IO in asmx web service?

I have a short lock guarded section in a method (that serves the request entirely) that makes all initializations (etc. log-related). So only 1 thread can be there at time. In this section I also load system data from database if not loaded. This is naturally executed only on 1st request and it does not matter it takes time and no threads can propagate since it's done only once (by dummy request).
static public void LoadAllSystemData()
{
SystemData newData = new SystemData(); //own type (etc. Hashtables in Hashtables).
LoadTables(ref newData);
LoadClasses(ref newData);
LoadAllSysDescrs(ref newData);
LoadFatFields(ref newData);
LoadAllFields(ref newData);
_allData = newData;
}
After the lock-guarded section the system data is accessed from concurrent threads only by reading and no locks are needed:
static public Hashtable GetTables()
{
return _allData.Tables;
}
Now the lock guarded section must have method that checks if system data is older than 24h and refresh it. If it done just by calling method (from lock guarded section) below that thread takes a long time and no other thread can enter the lock guarded section.
static public void CheckStatus()
{
DateTime timeStamp = DateTime.Now;
TimeSpan span = timeStamp.Subtract(_cacheTimeStamp);
if (span.Hours >= 24)
{
LoadAllSystemData();
_cacheTimeStamp = DateTime.Now;
}
}
My questions are:
How to spawn a non-threadpool thread best way to handle IO so the threadpool worker thread can propagate and all the threads spend minimum time in lock guarded section?
Is the _allData = newData; in LoadAllSystemData atomic? If it is, it feels the best way to implement that so GetXxx-methods like GetTables do not need any locking!
Is there any way to get LoadAllSystemData to be called before requests? For example on iisreset?
Thanks in advance for your answers!

Matti, you're asking multiple questions that point to the best structure for your application. I would summarize your questions as:
How do I pre-load data needed by my service prior to handling any legitimate requests?
How do I ensure the pre-loaded data is loaded "once"?
How do I refresh the pre-loaded data on a schedule, i.e. every 24 hours?
Understanding the Asp.Net pipeline and event structure will help you understand the answers to these questions.
First, the Asp.Net pipeline provides a one-time execution region in Application_Start. This is fired once per application cycle. This would also fire in 'iisreset', which cycles every application for a given server. However, application cycles themselves will recycle on their own, based on their configuration. All of which are controlled through IIS settings.
Second, Asp.Net is a request system; it won't fire refresh events for you, nor can you use it by itself as a scheduler. You need an outside agent to act on that for you.
Here's what you could do:
Add your pre-loaded data routine to Application_Start.
Configure your web site's Application cycle settings for every 24 hours.
This would ensure your data is loaded once, via Application_Start. It would ensure your data is loaded prior to your site serving any requests. Last, it would ensure your data is refreshed every 24 hours.
Here's what I would do:
Add pre-loaded data routine to Application_Start.
Modify pre-loaded data routine to use Asp.Net Cache, instead of static usage.
Modify lookup data to retrieve data from cache, and re-load if data is not found.
Provide capability for diagnostic/monitoring system, i.e. nagios, to refresh data asynchronously via web method call, i.e. "preload_refresh()".
This would provide essentially the same forward effect as the above solution, but with better reliability for your service.
As always, your mileage may vary. Hope this helps.

To answer part of your question anyway, you can use the "Application_Start" method in the Global.asax to execute the statement on the start of the application if you want to pre-fill the data as soon as the application comes online.