I have a real-time application built in c#.NET and sql as DB.
The application has three modules:
Grabbing under-vehicle images(UVSS) using a camera,
Detection of license plate number (ALPR)
Recording Driver info video(DIC).
We first receive a trigger from ADM 6060 which then activates the cameras(via code)and then these three modules happen simultaneously using Threads(and then stored into db). Lastly we display these on WinForm.
I am currently using a two threads:
Thread 1 for monitoring trigger value received by ADM
Thread 2 for grabbing live Under-vehicle images
Further, on receiving appropriate trigger value from ADM6060,
I create Two tasks - for ALPR detection and DIC video.
I cannot post the entire code due to comapny's policy but here's a snippet:
I have 2 threads in Form1:
ThreadStart tsmain = new ThreadStart(this.ServiceCaptureImage);
thread = new Thread(tsmain);
ThreadStart tsmain1 = new ThreadStart(this.Draw_Bitmap);
continuous = new Thread(tsmain1);
thread.start() and continuous.start() are executed in form_Load.
private void ServiceCaptureImage()
{
try{
while(true)
{
if(bdata[0]==false)
{
var task1 = Task.Factory.StartNew(() => ALPRdetect());
var task2 = Task.Factory.StartNew(() => DICcapture());
Task.WhenAll(task1, task2);
}
}
}
catch (Exception ex){
MessageBox.show(ex.message)
}
}
I manually counted the number of cars that were detected by the system.
In a total of 70 cars passed: 53 were totally detected(ALPR, under-vehicle and DIC), 4 cars went undetected, 9 cars ALPR was missing and in the remaining under-vehicle were missing.
The efficiency achieved so far is 84%, in some cases the application only performs ALPR but not UVSS or vice-versa.
What could be the possible reason?
Also, should i be using ThreadPool in this case?
Related
I am writing a script in C# for Unity to read messages from multiple sources. I have a function ReadMessage which takes in a port string and returns a string ID number. The issue is, I have a dozen different connections I need to read from and the messages have a 10ms timeout timer before it stops trying and lets the code continue. This causes a reduction in frame rate when I have a dozen threads waiting a few ms for the previous one to finish Joining. My thread code is as follows:
string threadOneString= null;
Thread ThreadOne;
//Repeat for 11 more threads
void Update () {
ThreadOne = new Thread(
() =>
{
threadOneString = ReadMessage(someClass.port);
});
ThreadOne.Start();
//Repeat for 11 more threads
}
void LateUpdate () {
ThreadOne.Join();
//Repeat for 11 more threads
UpdateClass(threadOneString);
//Repeat for 11 more threads
UpdateTextDisplay(); //Just updates a Unity Text object
}
And my ReadMessage code in case it matters.
private string ReadMessage(string port) //Change this name to ReadButtonPress
{
string fullMessage = "";
someStruct parsedMessage;
var timeout = new System.TimeSpan(0, 0, 0, 0, 10); // Less than 10 and it starts to miss most messages, ideally this would be a bit higher
string connectionString = PROTOCOL + CONTROLLER_IP + ":" + port;
AsyncIO.ForceDotNet.Force();
using (var subSocket = new SubscriberSocket())
{
subSocket.Connect(connectionString);
subSocket.Subscribe("");
subSocket.TryReceiveFrameString(timeout, out fullMessage);
UnityEngine.Debug.Log("Message: " + fullMessage);
subSocket.Close();
}
// Some message parsing and checking...
return parsedMessage.someString;
}
What I'd like to do, but I don't know if it's possible, is to call the Joins for each thread at the same time instead of calling one, waiting, then calling the next. The threads don't interact with each other, so I'm hoping this is possible. If not, I'd greatly appreciate another solution or suggestion.
EDIT: Clarification on what's happening. When I only run a few threads I get a FPS of 65-70. When I run all 12 threads my FPS drops to ~50 and I have a hard requirement of 60 FPS.
Thanks to some feedback here I am no longer creating a socket every update. Instead I create the sockets in Start() and just read what I need from a public function in the class. I based my code on https://stackoverflow.com/a/14797475/8635796
Don't do what I did and recreate the exact same threads and sockets every Update(). In hindsight, it was a really poor choice.
Use tasks instead, and put them into a list.
var myTasks = new List<Task>();
Task allUpdateTasks;
...
void Update () {
myTasks.Add(
Task.Factory.StartNew(() =>
{
threadOneString = ReadMessage(someClass.port);
}));
//Repeat for 11 more tasks
// Create a single tasks that will hold all tasks above.
allUpdateTasks = Task.WhenAll(myTasks);
}
void LateUpdate() {
allUpdateTasks.Wait();
...
}
define ManualResetEvent and signal it when you got results, you can wait up to 64 waithandles using WaitHandle.WaitAny
I have around 10 000 000 tasks that each takes from 1-10 seconds to complete. I am running those tasks on a powerful server, using 50 different threads, where each thread picks the first not-done task, runs it, and repeats.
Pseudo-code:
for i = 0 to 50:
run a new thread:
while True:
task = first available task
if no available tasks: exit thread
run task
Using this code, I can run all the tasks in parallell on any given number of threads.
In reality, the code uses C#'s Task.WhenAll, and looks like this:
ServicePointManager.DefaultConnectionLimit = threadCount; //Allow more HTTP request simultaneously
var currentIndex = -1;
var threads = new List<Task>(); //List of threads
for (int i = 0; i < threadCount; i++) //Generate the threads
{
var wc = CreateWebClient();
threads.Add(Task.Run(() =>
{
while (true) //Each thread should loop, picking the first available task, and executing it.
{
var index = Interlocked.Increment(ref currentIndex);
if (index >= tasks.Count) break;
var task = tasks[index];
RunTask(conn, wc, task, port);
}
}));
}
await Task.WhenAll(threads);
This works just as I wanted it to, but I have a problem: since this code takes a lot of time to run, I want the user to see some progress. The progress is displayed in a colored bitmap (representing a matrix), and also takes some time to generate (a few seconds).
Therefore, I want to generate this visualization on a background thread. But this other background thread is never executed. My suspicion is that it is using the same thread pool as the parallel code, and is therefore enqueued, and will not be executed before the parallel code is actually finished. (And that's a bit too late.)
Here's an example of how I generate the progress visualization:
private async void Refresh_Button_Clicked(object sender, RoutedEventArgs e)
{
var bitmap = await Task.Run(() => // <<< This task is never executed!
{
//bla, bla, various database calls, and generating a relatively large bitmap
});
//Convert the bitmap into a WPF image, and update the GUI
VisualizationImage = BitmapToImageSource(bitmap);
}
So, how could I best solve this problem? I could create a list of Tasks, where each Task represents one of my tasks, and run them with Parallel.Invoke, and pick another Thread pool (I think). But then I have to generate 10 million Task objects, instead of just 50 Task objects, running through my array of stuff to do. That sounds like it uses much more RAM than necessary. Any clever solutions to this?
EDIT:
As Panagiotis Kanavos suggested in one of his comments, I tried replacing some of my loop logic with ActionBlock, like this:
// Create an ActionBlock<int> that performs some work.
var workerBlock = new ActionBlock<ZoneTask>(
t =>
{
var wc = CreateWebClient(); //This probably generates some unnecessary overhead, but that's a problem I can solve later.
RunTask(conn, wc, t, port);
},
// Specify a maximum degree of parallelism.
new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = threadCount
});
foreach (var t in tasks) //Note: the objects in the tasks array are not Task objects
workerBlock.Post(t);
workerBlock.Complete();
await workerBlock.Completion;
Note: RunTask just executes a web request using the WebClient, and parses the results. It's nothing in there that can create a dead lock.
This seems to work as the old parallelism code, except that it needs a minute or two to do the initial foreach loop to post the tasks. Is this delay really worth it?
Nevertheless, my progress task still seems to be blocked. Ignoring the Progress< T > suggestion for now, since this reduced code still suffers the same problem:
private async void Refresh_Button_Clicked(object sender, RoutedEventArgs e)
{
Debug.WriteLine("This happens");
var bitmap = await Task.Run(() =>
{
Debug.WriteLine("This does not!");
//Still doing some work here, so it's not optimized away.
};
VisualizationImage = BitmapToImageSource(bitmap);
}
So it still looks like new tasks are not executed as long as the parallell task is running. I even reduced the "MaxDegreeOfParallelism" from 50 to 5 (on a 24 core server) to see if Peter Ritchie's suggestion was right, but no change. Any other suggestions?
ANOTHER EDIT:
The issue seems to have been that I overloaded the thread pool with all my simultaneous blocking I/O calls. I replaced WebClient with HttpClient and its async-functions, and now everything seems to be working nicely.
Thanks to everyone for the great suggestions! Even though not all of them directly solved the problem, I'm sure they all improved my code. :)
.NET already provides a mechanism to report progress with the IProgress< T> and the Progress< T> implementation.
The IProgress interface allows clients to publish messages with the Report(T) class without having to worry about threading. The implementation ensures that the messages are processed in the appropriate thread, eg the UI thread. By using the simple IProgress< T> interface the background methods are decoupled from whoever processes the messages.
You can find more information in the Async in 4.5: Enabling Progress and Cancellation in Async APIs article. The cancellation and progress APIs aren't specific to the TPL. They can be used to simplify cancellation and reporting even for raw threads.
Progress< T> processes messages on the thread on which it was created. This can be done either by passing a processing delegate when the class is instantiated, or by subscribing to an event. Copying from the article:
private async void Start_Button_Click(object sender, RoutedEventArgs e)
{
//construct Progress<T>, passing ReportProgress as the Action<T>
var progressIndicator = new Progress<int>(ReportProgress);
//call async method
int uploads=await UploadPicturesAsync(GenerateTestImages(), progressIndicator);
}
where ReportProgress is a method that accepts a parameter of int. It could also accept a complex class that reported work done, messages etc.
The asynchronous method only has to use IProgress.Report, eg:
async Task<int> UploadPicturesAsync(List<Image> imageList, IProgress<int> progress)
{
int totalCount = imageList.Count;
int processCount = await Task.Run<int>(() =>
{
int tempCount = 0;
foreach (var image in imageList)
{
//await the processing and uploading logic here
int processed = await UploadAndProcessAsync(image);
if (progress != null)
{
progress.Report((tempCount * 100 / totalCount));
}
tempCount++;
}
return tempCount;
});
return processCount;
}
This decouples the background method from whoever receives and processes the progress messages.
I have a situation in which I have a producer/consumer scenario. The producer never stops, which means that even if there is a time where there are no items in the BC, further items can be added later.
Moving from .NET Framework 3.5 to 4.0, I decided to use a BlockingCollection as a concurrent queue between the consumer and the producer. I even added some parallel extensions so I could use the BC with a Parallel.ForEach.
The problem is that, in the consumer thread, I need to have a kind of an hybrid model:
Im always checking the BC to process any item that arrived with a
Parallel.ForEach(bc.GetConsumingEnumerable(), item => etc
Inside this foreach, I execute all the tasks that dont depend between each other.
Here comes the problem. After paralelizing the previous tasks I need to manage their results in the same FIFO order in which they were in the BC. The processing of these results should be made in a sync thread.
A little example in pseudo code follows:
producer:
//This event is triggered each time a page is scanned. Any batch of new pages can be added at any time at the scanner
private void Current_OnPageScanned(object sender, ScannedPage scannedPage)
{
//The object to add has a property with the sequence number
_concurrentCollection.TryAdd(scannedPage);
}
consumer:
private void Init()
{
_cancelTasks = false;
_checkTask = Task.Factory.StartNew(() =>
{
while (!_cancelTasks)
{
//BlockingCollections with Parallel ForEach
var bc = _concurrentCollection;
Parallel.ForEach(bc.GetConsumingEnumerable(), item =>
{
ScannedPage currentPage = item;
// process a batch of images from the bc and check if an image has a valid barcode. T
});
//Here should go the code that takes the results from each tasks, process them in the same FIFO order in which they entered the BC and save each image to a file, all of this in this same thread.
}
});
}
Obviously, this cant work as it is because the .GetConsumingEnumerable() blocks until there is another item in the BC. I asume I could do it with tasks and just fire 4 or 5 task in a same batch, but:
How could I do this with tasks and still have a waiting point before the start of the tasks that blocks until there is an item to be consumed in the BC (I don't want to start processing if there is nothing. Once there is something in the BC i would just start the batch of 4 tasks, and use a TryTake inside each one so if there is nothing to take they don't block, because I don't know if I can always reach the number of items from the BC as the batch of tasks, for example, just one item left in the BC and a batch of 4 tasks) ?
How could I do this and take advantage of the efficiency that Parallel.For offers?
How could I save the results of the tasks in the same FIFO order in which the items were extracted from the BC?
Is there any other concurrency class more suited to this kind of hybrid processing of items in the consumer?
Also, this is my first question ever made in StackOverflow, so if you need any more data or you just think that my question is not correct just let me know.
I think I follow what you're asking, why not create a ConcurrentBag and add to it while processing like this:
while (!_cancelTasks)
{
//BlockingCollections with Paralell ForEach
var bc = _concurrentCollection;
var q = new ConcurrentBag<ScannedPage>();
Parallel.ForEach(bc.GetConsumingEnumerable(), item =>
{
ScannedPage currentPage = item;
q.Add(item);
// process a batch of images from the bc and check if an image has a valid barcode. T
});
//Here should go the code that takes the results from each tasks, process them in the same FIFO order in which they entered the BC and save each image to a file, all of this in this same thread.
//process items in your list here by sorting using some sequence key
var items = q.OrderBy( o=> o.SeqNbr).ToList();
foreach( var item in items){
...
}
}
This obviously doesn't enqueue them in the exact order they were added to the BC but you could add some sequence nbr to the ScannedPage object like Alex suggested and then sort the results after.
Here's how I'd handle the sequence:
Add this to the ScannedPage class:
public static int _counter; //public because this is just an example but it would work.
Get a sequence nbr and assign here:
private void Current_OnPageScanned(object sender, ScannedPage scannedPage)
{
lock( this){ //to single thread this process.. not necessary if it's already single threaded of course.
System.Threading.Interlocked.Increment( ref ScannedPage._counter);
scannedPage.SeqNbr = ScannedPage._counter;
...
}
}
Whenever you need the results of a parallel operation, using PLINQ is generally more convenient that using the Parallel class. Here is how you could refactor your code using PLINQ:
private void Init()
{
_cancelTasks = new CancellationTokenSource();
_checkTask = Task.Run(() =>
{
while (true)
{
_cancelTasks.Token.ThrowIfCancellationRequested();
var bc = _concurrentCollection;
var partitioner = Partitioner.Create(
bc.GetConsumingEnumerable(_cancelTasks.Token),
EnumerablePartitionerOptions.NoBuffering);
ScannedPage[] results = partitioner
.AsParallel()
.AsOrdered()
.Select(scannedPage =>
{
// Process the scannedPage
return scannedPage;
})
.ToArray();
// Process the results
}
});
}
The .AsOrdered() is what ensures that you'll get the results in the same order as the input.
Be aware that when you consume a BlockingCollection<T> with the Parallel class or PLINQ, it is important to use the Partitioner and the EnumerablePartitionerOptions.NoBuffering configuration, otherwise there is a risk of deadlocks. The default greedy behavior of the Parallel/PLINQ and the blocking behavior of the BlockingCollection<T>, do not interact well.
I think I may need to re-think my design. I'm having a hard time narrowing down a bug that is causing my computer to completely hang, sometimes throwing an HRESULT 0x8007000E from VS 2010.
I have a console application (that I will later convert to a service) that handles transferring files based on a database queue.
I am throttling the threads allowed to transfer. This is because some systems we are connecting to can only contain a certain number of connections from certain accounts.
For example, System A can only accept 3 simultaneous connections (which means 3 separate threads). Each one of these threads has their own unique connection object, so we shouldn't run in to any synchronization problems since they aren't sharing a connection.
We want to process the files from those systems in cycles. So, for example, we will allow 3 connections that can transfer up to 100 files per connection. This means, to move 1000 files from System A, we can only process 300 files per cycle, since 3 threads are allowed with 100 files each. Therefore, over the lifetime of this transfer, we will have 10 threads. We can only run 3 at a time. So, there will be 3 cycles, and the last cycle will only use 1 thread to transfer the last 100 files. (3 threads x 100 files = 300 files per cycle)
The current architecture by example is:
A System.Threading.Timer checks the queue every 5 seconds for something to do by calling GetScheduledTask()
If there's nothing to, GetScheduledTask() simply does nothing
If there is work, create a ThreadPool thread to process the work [Work Thread A]
Work Thread A sees that there are 1000 files to transfer
Work Thread A sees that it can only have 3 threads running to the system it is getting files from
Work Thread A starts three new work threads [B,C,D] and transfers
Work Thread A waits for B,C,D [WaitHandle.WaitAll(transfersArray)]
Work Thread A sees that there are still more files in the queue (should be 700 now)
Work Thread A creates a new array to wait on [transfersArray = new TransferArray[3] which is the max for System A, but could vary on system
Work Thread A starts three new work threads [B,C,D] and waits for them [WaitHandle.WaitAll(transfersArray)]
The process repeats until there are no more files to move.
Work Thread A signals that it is done
I am using ManualResetEvent to handle the signaling.
My questions are:
Is there any glaring circumstance which would cause a resource leak or problem that I am experiencing?
Should I loop thru the array after every WaitHandle.WaitAll(array) and call array[index].Dispose()?
The Handle count under the Task Manager for this process slowly creeps up
I am calling the initial creation of Worker Thread A from a System.Threading.Timer. Is there going to be any problems with this? The code for that timer is:
(Some class code for scheduling)
private ManualResetEvent _ResetEvent;
private void Start()
{
_IsAlive = true;
ManualResetEvent transferResetEvent = new ManualResetEvent(false);
//Set the scheduler timer to 5 second intervals
_ScheduledTasks = new Timer(new TimerCallback(ScheduledTasks_Tick), transferResetEvent, 200, 5000);
}
private void ScheduledTasks_Tick(object state)
{
ManualResetEvent resetEvent = null;
try
{
resetEvent = (ManualResetEvent)state;
//Block timer until GetScheduledTasks() finishes
_ScheduledTasks.Change(Timeout.Infinite, Timeout.Infinite);
GetScheduledTasks();
}
finally
{
_ScheduledTasks.Change(5000, 5000);
Console.WriteLine("{0} [Main] GetScheduledTasks() finished", DateTime.Now.ToString("MMddyy HH:mm:ss:fff"));
resetEvent.Set();
}
}
private void GetScheduledTask()
{
try
{
//Check to see if the database connection is still up
if (!_IsAlive)
{
//Handle
_ConnectionLostNotification = true;
return;
}
//Get scheduled records from the database
ISchedulerTask task = null;
using (DataTable dt = FastSql.ExecuteDataTable(
_ConnectionString, "hidden for security", System.Data.CommandType.StoredProcedure,
new List<FastSqlParam>() { new FastSqlParam(ParameterDirection.Input, SqlDbType.VarChar, "#ProcessMachineName", Environment.MachineName) })) //call to static class
{
if (dt != null)
{
if (dt.Rows.Count == 1)
{ //Only 1 row is allowed
DataRow dr = dt.Rows[0];
//Get task information
TransferParam.TaskType taskType = (TransferParam.TaskType)Enum.Parse(typeof(TransferParam.TaskType), dr["TaskTypeId"].ToString());
task = ScheduledTaskFactory.CreateScheduledTask(taskType);
task.Description = dr["Description"].ToString();
task.IsEnabled = (bool)dr["IsEnabled"];
task.IsProcessing = (bool)dr["IsProcessing"];
task.IsManualLaunch = (bool)dr["IsManualLaunch"];
task.ProcessMachineName = dr["ProcessMachineName"].ToString();
task.NextRun = (DateTime)dr["NextRun"];
task.PostProcessNotification = (bool)dr["NotifyPostProcess"];
task.PreProcessNotification = (bool)dr["NotifyPreProcess"];
task.Priority = (TransferParam.Priority)Enum.Parse(typeof(TransferParam.SystemType), dr["PriorityId"].ToString());
task.SleepMinutes = (int)dr["SleepMinutes"];
task.ScheduleId = (int)dr["ScheduleId"];
task.CurrentRuns = (int)dr["CurrentRuns"];
task.TotalRuns = (int)dr["TotalRuns"];
SchedulerTask scheduledTask = new SchedulerTask(new ManualResetEvent(false), task);
//Queue up task to worker thread and start
ThreadPool.QueueUserWorkItem(new WaitCallback(this.ThreadProc), scheduledTask);
}
}
}
}
catch (Exception ex)
{
//Handle
}
}
private void ThreadProc(object taskObject)
{
SchedulerTask task = (SchedulerTask)taskObject;
ScheduledTaskEngine engine = null;
try
{
engine = SchedulerTaskEngineFactory.CreateTaskEngine(task.Task, _ConnectionString);
engine.StartTask(task.Task);
}
catch (Exception ex)
{
//Handle
}
finally
{
task.TaskResetEvent.Set();
task.TaskResetEvent.Dispose();
}
}
0x8007000E is an out-of-memory error. That and the handle count seem to point to a resource leak. Ensure you're disposing of every object that implements IDisposable. This includes the arrays of ManualResetEvents you're using.
If you have time, you may also want to convert to using the .NET 4.0 Task class; it was designed to handle complex scenarios like this much more cleanly. By defining child Task objects, you can reduce your overall thread count (threads are quite expensive not only because of scheduling but also because of their stack space).
I'm looking for answers to a similar problem (Handles Count increasing over time).
I took a look at your application architecture and like to suggest you something that could help you out:
Have you heard about IOCP (Input Output Completion Ports).
I'm not sure of the dificulty to implement this using C# but in C/C++ it is a piece of cake.
By using this you create a unique thread pool (The number of threads in that pool is in general defined as 2 x the number of processors or processors cores in the PC or server)
You associate this pool to a IOCP Handle and the pool does the work.
See the help for these functions:
CreateIoCompletionPort();
PostQueuedCompletionStatus();
GetQueuedCompletionStatus();
In General creating and exiting threads on the fly could be time consuming and leads to performance penalties and memory fragmentation.
There are thousands of literature about IOCP in MSDN and in google.
I think you should reconsider your architecture altogether. The fact that you can only have 3 simultaneously connections is almost begging you to use 1 thread to generate the list of files and 3 threads to process them. Your producer thread would insert all files into a queue and the 3 consumer threads will dequeue and continue processing as items arrive in the queue. A blocking queue can significantly simplify the code. If you are using .NET 4.0 then you can take advantage of the BlockingCollection class.
public class Example
{
private BlockingCollection<string> m_Queue = new BlockingCollection<string>();
public void Start()
{
var threads = new Thread[]
{
new Thread(Producer),
new Thread(Consumer),
new Thread(Consumer),
new Thread(Consumer)
};
foreach (Thread thread in threads)
{
thread.Start();
}
}
private void Producer()
{
while (true)
{
Thread.Sleep(TimeSpan.FromSeconds(5));
ScheduledTask task = GetScheduledTask();
if (task != null)
{
foreach (string file in task.Files)
{
m_Queue.Add(task);
}
}
}
}
private void Consumer()
{
// Make a connection to the resource that is assigned to this thread only.
while (true)
{
string file = m_Queue.Take();
// Process the file.
}
}
}
I have definitely oversimplified things in the example above, but I hope you get the general idea. Notice how this is much simpler as there is not much in the way of thread synchronization (most will be embedded in the blocking queue) and of course there is no use of WaitHandle objects. Obviously you would have to add in the correct mechanisms to shut down the threads gracefully, but that should be fairly easy.
It turns out the source of this strange problem was not related to architecture but rather because of converting the solution from 3.5 to 4.0. I re-created the solution, performing no code changes, and the problem never occurred again.
Using C# / .NET 3.5.
Currently I'm populating 2 DataTables one after the other using SqlDataAdapter.Fill().
I want to populate both of these DataTables in parallel, at the same time by doing each one asynchronously. However, there is no asynchronous version of the Fill() method - i.e. BeginFill() would be great!
One approach I've tried is (pseudo):
SqlCommand1.BeginExecuteReader // 1st query, for DataTable1
SqlCommand2.BeginExecuteReader // 2nd query, for DataTable2
SqlCommand1.EndExecuteReader
SqlCommand2.EndExecuteReader
DataTable1.Load(DataReader1)
DataTable2.Load(DataReader2)
However, DataTable.Load() takes a long time:
It takes 3 seconds to do step 1 to step 4.
Step 5 then takes 22 seconds.
Step 6 takes 17 seconds.
So, combined 39 seconds for steps 5 and 6.
The end result is, this gives me no benefit over just doing 2 SqlDataAdapter.Fills, one after the other. I want the net result to be that the entire process takes only as long as the longest query (or as close to that as possible).
Looking for recommended ways forward to end up with something that is truly an asynchronous approach to filling a DataTable.
Or do I just manage it myself and roll 2 separate threads, each one filling a DataTable?
I would suggest have a separate worker thread for each. You could use ThreadPool.QueueUserWorkItem.
List<AutoResetEvent> events = new List<AutoResetEvent>();
AutoResetEvent loadTable1 = new AutoResetEvent(false);
events.Add(loadTable1);
ThreadPool.QueueUserWorkItem(delegate
{
SqlCommand1.BeginExecuteReader;
SqlCommand1.EndExecuteReader;
DataTable1.Load(DataReader1);
loadTable1.Set();
});
AutoResetEvent loadTable2 = new AutoResetEvent(false);
events.Add(loadTable2);
ThreadPool.QueueUserWorkItem(delegate
{
SqlCommand2.BeginExecuteReader;
SqlCommand2.EndExecuteReader;
DataTable2.Load(DataReader2);
loadTable2.Set();
});
// wait until both tables have loaded.
WaitHandle.WaitAll(events.ToArray());
This is because the DataTable has a lot of objects to create (rows, values). You should have the execution of the adapter and population of a datatable all done in a different thread, and synchronise waiting for each operation to finish before you continue.
The following code was written in Notepad and probably doesn't even compile, but hopefully you get the idea...
// Setup state as a parameter object containing a table and adapter to use to populate that table here
void DoWork()
{
List<AutoResetEvent> signals = GetNumberOfWaitHandles(2);
var params1 = new DataWorkerParameters
{
Command = GetCommand1();
Table = new DataTable();
}
var params2 = new DataWorkerParameters
{
Command = GetCommand2();
Table = new DataTable();
}
ThreadPool.QueueUserWorkItem(state =>
{
var input = (DataWorkerParameters)state;
PopulateTable(input);
input.AutoResetEvent.Set(); // You can use AutoResetEvent.WaitAll() in the caller to wait for all threads to complete
},
params1
);
ThreadPool.QueueUserWorkItem(state =>
{
var input = (DataWorkerParameters)state;
PopulateTable(input);
input.AutoResetEvent.Set(); // You can use AutoResetEvent.WaitAll() in the caller to wait for all threads to complete
},
params2
);
WaitHandle.WaitAll(signals.ToArray());
}
void PopulateTable(DataWorkerParameters parameters)
{
input.Command.ExecuteReader();
input.Table.Load(input.Command);
}