Develop Reference

multiprocessing in winforms application

I'm working on about 35 batches updating many databases as a part of our daily process at work. Every batch of them was developed in a single web app. Due to database issues, i have collected all of them in one windows application to make use of DB connection pooling and i have assigned a single backgroundworker for each batch. Reaching to 20 batch in the application, every thing is working good. But when i add any other backgroundworker for any other batch, the application hangs.I think this is because i'm running too many threads in one process. Is there a solution for this problem, for example, making the application working with many processes ??!!!.
Regards,
Note,
I have assigned a single machine for this application (Core i7 cpu, 8 gb ram).

How many databases you have to update?
I think it is more recommended to have the number of Threads as the number of Databases

If your UI is freezing while many background workers are active, but recovers when those background workers are finished processing, then likely the UI thread is executing a method which waits for a result or signal from one of the background worker threads.
To fix your problem, you will have to look for UI-related code that deals with synchronization / multi-threading. This might be places where one of the many synchronization objects of .NET are being used (including the lock statement), but it could also involve "dumb" polling loops a-ka while(!worker.IsFinished) Thread.Sleep();.
Another possible reason for the freeze might be that you are running a worker (or worker-related method) accidentally in the UI thread instead in a background thread.
But you will find out when you use the debugger.
To keep the scope of your hunt for problematic methods managable, let your program run in the debugger until the UI freezes. At that moment, pause the program execution in the debugger. Look which code the UI thread is processing then, and you will have found one instance of offending code. (Whatever there is wrong, i can't tell you - because i don't know your code.)
It is quite possible that different UI-related methods in your code will suffer from the same issue. So, if you found the offending code (and were able to fix it) you would want to check on for other problematic methods, but that should be rather easy since at that point of time you will know what to look for...

Step Debug Parallel

With parallel processing be it background or task in debug will jump around. I end up adding a break point at every line. Is there a way to only step in the task and just put a single break point at the beginning?

While paused in the debugger, you can use the Threads window to have more control over context switching. You can right click on a Thread and select Freeze. This will prevent the debugger from switching to that thread while you're stepping through code. You can also Shift-Select multiple threads and Freeze them all. If you Freeze all threads other than the thread you're stepping though, you can step through unhindered by other processing.
It's a bit awkward, but you can also use this to investigate some types of race conditions by explicitly Thawing only one thread and then forcing the active thread to change (using Switch to Thread) at the specific point you want to test. This won't replicate all types of threading synchronization issues (some are much more subtle, dealing with memory caching on separate CPUs and the like), but you can see the effects of alternate execution orders (such as some deadlock scenarios.)

Processing Windows Messages in blocked UI Thread?

This is possibly related to ProgressBar updates in blocked UI thread but is a little different.
While troubleshooting a crashing WinForms control (a DevExpress treelist) a member of our team encountered an unusual situation, and I'm wondering if anyone can help us understand what's going on. English is not his first language, so I'm posting on his behalf.
Please see this screenshot from Visual Studio 2005.
Note the following points:
The main UI thread is stopped and is currently in a DevExpress control draw method.
The code shown on screen is from a point earlier in the same call-stack. This code is in the data layer and was called in response to the control's request for an image to display for the tree node. (perhaps also originating from a Paint handler)
The displayed code is from earlier in the callstack, on the main UI thread, and is currently waiting on a lock! Since remote systems can send events which are processed on background threads in the data model (i.e., data models are sync'd between client and servers), we lock to keep the data collections thread safe.
As the callstack shows, we continued to process paint messages on the UI thread, while we would expect the thread to be blocked.
This is very difficult to replicate, and I have not been able to do so using a simpler test project on my own box. When this situation arises, however, the result is that the DevExpress control's internal state can be messed up, causing the control to crash. This doesn't really seem like a bug in the control, since it was no doubt written with the assumption that these paint methods are running only on the UI thread. What we see here makes it look like the UI thread is acting like two threads.
It would seem possible that this is merely a Visual Studio bug in the presentation of the callstack, except that this whole endeavor is resulting from an effort to troubleshoot the occasional crash of the control in the released app (in which case it shows as a big red X in the UI), so it seems the problem is not isolated to the debug environment.
Alright, that was complicated, but hopefully made sense. Any ideas?

I would strongly recommend against locking the UI to wait for background processing. Consider something like multiple buffering. You can probably get this behavior fairly easily by utilizing thread-safe collections in .NET 4, but if that's not an option there are versions of those in the Parallel Extensions released prior to v4.

What about altering the synchronization scheme so that you don't need to acquire exclusive locks to read data?
In situations where you can be sure that a read will always produce consistent data even when it happens while the data is also being written, you might be able to get away with having no lock statements for getters. Otherwise there's ReaderWriterLockSlim, which permits multiple concurrent readers but still allows you to stop the presses for write operations.
It doesn't fix everything, but it does at least reduce the number of opportunities for deadlocks.

We see something similar in our project. The stack trace looks like the pump's event loop is called while the UI thread is waiting on a lock. This could happen if Monitor.enter has some special behavior when called on the UI thread. I believe this is what's happening, but I haven't found any documentation to back it up yet.
Probably has something to do with synchronization contexts :)

Monitoring C# Threads - Which does what/when

As everybody, I am used to debugging my code in VS in step-by-step mode. Well, now that I have an application with many Background Workers everywhere, I am not in Kansas anymore.
What is the most efficient way to debug threaded applications and be able to monitor each and every thread to keep track of what's happening all over the code?
As of now, I stick to good ol' debugging using separate logger instances for each Thread, but this is slowly becoming a nightmare and I'll soon be drowning into my own logs.

Don't try to debug everything all at once. Narrow your focus to a particular behavior in one thread or pair of threads that interact around some mutex lock. If accessing a shared resource is the problem, set breakpoints around use of that resource (which should be in common code, not all over the place).
If you just want to see that thread 3 completed before thread 1, or that thread 2 used up all its work items and is sitting idle, use logs for that.
You can also use the VS Threads view to see what each thread is doing whenever the process is stopped at any breakpoint on any thread. This can give you some insight into what all the threads are doing at any given instant.

A small tip that might ease your pain is to use Visual Studio to freeze threads that you are not interested in. Then when you tell the debugger to continue, the frozen threads will never execute and will not hit breakpoints and confuse you.
Maybe you can use this method to allow only the threads you are debugging to work. E.g. keep one thread that enqueues and one thread that dequeues active, but freeze everything else.
You can freeze/thaw threads from Visual Studio's Threads window, by right-clicking on a thread.

Write it correctly the first time.
Joking aside, the trick to debugging is to break it down into manageable parts. Tackle one worker task at a time, make absolutely sure it does what it's supposed to.
Once you've done that, debugging issues in the main thread is a lot easier, because you can pretty much ignore the background workers and just presume they're yielding correct results when they should be.
The only place left that's harder to debug than a single-threaded application is the interconnect between the threads, which shouldn't be much more difficult if you're using the libraries the way you should be.

I Stumbled upon a detailed MSDN article about Debugging Multithreaded Applications
wich was of great help. Thanx for all the previous answers that guided me towards the right track.

Multithreading: how to update UI to indicate progress

I've been working on the same project now since Christmas 2008. I've been asked to take it from a Console Application (which just prints out trace statements), to a full Windows App. Sure, that's fine. The only thing is there are parts of the App that can take several minutes to almost an hour to run. I need to multithread it to show the user status, or errors. But I have no idea where to begin.
I've aready built a little UI in WPF. It's very basic, but I'd like to expand it as I need to. The app works by selecting a source, choosing a destination, and clicking start. I would like a listbox to update as the process goes along. Much in the same way SQL Server Installs, each step has a green check mark by its name as it completes.
How does a newbie start multithreading? What libraries should I check out? Any advice would be greatly appreciated.
p.s. I'm currently reading about this library, http://www.codeplex.com/smartthreadpool
#Martin: Here is how my app is constructed:
Engine: Runs all major components in pre-defined order
Excel: Library I wrote to wrap COM to open/read/close/save Workbooks
Library: Library which understands different types of workbook formats (5 total)
Business Classes: Classes I've written to translate Excel data and prep it for Access
Db Library: A Library I've written which uses ADO.NET to read in Access data
AppSettings: you get the idea
Serialier: Save data in-case of app crash
I use everything from LINQ to ADO.NET to get data, transform it, and then output it.
My main requirement is that I want to update my UI to indicate progress
#Frank: What happens if something in the Background Worker throws an Exception (handled or otherwise)? How does my application recieve notice?
#Eric Lippert: Yes, I'm investigating that right now. Before I complicate things.
Let me know if you need more info. Currently I've running this application from a Unit Test, so I guess callig it a Console Application isn't true. I use Resharper to do this. I'm the only person right now who uses the app, but I'd like a more attractive interface

I don't think you specify the version of the CLR you are using, but you might check out the "BackgroundWorker" control. It is a simple way to implemented multiple threads.
The best part, is that it is a part of the CLR 2.0 and up
Update in response to your update: If you want to be able to update the progress in the UI -- for example in a progress bar -- the background worker is perfect. It uses an event that I think is called: ProgressChanged to report the status. It is very elegant. Also, keep in mind that you can have as many instances that you need and can execute all the instances at the same time (if needed).
In response to your question: You could easily setup an example project and test for your question. I did find the following, here (under remarks, 2nd paragraph from the caution):
If the operation raises an exception
that your code does not handle, the
BackgroundWorker catches the exception
and passes it into the
RunWorkerCompleted event handler,
where it is exposed as the Error
property of
System.ComponentModel..::.RunWorkerCompletedEventArgs.

Threading in C# from Joseph Albahari is quite good.

This page is quite a good summary of threading.
By the sound of it you probably don't need anything very complex - if you just start the task and then want to know when it has finished, you only need a few lines of code to create a new thread and get it to run your task. Then your UI thread can bumble along and check periodically if the task has completed.

Concurrent Programming on Windows is THE best book in the existence on the subject. Written by Joe Duffy, famous Microsoft Guru of multithreading. Everything you ever need to know and more, from the way Windows thread scheduler works to .NET Parallels Extensions Library.

Remember to create your delegates to update the UI so you don't get cross-threading issues and the UI doesn't appear to freeze/lockup
Also if you need a lot of notes/power points/etc etc
Might I suggest all the lecture notes from my undergrad
http://ist.psu.edu/courses/SP04/ist411/lectures.html

The best way for a total newcomer to threading is probably the threadpool. We'll probably need to know a little more about these parts to make more in depth recommendations
EDIT::
Since we now have a little more info, I'm going to stick with my previous answer, it looks like you have a loads of tasks which need doing, the best way to do a load of tasks is to add them to the threadpool and then just keep checking if they're done, if tasks need to be done in a specific order then you can simply add the next one as the previous one finishes. The threadpool really is rather good for this kind of thing and I see no reason not to use it in this case

Jason's link is a good article. Things you need to be aware of are that the UI can only be updated by the main UI thread, you will get cross threading exceptions if you try to do it in the worker thread. The BackgroundWorker control can help you there with the events, but you should also know about Control.Invoke (or Control.Begin/EndInvoke). This can be used to execute delegates in the context of the UI thread.
Also you should read up on the gotchas of accessing the same code/variables from different threads, some of these issues can lead to bugs that are intermittent and tricky to track down.
One point to note is that the volatile keyword only guarantees 'freshness' of variable access, for example, it guarantees that each read and write of the variable will be from main memory, and not from a thread or processor cache or other 'feature' of the memory model. It doesnt stop issues like a thread being interrupted by another thread during its read-update-write process (e.g. changing the variables value). This causes errors where the 2 threads have different (or the same) values for the variable, and can lead to things like values being lost, 2 threads having the same value for the variable when they should have different values, etc. You should use a lock/monitor (or other thread sync method, wait handles, interlockedincrement/decrement etc) to prevent these types of problems, which guarantee only one thread can access the variable. (Monitor also has the advantage that it implicitly performs volatile read/write)
And as someone else has noted, you also should try to avoid blocking your UI thread whilst waiting for background threads to complete, otherwise your UI will become unresponsive. You can do this by having your worker threads raise events that your UI subscribes to that indicate progress or completion.
Matt

Typemock have a new tool called Racer for helping with Multithreading issues. It’s a bit advanced but you can get help on their forum and in other online forums (one that strangely comes to mind is stackoverflow :-) )

I'm a newbie to multithreading as well, but I agree with Frank that a background worker is probably your best options. It works through event subscriptions. Here's the basics of how you used it.
First Instantiate a new background worker
Subscribed methods in your code to the background workers major events:
DoWork: This should contain whatever code that takes a long time to process
ProgressChanged: This is envoked whenever you call ReportProgress() from inside the method subscribed to DoWork
RunWorkerCompleted: Envoked when the DoWork method has completed
When you are ready to run your time consuming process you call the RunAsync() method of the background worker. This starts DoWork method on a separate thread, which can then report it's progress back through the ProgressChanged event. Once it completed RunWorkerComplete will be evoked.
The DoWork event method can also check if the user somehow requested that the process be canceled (CanceLAsync() was called)) by checking the value of the CancelPending property.