I have a Winforms application that does some hefty database work. It does a lot of reading and some inserting of records. The operation can take up to 30 seconds to a minute, depending on the size of the database. Right now, my program works just fine.
My problem is, while my program is doing this, my UI thread is being blocked till the database stuff is done. I'd like to put the database functions into a thread, however, according to this MSDN article, ADO.NET is not thread-safe.
Is there a way to provide a progress bar on my UI or some way around the "ADO.NET is not thread safe"? Maybe to write my own classes that inherit SqlDataReader or SqlConnection to make it thread-safe. Is this even possible?
Just use a backgroundworker and make sure each thread (bgw) creates its own Connection object.
And regarding to the linked article, make sure you have a Concurrency-safe way of generating keys. Letting the Db generate them is the best option.
Is there a way to provide a progress bar on my UI
That's tricky. A Bgw can ReportProgress just fine but getting it from a running query is a problem. Even the total nr of rows is usually only known at the end.
But you can always fake it, the user doesn't mind/know.
Related
I have a series of calculations that need to be processed - the calculations and the order they run are all defined by the user on the UI.
If they just ran one after each other, it wouldn't be too hard. However, some of the calculations need to be processed concurrently and all calculations must have the ability to be individually paused at any time. I also need to be able to re-arrange orders or add new calculations to be processed at any time. So whatever I do must be flexible enough to handle this.
On the UI, imagine a listbox (a queue, if you like) of usercontrols - with each usercontrol displaying the name of the calculation and a pause button. And I can add calculations to this list at any time during processing.
What is the best way to do this?
Should I be running each calculation in its own thread? If so, how should I store the list of running processes? How will I pass the queue to the calculation processor? How will I be able to ensure that every time the queue changes (new ordering or new calculation) the calculation processor will be made aware of this?
My initial thoughts were to have:
CalcProcessor class
CalcCalculation class
In CalcProcessor have 2 Lists of CalcCalculations. One being the "queue" as shown on the UI (perhaps a pointer to it? Or some other way to ensure it updates live), and the other being the list of currently running calculations.
Somehow I need to get the CalcCalculation to be running in its own thread to process the calculation, and be able to handle any pause events. So I need some way to transmit the info of the Pause button being pressed in the UI to the CalcProcessor object, and then into the correct CalcCalculation.
Edit in response to David Hope:
Thanks for your reply.
Yes, there are n calculations but this could change at any time due to being able to add more calculations to process on the UI.
They do not need to share data in anyway. There will be a setting in the application to specify how many should run concurrently (ie. 10 at any given time, the first 10 in the queue for example - and when 1 finishes the next calculation in the queue will start processing).
The calculation will involve taking data from some data source - it could be a database or a file, and analysing it and performing some calculations on that data. When I say the calculation needs to be paused, I don't mean pausing the thread... I just mean (for example, as I haven't written this part of the application yet) if it is reading row by row from a database and doing some live calculations pausing at the completion of processing the current row... and continuing on when the pause button is unclicked on the UI - which could be done with something as primitive as a while(notPaused) loop providing I can get the Pause information from the UI into the thread.
There are several questions here:
How to synchronize the UI and the model?
I think you got this one backwards. Your model shouldn't have a “pointer” to the queue you're showing in the UI. Instead, the queue should be in your model and you should use databinding together with INotifyPropertyChange and ObservableCollection to show the queue on the UI. (At least that's how it's done in WPF.)
This way, you can manipulate your queue directly from your model, and it will automatically show on the UI.
How to start and monitor calculations?
I think Tasks are ideal for this. You can start a Task using Task.Factory.StartNew(). Since it seems your Tasks will take long to execute, you might consider using TaskCreationOptions.LongRunning. You can also use the Task to find out when is the calculation complete (or if it failed with an exception).
How to pause running calculations?
You can use ManualReserEventSlim for that. Normally, it would be set, but if you wanted to pause a running Task, you would Reset() it. The calculation will need to periodically call Wait() on that event. It's not possible to reasonably pause a running thread without cooperation from the calculation on that thread.
If you were using C# 5.0, a better approach would be to use something like PauseToken.
In Framework 4.5, the answer here is the Async API, which removes the need to manage threads. For details, look at the async/await keywords.
From a broader perspective, a "CalcProcessor" class is a good idea, but I think the Task object will suffice to replace your "CalcCalculation" class. The Processor can simply have an Enumerable of Tasks. The Processor can expose methods for managing the queue, if needed, as well as returning information about its status. When your application finally reaches a state where it must have the results, you can use the AwaitAll method to block the CalcProcessor's thread until all of the tasks complete.
Without more information about the actual goal here, it's hard to give better advice.
You can use Observer Pattern to display results on UI and order changes back in to Processor. State and Command patterns will help you to start, pause, cancel the calculations. These patterns have great answers to your questions in design way. Concurrency is still a problem, they do not answer multi-threading problems but they open an easier road to manage threads.
I suggest that you haven't broken the problem down far enough, which is the reason you are frustrated.
You need to start small and build up from there. You mention, but don't define your actual requirements, but they seem to be...
Need to be able to run ?N? calculations
Some need to be run concurrently (does this imply that they share data, if so how are you going to share the data)
Must be able to pause the calculation (don't use Thread.Suspend, as it potentially leaves a thread in an unstable state, particularly bad if you are sharing data), so you will need to build pause points into each calculation. Also need to consider how you are going to communicate the pause/unpause to the calculation
As far as methods, there are several to consider...
Threads are an obvious choice, but require careful tending too (starting, pausing, stopping, etc...)
You could also use BackGroundWorker or possibly Parallel.ForEach
BackGroundWorker contains the framework for cancelling the worker and providing progress (which can be useful).
My recommendation to start would be to go with BackGroundWorker, potentially subclass it to add the Pause/Resume functionality you need. Determine how you are going to manage data sharing (at least use lock to protect against simultaneous access).
You may find BackGroundWorker too restrictive and need to go with Threads, but I'm usually able to avoid it.
If you post more clear requirements, or samples of what you've tried and didn't work, I'll be happy to help more.
For queue you can use heap data structure (priority queue). This will help prioritize yours tasks. Also you should use Thread Pool for effectively calculations. And try to split you tasks to little parts.
I'll have a database object that can be accessed from multiple threads as well as from the main thread. I don't want them to access the underlying database object concurrently, so I'll write a set of thread safe public methods that can be accessed from multiple threads.
My first idea was to use a lock around my connection such as lock(oleDbConnection), but the problem is that I would have to lock it for the main thread since is one more thread that can access it. Which would mean rewriting lots of code.
But, since these threads and the main thread wont access the database very often, how about just using some of my control's (maybe the main form's) Invoke method every time I call any of the database methods from another thread. This way, as far as I understand, these methods would be never called concurrently, and I wouldn't need to worry about the main thread. I guess the only problem would be degrading performance a little bit, but as I said, the database is not accessed that often; the reason why I use threads is not so that they can access the database concurrently but so that they can perform other operations concurrently.
So does this sound like a good idea? Am I missing something? Sounds a bit too easy so I'm suspicious.
It sounds like it would work AFAIK, but it also sounds like a really bad idea.
The problem is that when writing lock you are saying "I want this code to be a critical section", whereas when writing Invoke you are saying "I want this to be executed on the UI thread". These two things are certainly not equivalent, which can lead to lots of problems. For example:
Invoke is normally used to access UI controls. What if a developer sees Invoke and nothing UI-related, and goes "gee, that's an unneeded Invoke; let's get rid of it"?
What if more than one UI thread ends up existing?
What if the database operation takes a long time (or times out)? Your UI would stop responding.
I would definitely go for the lock. You typically want the UI thread responsive when performing operations that may take time, which includes any sort of DB access; you don't know whether it's alive or not for instance.
Also, the typical way to handle connections is to create, use and dispose the connection for each request, rather than reusing the same connection. This might perhaps solve some of your concurrency problems.
Why don't you try to use Connection Pool. Every thread can do its work with a different DB connection and send the result to main thread with Invoke. Connection Pooling is a very common approach used in Servers.
See Using Connection Pooling with SQL Server
i have created a windows application(using c# .net) for debugging contest in our department.
in this many user use the same database to select the list of questions and update the marks in their respective id alone.
does it required to use threading concept when they update their marks in the database..
any1 please help me..
thanks in advace...
Mutil-Threading or multiple threads are used in scenarios where you want to do more than one task at one time or do some tasks simultaneously. You should think about your scenario and possible use of multiple threads in your scenario. If you think there is some task which can be divided in to two separate tasks and they can run in parallel, you can use multi-threading to gain performance improvements. Similarly if you think there is some task that is heavy and takes huge time you can move that task to Background Thread and use main thread to deal with some other task in parallel. It all depends on your scenario.
Now coming to your scenario if it is a windows forms application most likely there will be only one user of this app at one time who will be interacting through UI. If this assumption is correct i don't think so you will need multi-threading. If user is doing some inputs thorough UI and he clicks save button at the end to save info in DB you don't need multi-threading a single UI thread will be enough to do this
No this is not needed. Each user will cosume a connection from the database connection pool, and those work concurrently and no parallel programming is required.
If you update a database from different threads, it will not corrupt. This is different from regular C#, where you need to apply locks to protect your objects. You may be required to use transactions to ensure that your database updates don't interfere with each other at a higher level. Very simply put, transactions ensure that your database stays consistent if you edit your database at multiple tables, or if your database changes depend on the database contents, such as adding an order from a customer, transactions prevent you add an order for a deleted customer.
You need to use non-UI thread for any database interactions, otherwise UI may become unresponsive. E.g. if you execute a long query from UI thread (or your connection was disrupted, or the database is under heavy use or whatever, anything that can go wrong - will go wrong), UI thread gets blocked until full response is received.
In the situations where you have multiple users, which may update the same data in the database, you may need to introduce transactions to ensure correct control and data flow - ACID.
My application connects to MySQL but sometimes it takes a while and the GUI is getting frozen. I would like to do the connection on the other thread, I guess BeginInvoke would be the best way (I know about background worker but I would like to learn this). I have studied MSDN page but I did not understand what is the best way to use?
They also say that you can use only callback when the thread that called the async.method does not need to know the results...I dont understand it as I believe I can set some variable in the other thread to "pass" the result well.
I would just need the GUI to be not frozen while the connection is being established. Thank you for your advice.
By far the easiest way to handle it is to use a BackgoundWorker. It is specifically designed to take care of most threading issues such as marshalling progress events and completion notices from the background thread to the GUI thread. I've used it to great success with both WinForms and WPF.
I know many other methods for doing this, but they all take two or three attempts for me to get right.
I've been working on the same project now since Christmas 2008. I've been asked to take it from a Console Application (which just prints out trace statements), to a full Windows App. Sure, that's fine. The only thing is there are parts of the App that can take several minutes to almost an hour to run. I need to multithread it to show the user status, or errors. But I have no idea where to begin.
I've aready built a little UI in WPF. It's very basic, but I'd like to expand it as I need to. The app works by selecting a source, choosing a destination, and clicking start. I would like a listbox to update as the process goes along. Much in the same way SQL Server Installs, each step has a green check mark by its name as it completes.
How does a newbie start multithreading? What libraries should I check out? Any advice would be greatly appreciated.
p.s. I'm currently reading about this library, http://www.codeplex.com/smartthreadpool
#Martin: Here is how my app is constructed:
Engine: Runs all major components in pre-defined order
Excel: Library I wrote to wrap COM to open/read/close/save Workbooks
Library: Library which understands different types of workbook formats (5 total)
Business Classes: Classes I've written to translate Excel data and prep it for Access
Db Library: A Library I've written which uses ADO.NET to read in Access data
AppSettings: you get the idea
Serialier: Save data in-case of app crash
I use everything from LINQ to ADO.NET to get data, transform it, and then output it.
My main requirement is that I want to update my UI to indicate progress
#Frank: What happens if something in the Background Worker throws an Exception (handled or otherwise)? How does my application recieve notice?
#Eric Lippert: Yes, I'm investigating that right now. Before I complicate things.
Let me know if you need more info. Currently I've running this application from a Unit Test, so I guess callig it a Console Application isn't true. I use Resharper to do this. I'm the only person right now who uses the app, but I'd like a more attractive interface
I don't think you specify the version of the CLR you are using, but you might check out the "BackgroundWorker" control. It is a simple way to implemented multiple threads.
The best part, is that it is a part of the CLR 2.0 and up
Update in response to your update: If you want to be able to update the progress in the UI -- for example in a progress bar -- the background worker is perfect. It uses an event that I think is called: ProgressChanged to report the status. It is very elegant. Also, keep in mind that you can have as many instances that you need and can execute all the instances at the same time (if needed).
In response to your question: You could easily setup an example project and test for your question. I did find the following, here (under remarks, 2nd paragraph from the caution):
If the operation raises an exception
that your code does not handle, the
BackgroundWorker catches the exception
and passes it into the
RunWorkerCompleted event handler,
where it is exposed as the Error
property of
System.ComponentModel..::.RunWorkerCompletedEventArgs.
Threading in C# from Joseph Albahari is quite good.
This page is quite a good summary of threading.
By the sound of it you probably don't need anything very complex - if you just start the task and then want to know when it has finished, you only need a few lines of code to create a new thread and get it to run your task. Then your UI thread can bumble along and check periodically if the task has completed.
Concurrent Programming on Windows is THE best book in the existence on the subject. Written by Joe Duffy, famous Microsoft Guru of multithreading. Everything you ever need to know and more, from the way Windows thread scheduler works to .NET Parallels Extensions Library.
Remember to create your delegates to update the UI so you don't get cross-threading issues and the UI doesn't appear to freeze/lockup
Also if you need a lot of notes/power points/etc etc
Might I suggest all the lecture notes from my undergrad
http://ist.psu.edu/courses/SP04/ist411/lectures.html
The best way for a total newcomer to threading is probably the threadpool. We'll probably need to know a little more about these parts to make more in depth recommendations
EDIT::
Since we now have a little more info, I'm going to stick with my previous answer, it looks like you have a loads of tasks which need doing, the best way to do a load of tasks is to add them to the threadpool and then just keep checking if they're done, if tasks need to be done in a specific order then you can simply add the next one as the previous one finishes. The threadpool really is rather good for this kind of thing and I see no reason not to use it in this case
Jason's link is a good article. Things you need to be aware of are that the UI can only be updated by the main UI thread, you will get cross threading exceptions if you try to do it in the worker thread. The BackgroundWorker control can help you there with the events, but you should also know about Control.Invoke (or Control.Begin/EndInvoke). This can be used to execute delegates in the context of the UI thread.
Also you should read up on the gotchas of accessing the same code/variables from different threads, some of these issues can lead to bugs that are intermittent and tricky to track down.
One point to note is that the volatile keyword only guarantees 'freshness' of variable access, for example, it guarantees that each read and write of the variable will be from main memory, and not from a thread or processor cache or other 'feature' of the memory model. It doesnt stop issues like a thread being interrupted by another thread during its read-update-write process (e.g. changing the variables value). This causes errors where the 2 threads have different (or the same) values for the variable, and can lead to things like values being lost, 2 threads having the same value for the variable when they should have different values, etc. You should use a lock/monitor (or other thread sync method, wait handles, interlockedincrement/decrement etc) to prevent these types of problems, which guarantee only one thread can access the variable. (Monitor also has the advantage that it implicitly performs volatile read/write)
And as someone else has noted, you also should try to avoid blocking your UI thread whilst waiting for background threads to complete, otherwise your UI will become unresponsive. You can do this by having your worker threads raise events that your UI subscribes to that indicate progress or completion.
Matt
Typemock have a new tool called Racer for helping with Multithreading issues. It’s a bit advanced but you can get help on their forum and in other online forums (one that strangely comes to mind is stackoverflow :-) )
I'm a newbie to multithreading as well, but I agree with Frank that a background worker is probably your best options. It works through event subscriptions. Here's the basics of how you used it.
First Instantiate a new background worker
Subscribed methods in your code to the background workers major events:
DoWork: This should contain whatever code that takes a long time to process
ProgressChanged: This is envoked whenever you call ReportProgress() from inside the method subscribed to DoWork
RunWorkerCompleted: Envoked when the DoWork method has completed
When you are ready to run your time consuming process you call the RunAsync() method of the background worker. This starts DoWork method on a separate thread, which can then report it's progress back through the ProgressChanged event. Once it completed RunWorkerComplete will be evoked.
The DoWork event method can also check if the user somehow requested that the process be canceled (CanceLAsync() was called)) by checking the value of the CancelPending property.