I have the following scenario:
I have a Database storing Jobs which are catched and processed by a server. The database is accessed via Entity Framework.
The server processes the jobs in parallel using multiple threads. For this I have one thread which periodically checks for new Jobs in the database and Distributes them to worker threads.
My problem now is that my Entities have a Progress property which should be updated by the worker threads and periodically written to the database.
The worker threads update the property quite often (many times per second), but for my requirements it's enough if the database is updated every few seconds and I don't want to make to many unnecessary updates to the database.
So far my solution is to have the worker threads write the progress to the Entity directly and have the thread which checks for updates also issue these changes to the database.
My question here is: Is this thread safe from the EF point of view. Can I update properties of an Entity from one thread and write the changes to the Database on another thread? Do I need any case of locking? Keep in mind that I use the DataContext only in one thread (add least explicitly since I don't know what EF does internally when I update an (Non-POCO) entity.
Another requirement now is that I need to load additional data from the database in the worker processes. I assume I have to use separate DataContexts for this and I don't really like having to manage entities from two separate data contexts in the same thread.
Do you have any advice how to structure this in a nice way?
Since every worker is only updating the status for one Job-Entity, one idea was to expose the progress as a property in the worker-threads class which is fetched by the main thread which would then update the entities and issue the update to the database. But I still need the original Job-Entity in the worker thread to read configuration data and if I reattach it to the DataContext of the worker thread I cannot use the Entity anymore in the main thread. I want to avoid loading the same entity 2 times if it's not really necessary...
Is it possible to duplicate an Entity automatically, to use it in 2 separate DataContexts?
Thanks for any ideas!
At the end I made the following decision:
My main class / main thread reads the jobs from the database and distributes them to various worker threads. For each job there is a corresponding Job-Executor whose .Execute() method is run by the worker thread.
By convention, the Executor-classes read all necessary configuration data from the Job-Entity when it's constructed and isn't allowed to touch it anymore during the execution period. Since construction of the Executor class is done from the main thread, there is no multi threaded access here.
Changing state, like the progress of a Job is exposed via properties on the executor class and periodically synched to the entities/database from the main thread.
The worker threads also have their own DataContext to load additional data when necessary.
All other multi-thread access to DataContext is synchronized with locks.
I think that you should redesign your system a bit.
You are running into trouble because the progress of an entity is stored within the entity.
If you seperate it such that you have one table / context that contains the progress of all jobs. Each thread could update this and it could be saved to the database periodically using a timer.
Related
I have many-tables GUI application. Each table filled with assistance of its own BackgroundWorker instance. But now I need render a table that rows are gotten from different slow remote sources. So each its row also should be shown parallel as soon as it be received.
I see two way:
instead of BackgroundWorker for the table to create the instance per
row;
continue use BackgroundWorker (for interactions with UI) but in
DoWorkEventHandler perform Parallel.ForEach requests for source in
collection with ProgressChanged call after its response.
Which is more correct?
assuming it's like my case i have source that are mostly third party and no control over and split around different format (web service, WCF, Local Unmanaged DLL,.Net DLL, Java Service and Excel) just to fill a single list.
Anyhow in my case i used 7 workers.
the first one list all the source and nowaday i have 40-ish sources.
then the worker start up to 6 other workers with 1 source each and update progress async. once one worker finish the main worker start a new one with the next of the list and so on.
I noticed that over 6 i slow performance in my case but that depend on your architecture and type of source. if i had less source accessed by the web i could increase that but bandwidth slows down.
I have a Winforms application that does some hefty database work. It does a lot of reading and some inserting of records. The operation can take up to 30 seconds to a minute, depending on the size of the database. Right now, my program works just fine.
My problem is, while my program is doing this, my UI thread is being blocked till the database stuff is done. I'd like to put the database functions into a thread, however, according to this MSDN article, ADO.NET is not thread-safe.
Is there a way to provide a progress bar on my UI or some way around the "ADO.NET is not thread safe"? Maybe to write my own classes that inherit SqlDataReader or SqlConnection to make it thread-safe. Is this even possible?
Just use a backgroundworker and make sure each thread (bgw) creates its own Connection object.
And regarding to the linked article, make sure you have a Concurrency-safe way of generating keys. Letting the Db generate them is the best option.
Is there a way to provide a progress bar on my UI
That's tricky. A Bgw can ReportProgress just fine but getting it from a running query is a problem. Even the total nr of rows is usually only known at the end.
But you can always fake it, the user doesn't mind/know.
i have created a windows application(using c# .net) for debugging contest in our department.
in this many user use the same database to select the list of questions and update the marks in their respective id alone.
does it required to use threading concept when they update their marks in the database..
any1 please help me..
thanks in advace...
Mutil-Threading or multiple threads are used in scenarios where you want to do more than one task at one time or do some tasks simultaneously. You should think about your scenario and possible use of multiple threads in your scenario. If you think there is some task which can be divided in to two separate tasks and they can run in parallel, you can use multi-threading to gain performance improvements. Similarly if you think there is some task that is heavy and takes huge time you can move that task to Background Thread and use main thread to deal with some other task in parallel. It all depends on your scenario.
Now coming to your scenario if it is a windows forms application most likely there will be only one user of this app at one time who will be interacting through UI. If this assumption is correct i don't think so you will need multi-threading. If user is doing some inputs thorough UI and he clicks save button at the end to save info in DB you don't need multi-threading a single UI thread will be enough to do this
No this is not needed. Each user will cosume a connection from the database connection pool, and those work concurrently and no parallel programming is required.
If you update a database from different threads, it will not corrupt. This is different from regular C#, where you need to apply locks to protect your objects. You may be required to use transactions to ensure that your database updates don't interfere with each other at a higher level. Very simply put, transactions ensure that your database stays consistent if you edit your database at multiple tables, or if your database changes depend on the database contents, such as adding an order from a customer, transactions prevent you add an order for a deleted customer.
You need to use non-UI thread for any database interactions, otherwise UI may become unresponsive. E.g. if you execute a long query from UI thread (or your connection was disrupted, or the database is under heavy use or whatever, anything that can go wrong - will go wrong), UI thread gets blocked until full response is received.
In the situations where you have multiple users, which may update the same data in the database, you may need to introduce transactions to ensure correct control and data flow - ACID.
My team and I are working on an application that accesses a "huge" database, roughly 32M rows in 8 months. The application is a RIA Domain Service application. We have optimized the application and the database design in such a way that even on a box with very limited resources the response time is never more than few seconds.
However, there are certain tasks that need to be performed on a large record set (at least 2-3M records per operation). An example is the generation of a monthly report. Definitely we cannot keep the application waiting for a result, because it would hit either the 30 seconds timeout.
After reading this post, I thought I could create an [Invoke] method, which spawns a new thread, and consequently it would free the client up. The thread would be in charge of extracting data from the DB and writing them nicely in a PDF. I've tried to implement this scenario, but I get an exception, which says that the underlying connection has already been disposed...
Is this approach correct? Can I achieve what I am trying to do or there is some issue I cannot overcome? And is there any better way to do this?
Cheers,
Gianluca.
Ok, I've realized my question is a bit silly.
As far as I understood, the ObjectContext exists as long as the client is connected, otherwise it gets disposed. Because I was writing an Invoke method that does not require any change tracking, I have resolved by:
- spawning a new thread from within the Invoke method
- instantiating a new EF context inside the worker thread
- disposing the new EF context as soon as the separate thread operation is terminated.
Cheers,
Gianluca.
I have developed an application that pulls X amount of records from my database per X amount of threads. Each thread then iterates the collections that are created and does some data validation for each record.
Once a record is validated / not validated by the application it is updated in the database as being valid / not valid. Records are only pulled by each thread if there are items in the database that have not been run through the application. There is a bit column to indicate if the application retrieved the data.
So, potentially, the system can run out of data depending on the number of threads and records per thread. I want the application to continue to check the database for any records that have not be run, then start the process of creating the threads, and finally validating the data.
Here is an example:
There are 50 records in the database
we are running 5 threads with 10 records per thread.
The application runs, the threads are created, the records are pulled and then processed. Now, the system is out of data. A user imports more data into the DB. The application, still looking to see if there are any records, sees that there are 5 new records in the database. It then starts the process all over to create the threads and process the records.
How can I have the system continue to look for data but allow the user to stop the system if need be. I tried using this:
while(RecordsFound <=0){
…sleepcode
} ;
RunProcessMethod
But the winform locks, obviously, during the waiting period. I tried adding the wait logic to another thread, but was afraid that if I run the process method from that thread, via a delegate, things would get weird since I am creating additional threads inside that method.
Thoughts?
The easiest way to fix this is to use a notification mechanism instead of polling. That is once you've spawned off the threads to read data from the data base, make them responsible for notifying the UI when they are complete instead of having the UI wait for them to be complete.
The easiest way to do this is to pass in a delegate for the threads to call when they are complete with the set of records found. The UI can then update when the records are available
delegate void CallBack(List<Data> found);
void OnDataFound(List<Data> found) {
// Get back on the UI thread
if ( this.InvokeRequired ) {
this.Invoke( new CallBack(OnDataFound), new object[] { found } );
return;
}
// Update display
}
I tried adding the wait logic to another thread, but was afraid that if I run the process method from that thread, via a delegate, things would get weird since I am creating additional threads inside that method. Thoughts?
You don't need to fear this. It's the proper way to handle this type of scenario. There is no problem with a background thread creating additional threads.