My server application that is written in C# starts a new thread every time it needs to insert or remove data from the database. The problem is that since the execution of the threads is arbitrary it is not ensured that a delete command is executed after the insertion of the same object if those events occur almost at the same time.
E.g.: The server receives the command to insert multiple objects. It takes about 5 seconds to insert all the object. After 1 second of execution the server receives the command to delete all those objects again from the database. Since the removal could happen before all objects are completely stored the outcome is unknown.
How can the order of execution of certain thread be managed?
You can use transactions for this and specify different levels for different operations.
For example, you can use the highest level of transactions for writes/updates/deletes but a low level for reads. You can also fine-tune this to allow blocking of only specific rows, compared to tables. Specific terminology depends on the database and data access library you use.
I would advice against using any ordering. Parallel and ordered just don't go well together. For example:
You need to horizontally scale servers, once you add a second server and a load balancer a mutex solution will not work
In a large and distributed systems a message queue won't work either as by the time one thread completed a scan and decided that we good to go, another thread can write a message that should have prevented operation execution. Moreover, given you receive high load, scanning the same queue multiple times is inefficient.
If you know that you receive insert before delete and the problem is just that you don't want to interrupt your insertion then you can just use lock on your insertion code.
static object m_Lock = new object();
public void Insert()
{
lock (m_Lock)
{
InsertRecords();
}
}
public void Remove()
{
lock (m_Lock)
{
RemoveRecords();
}
}
This way you are sure that remove won't happen during insert.
P.S. Seems strange though that you need to insert and then delete right away.
I think the simplest way is to queue all incoming requests to insert objects in one collection, and to queue all incoming requests to delete objects in a second collection.
The server should have a basic loop that does :
a. check if there are incoming inserts , if so -> perform all inserts.
b. check if there are incoming delete requests, if so -> perform all delete requests.
c. sleep for X milli-seconds.
Now, if you have a delete request on an object that does not exist.
you have two options:
a. igore this request and discard it.
b. ignore this request for this round and keep it in the collection for the next N rounds,
before deleting it (Finally deleting it- assuming this is simply a bad request and is not a problem of race condition.)
Use a Queue (with a single servicing thread) to enforce the ordering. You can also use Task Parallel Library to manage tasks with dependencies on other tasks, though that's very difficult with arbitrary DB operations.
I think you need to rethink how you manage the incoming operations, and whether or not their inter-dependencies are predictable enough that you can safely use multiple threads in this way. You may need to add some "depends on" information into incoming operations to achieve that goal.
Related
I have a single instance application that need to be moved in order that can run from multiple servers for performance reasons.
When a specific operation is executed the software need to be sure that the other instances was not working on the same operation.
I made some tests with Redis distributed locks but have some problems: also trying for 10 seconds with a retry each 50 msec sometimes the software is unlucky and is not able to acquire a lock because in the meanwhile other instances that started later was able to acquire the lock.
Exists some distributed locks services that are able to manage queue? Basically what I need is that when the software was not able to acquire the lock because is used by another instance can keep the priority and been the first to be served when the lock was released.
Of course my last solution can be write a custom software that can manage locks with queue, but I'm trying to understand if already exists some other solutions.
Exists some distributed locks services that are able to manage queue? Basically what I need is that when the software was not able to acquire the lock because is used by another instance can keep the priority and been the first to be served when the lock was released.
Do you need the hard canonical way or can you allow some simplification?
The hard canonical way is the one you study at university and requires a lot of custom software. I am a bit rusty, but in a distributed system you must use an ordered and reliable message delivery to be barely sure to keep the FIFO order. A service bus or a queue can guarantee to deliver messages in order, but then in order to hold a distributed lock you need to implement consensus, which is extremely hard. After that, you must deal with all sorts of implementation flaws that are very common in software science.
A consensus algorithm, leader or leaderless, involves a lot of communication to make sure all peers receive the same messages in the same order and are able to determine who is the owner of the lock, even without sending an explicit lock signal after release (which is explicit). Of course, the service bus must deliver the message to the node self before the node acquires the lock.
Think of a distributed algorithm like: the node is listening on the bus. When the first lock request comes in, node knows that the sender is the owner of the lock. Node awaits for other lock requests and puts those in a local queue. When the original holder sends a release message, every node knows the lock owner is the one that sent the second lock request. In order to acquire the lock, a node sends a message to the broadcast, and waits until it receives it to know its queue position. When it received n release messages, it knows it's their turn.
Make it practical
You could use a relational database. I don't have knowledge of your scenario, but a single point of failure could be mitigated by some kind of clustering, master-slave primary-secondary, etc.
SQL databases are f***in great at handling concurrency. They ensure that transactions are executed in a consistent order. If your application uses a database table as queue, you will see that it's very easy to run SQL UPDATE LOCKS SET IS_LOCKED = 1 WHERE LOCK_ID = ? AND IS_LOCKED = 0 without clashes. To handle the queue, put the node name in another table and make sure you ORDER BY a SQL sequence.
I have a long running process that uses a parallel.foreach loop. While in that loop, I'm creating instance of two different classes based on what is passed, performing minor tasks, then adding to thread safe collections. When complete, all data needs to be inserted to sql.
The problem I have is the amount of work done is too large to keep in the collections until all processing is done. I have to push what is retained to SQL occasionally then remove what was pushed from the collection so more processing can continue without running out of memory and I don't know the best way to do that. I can easily do it if it wasn't multi-threaded by checking the count of the collections and if it is over a certain amount, call a function that would push the contents to SQL through a bulk insert or valued table then clear that collection on the next statement. What is the best way to accomplish this within a parallel.foreach?
I'm open to using any collection that is thread safe. So far I'm been using the ConcurrentQueue and thought about switching to BlockingCollection because I didn't see a way to clear the Concurrent Queue. I don't care what order the contents are inserted, but I do need to be able to at least remove what is pushed to sql.
My best solution is to use BlockingCollection.GetConsumingEnumerable(). This way, once over x amount, I can copy the contents of that collection to another thread safe collection, perform my insert, then use that list to remove from the original using BlockingCollection.GetConsumingEnumerable(). Once that's done, dispose of the temp list. I just thought there was a better way because if I have to iterate one at a time to remove, it kind of defeats the purpose of have it multi threaded.
I have seen the use of pulse and wait used but I couldn't find a good use case that seems safe. I could have something come through after I tested the collection to be over a certain amount and get cleared before it was inserted to sql.
I'm using the 4.5 Framework and I'm managing two different collections that need to be pushed but not necessarily at the same time.
I wouldn't recommend clearing a concurrent collection. Instead, I would 'replace' it with a new one - and process the content from the old one while the other threads push their content to the new one.
Interlocked.Exchange is the technique I would use to accomplish this.
I need to setup an automated task that runs every minute and sends emails in the queue. I'm using ASP.NET 4.5 and C#. Currently, I use a scheduler class that starts in the global.asax and makes use of caching and cache callback. I've read this leads to several problems.
The reason I did it that way is because this app runs on multiple load balanced servers and this allows me to have the execution in one place and the code will run even if one or more servers are offline.
I'm looking for some direction to make this better. I've read about Quartz.NET but never used it. Does Quartz.NET call methods from the application? or from a windows service? or from a web service?
I've also read about using a Windows service, but as far as I can tell, those are installed to the server direct. The thing is, I need the task to execute regardless of how many servers are online and don't want to duplicate it. For example, if I have a scheduled task setup on server 1 and server 2, they would both run together therefore duplicating the requests. However, if server 1 was offline, I need server 2 to run the task.
Any advice on how to move forward here or is the global.asax method the best way for the multi-server environment? BTW, the web servers are running Win Server 2012 with IIS 8.
EDIT
In a request for more information, the queue is stored in a database. I should also make mention that the database servers are separate from the web servers. There are two database servers, but only one runs at a time. There is a central storage they both read from so there is only one instance of the database. When one database server goes down, the other comes online.
That being said, would it make more sense to put a Windows Service deployed to both database servers? That would make sure only one runs at a time.
Also, what are your thoughts about running Quartz.NET from the application? As millimoose mentions, I don't necessarily need it running on the web front end, however, doing so allows me to not deploy a windows service to multiple machines and I don't think there would be a performance difference going either way. Thoughts?
Thanks everyone for the input so far. If any additional info is needed, please let me know.
I have had to tackle the exact problem you're facing now.
First, you have to realize that you absolutely cannot reliably run a long-running process inside ASP.NET. If you instantiate your scheduler class from global.asax, you have no control over the lifetime of that class.
In other words, IIS may decide to recycle the worker process that hosts your class at any time. At best, this means your class will be destroyed (and there's nothing you can do about it). At worst, your class will be killed in the middle of doing work. Oops.
The appropriate way to run a long-lived process is by installing a Windows Service on the machine. I'd install the service on each web box, not on the database.
The Service instantiates the Quartz scheduler. This way, you know that your scheduler is guaranteed to continue running as long as the machine is up. When it's time for a job to run, Quartz simply calls a method on a IJob class that you specify.
class EmailSender : Quartz.IJob
{
public void Execute(JobExecutionContext context)
{
// send your emails here
}
}
Keep in mind that Quartz calls the Execute method on a separate thread, so you must be careful to be thread-safe.
Of course, you'll now have the same service running on multiple machines. While it sounds like you're concerned about this, you can actually leverage this into a positive thing!
What I did was add a "lock" column to my database. When a send job executes, it grabs a lock on specific emails in the queue by setting the lock column. For example, when the job executes, generate a guid and then:
UPDATE EmailQueue SET Lock=someGuid WHERE Lock IS NULL LIMIT 1;
SELECT * FROM EmailQueue WHERE Lock=someGuid;
In this way, you let the database server deal with the concurrency. The UPDATE query tells the DB to assign one email in the queue (that is currently unassigned) to the current instance. You then SELECT the the locked email and send it. Once sent, delete the email from the queue (or however you handle sent email), and repeat the process until the queue is empty.
Now you can scale in two directions:
By running the same job on multiple threads concurrently.
By virtue of the fact this is running on multiple machines, you're effectively load balancing your send work across all your servers.
Because of the locking mechanism, you can guarantee that each email in the queue gets sent only once, even though multiple threads on multiple machines are all running the same code.
In response to comments: There's a few differences in the implementation I ended up with.
First, my ASP application can notify the service that there are new emails in the queue. This means that I don't even have to run on a schedule, I can simply tell the service when to start work. However, this kind of notification mechanism is very difficult to get right in a distributed environment, so simply checking the queue every minute or so should be fine.
The interval you go with really depends on the time sensitivity of your email delivery. If emails need to be delivered ASAP, you might need to trigger every 30 seconds or even less. If it's not so urgent, you can check every 5 minutes. Quartz limits the number of jobs executing at once (configurable), and you can configure what should happen if a trigger is missed, so you don't have to worry about having hundreds of jobs backing up.
Second, I actually grab a lock on 5 emails at a time to reduce query load on the DB server. I deal with high volumes, so this helped efficiency (fewer network roundtrips between the service and the DB). The thing to watch out here is what happens if a node happens to go down (for whatever reason, from an Exception to the machine itself crashing) in the middle of sending a group of emails. You'll end up with "locked" rows in the DB and nothing servicing them. The larger the size of the group, the bigger this risk. Also, an idle node obviously can't work on anything if all remaining emails are locked.
As far as thread safety, I mean it in the general sense. Quartz maintains a thread pool, so you don't have to worry about actually managing the threads themselves.
You do have to be careful about what the code in your job accesses. As a rule of thumb, local variables should be fine. However, if you access anything outside the scope of your function, thread safety is a real concern. For example:
class EmailSender : IJob {
static int counter = 0;
public void Execute(JobExecutionContext context) {
counter++; // BAD!
}
}
This code is not thread-safe because multiple threads may try to access counter at the same time.
Thread A Thread B
Execute()
Execute()
Get counter (0)
Get counter (0)
Increment (1)
Increment (1)
Store value
Store value
counter = 1
counter should be 2, but instead we have an extremely hard to debug race condition. Next time this code runs, it might happen this way:
Thread A Thread B
Execute()
Execute()
Get counter (0)
Increment (1)
Store value
Get counter (1)
Increment (2)
Store value
counter = 2
...and you're left scratching your head why it worked this time.
In your particular case, as long as you create a new database connection in each invocation of Execute and don't access any global data structures, you should be fine.
You'll have to be more specific about your architecture. Where is the email queue; in memory or a database? If they exist on a database, you could have a flag column named "processing" and when a task grabs an email from the queue it only grabs emails that are not currently processing, and sets the processing flag to true for emails it grabs. You then leave concurrency woes to the database.
I have a database table that contains some records to be processed. The table has a flag column that represents the following status values. 1 - ready to be processed, 2- successfully processed, 3- processing failed.
The .net code (repeating process - console/service) will grab a list of records that are ready to be processed, and loop through them and attempt to process them (Not very lengthy), update status based on success or failure.
To have better performance, I want to enable multithreading for this process. I'm thinking to spawn say 6 threads, each threads grabbing a subset.
Obviously I want to avoid having different threads process the same records. I dont want to have a "Being processed" flag in the database to handle the case where the thread crashes leaving the record hanging.
The only way I see doing this is to grab the complete list of available records and assigning a group (maybe ids) to each thread. If an individual thread fails, its unprocessed records will be picked up next time the process runs.
Is there any other alternatives to dividing the groups prior to assigning them to threads?
The most straightforward way to implement this requirement is to use the Task Parallel Library's
Parallel.ForEach (or Parallel.For).
Allow it to manage individual worker threads.
From experience, I would recommend the following:
Have an additional status "Processing"
Have a column in the database that indicates when a record was picked up for processing and a cleanup task / process that runs periodically looking for records that have been "Processing" for far too long (reset the status to "ready for processing).
Even though you don't want it, "being processed" will be essential to crash recovery scenarios (unless you can tolerate the same record being processed twice).
Alternatively
Consider using a transactional queue (MSMQ or Rabbit MQ come to mind). They are optimized for this very problem.
That would be my clear choice, having done both at massive scale.
Optimizing
If it takes a non-trivial amount of time to retrieve data from the database, you can consider a Producer/Consumer pattern, which is quite straightforward to implement with a BlockingCollection. That pattern allows one thread (producer) to populate a queue with DB records to be processed, and multiple other threads (consumers) to process items off of that queue.
A New Alternative
Given that several processing steps touch the record before it is considered complete, have a look at Windows Workflow Foundation as a possible alternative.
I remember doing something like what you described...A thread checks from time to time if there is something new in database that needs to be processed. It will load only the new ids, so if at time x last id read is 1000, at x+1 will read from id 1001.
Everything it reads goes into a thread safe Queue. When items are added to this queue, you notify the working threads (maybe use autoreset events, or spawn threads here). each thread will read from this thread safe queue one item at a time, until the queue is emptied.
You should not assign before the work foreach thread (unless you know that foreach file the process takes the same amount of time). if a thread finishes the work, then it should take the load from the other ones left. using this thread safe queue, you make sure of this.
Here is one approach that does not rely/use an additional database column (but see #4) or mandate an in-process queue. The premise this approach is to "shard" records across workers based on some consistent value, much like a distributed cache.
Here are my assumptions:
Re-processing does not cause unwanted side-effects; at most some work "is wasted".
The number of threads is fixed upon start-up. This is not a requirement, but it does simplify the implementation and allows me to skip transitory details in the simple description below.
There is only one "worker process" (but see #1) controlling the "worker threads". This simplifies dealing with how the records are split between workers.
There is some [immutable] "ID" column which is "well distributed". This is required so search worker gets about the same amount of work.
Work can be done "out of order" as long as it is "eventually done". Also, workers might not always run "at 100%" due to each one effectively working on a different queue.
Assign each thread a unique bucket value from [0, thread_count). If a thread dies/is restarted it will take the same bucket as that which it vacated.
Then, each time a thread needs a new record is needed it will fetch from the database:
SELECT *
FROM record
WHERE state = 'unprocessed'
AND (id % $thread_count) = $bucket
ORDER BY date
There could of course be other assumptions made about reading the "this threads tasks" in batch and storing them locally. A local queue, however, would be per thread (and thus re-loaded upon a new thread startup) and thus it would only deal with records associated for the given bucket.
When the thread is finished processing a record should mark the record as processed using the appropriate isolation level and/or optimistic concurrency and proceed to the next record.
I have developed an application that pulls X amount of records from my database per X amount of threads. Each thread then iterates the collections that are created and does some data validation for each record.
Once a record is validated / not validated by the application it is updated in the database as being valid / not valid. Records are only pulled by each thread if there are items in the database that have not been run through the application. There is a bit column to indicate if the application retrieved the data.
So, potentially, the system can run out of data depending on the number of threads and records per thread. I want the application to continue to check the database for any records that have not be run, then start the process of creating the threads, and finally validating the data.
Here is an example:
There are 50 records in the database
we are running 5 threads with 10 records per thread.
The application runs, the threads are created, the records are pulled and then processed. Now, the system is out of data. A user imports more data into the DB. The application, still looking to see if there are any records, sees that there are 5 new records in the database. It then starts the process all over to create the threads and process the records.
How can I have the system continue to look for data but allow the user to stop the system if need be. I tried using this:
while(RecordsFound <=0){
…sleepcode
} ;
RunProcessMethod
But the winform locks, obviously, during the waiting period. I tried adding the wait logic to another thread, but was afraid that if I run the process method from that thread, via a delegate, things would get weird since I am creating additional threads inside that method.
Thoughts?
The easiest way to fix this is to use a notification mechanism instead of polling. That is once you've spawned off the threads to read data from the data base, make them responsible for notifying the UI when they are complete instead of having the UI wait for them to be complete.
The easiest way to do this is to pass in a delegate for the threads to call when they are complete with the set of records found. The UI can then update when the records are available
delegate void CallBack(List<Data> found);
void OnDataFound(List<Data> found) {
// Get back on the UI thread
if ( this.InvokeRequired ) {
this.Invoke( new CallBack(OnDataFound), new object[] { found } );
return;
}
// Update display
}
I tried adding the wait logic to another thread, but was afraid that if I run the process method from that thread, via a delegate, things would get weird since I am creating additional threads inside that method. Thoughts?
You don't need to fear this. It's the proper way to handle this type of scenario. There is no problem with a background thread creating additional threads.