I have an asp.net website with a background service running that sends queued emails every minute. The problem is we moved to a web garden (multiple worker processes) and the emails get sent multiple times (at the very same time or close to it) because multiple threads are invoking that background process.
Is there a way to assign one thread or have only one thread run this process every minute and tell the other ones to ignore it?
Thanks!
I had to use mutex locking to prevent this
Related
I am developing a Windows Service application, in .NET, which executes many functions (it is a WCF service host), and one of the targets is running scheduled tasks.
I chose to create a System.Threading.Timer for every operation, with a dueTime set to the next execution and no period to avoid reentrancy.
Every time the operation ends, it changes the dueTime to match the next scheduled execution.
Most of the operations are scheduled to run every minute, not all toghether but delayed by some seconds each other.
Now, after adding a number of operations, about 30, it seems that the timers start to be inaccurate, starting the operations many seconds late, or even minutes late.
I am running the operation logic directly in the callback method of the timer, so the running thread should be the same as the timer.
Should I create a Task to run the operation instead of running it in the callback method to improve accuracy?
Or should I use a single timer with a fixed (1 second) dueTime to check which operations need to be started?
I don't like this last option because it would be more difficult to handle reentrancy..
Timers fire on a thread pool thread, so you are probably finding that as you add lots of timers that you are exhausting the thread pool.
You could increase the size of the thread pool, or alternatively ensure you have fewer timers than the thread pool size.
Firing off Tasks from the callback likely won't help - since you are going to be fighting for threads from the same thread pool. Unless you use long-running tasks.
We usually setup multiple timers to handle different actions within a single service. We set the intervals and start, stop the timer on the Service Start/Stop/Shutdown events (and have a variable indicating the status for each one, i.e. bool Stopped)
When the timer ticks over, we stop the timer, run the processing (which may take a while depending on the process, i.e. may take longer than the interval if its short.. (this code needs to be in a try--catch so it keeps going on errors)
After the code has processed, we check the Stopped variable and if its not stopped we start the timer again (this handles the reentrancy that you've mentioned and allows the code to stick to the interval as much as possible)
Timers are generally more accurate after about 100ms as far as I know, but should be close enough for what you want to do.
We have run this concept for years, and it hasn't let us down.
If you running these tasks as a sub-system of an ASP.NET app, you should also look at HangFire, which can handle background processing, eliminating the need for the windows service.
How accurate do the timers need to be? you could always use a single timer and run multiple processing threads at the same time? or queue the calls to some operations if less critical.
Ok, I came to a decision: since I am not able to easily reproduce the behavior, I chose to solve the root problem and use the Service process to only:
serve WCF requests done by clients
schedule operations (which was problematic)
Every operation that could eat CPU is executed by another process, which is controlled directly by the main process (with System.Diagnostics.Process and its events) and communicates with it through WCF.
When I start the secondary process, I pass to it the PID of the main process through command line. If the latter gets killed, the Process.Exited event fires, and I can close the child process too.
This way the main service usually doesn't use much CPU time, and is free to schedule happily without delays.
Thanks to all who gave me some advices!
I have an application pool that has multiple worker processes (e.g 4). Is there somehow any way to access the website, such that you know you are always on worker process #1, or #2, etc?
I need to do specific tasks like clearing memory cache on a particular worker process, and since you don't know which one you are on, I'm finding it a problem to clear the cache for all the worker processes.
Make a request loop until System.Diagnostics.Process.GetCurrentProcess().Id returns the process you want to work with.
I have an ASP.Net application acting as an interface between systems which tend to send data in bursts via multiple requests and I would like to kick off a background task to perform some processing but would really like to only have one single background task do it. I can use HostingEnvironment.QueueBackgroundWorkItem but that will indiscriminately launch a thread for each incoming request which is a problem.
When I launch a background process I want it to queue up the work it has (in the connected database) and then check if there is another background process running. If there is then it should finish because the other process will process the queued work. If there is no other background process running then I want it to start processing the queued work until there is no more to do and then it will stop.
The process is not a heavy task or long running task but the main constraint is that everything is processed in a strict order making parallel threading risky. In a single process it's easy to ensure everything gets processed in order.
How do I achieve this without shifting to an external service?
Seems like a classic producer consumer scenario. Create a BlockingCollection that producers enqueue to. Create one permanent LongRunning Task that drains that collection.
You can drain in batches if you want.
This would not work with QueueBackgroundWorkItem because the you need to eventually exit the work that you put into QueueBackgroundWorkItem so that the worker process can shutdown gracefully.
My application starts multiple long-running functions/methods on their own threads. When one of these threads encounters an error, my program crashes (as expected). My question, though, is do all the threads spawned by the application stop executing? or do they hang around somewhere? If so, do I need to write another application to monitor them so I can kill them?
Thank you!
When a process finishes all threads contained in that process are terminated. A thread cannot exist outside of a process. Application crash = process termination.
If you are spawning processes from threads the processes you spawn will continue to run after the spawning process ends. You can kill the process tree in task manager and in code, but by default the spawned processes will keep running if your application ends ungracefully.
I'm assuming this is what you mean when you said
My application starts multiple long-running processes on their own threads
But maybe I'm misunderstanding what you are meaning to say
If the process which created all threads is killed/aborted, threads are aborted by the OS automatically. In other words: running process = at least one running thread, killed process = all threads terminated.
I'm making a thread that will trigger a change of files in App_GlobalResources, which again will cause the application to recompile, which will abort all other possible running threads.
I need to make this thread that will abort the other threads to wait until all other threads are not running. I have registered all threads in (Hashtable)Application["ThreadList"], but I don't seem to be able to access Application from Threads. Any ideas?
As far as I know ASP.NET doesn't abort currently running worker threads before it recompiles. It starts queuing up requests and waits for the existing requests to finish processing before it restarts AppDomain.
Update:
If your resources change daily than they shouldn't be hard-coded in resx files. Create a new resource provider that loads them from database or external files.