I'm developing a system which works like following structure
Level1-sub-tasks are created by task via TaskFactory of .net 4.0, and Level2-sub-tasks are created by level1-sub-tasks similarly.
All tasks work file in debug, but when I deploy that on IIS server, all level2 sub tasks stops executing.
I'm not an experienced developer of iis, does it has any limitation for multi-level thread(task)?
If you do spawn new threads while handling a request, make sure they have all finished before returning a response. You may think "I'm gonna return to the user asap, and leave a thread in the background saving stuff to the database." This is dangerous, mainly because the AppDomain may be recycled, aborting your background threads.
Alternatively, use the IRegisteredObject interface to tell asp.net you're doing some work in the background. The approach is detailed here: http://haacked.com/archive/2011/10/16/the-dangers-of-implementing-recurring-background-tasks-in-asp-net.aspx/
Do note that this is not fail-proof though, and that there are better approaches to this - it all depends on what you're trying to achieve. In most cases (e.g., sending out email notifications), the best solution is to schedule a task (in a separate process) that dequeues work items periodically.
Related
I have a fairly long-running process (several minutes) that I need to run roughly once a month - but not on a fixed schedule, but after a user clicks Go in a ASP.NET Webforms GUI page.
Since ASP.NET really isn't designed to handle long-running background tasks, my idea was to put this into a separate console app. But how to launch that as needed?
What happens if I use Process.Start(....) from my ASP.NET page's code-behind? I would like to avoid blocking the whole Web UI for 20 minutes or so... and also: even if it doesn't block my UI, what happens to my long-running task if the ASP.NET app pool recycles while it's still running?
Another idea was to have a frequently running job (runs every 2 minutes) check for some kind of a flag (e.g. existence of some database entries), and if needed, that job would then launch the long-running task.
But the same question: if I launch my 20-minute task from a job using Process.Start() - does that block the caller?
It seems like a bit of overkill to schedule that long running tasks five times a day since it typically is run only once a month - but at the same time, the user expects to have his results within a reasonable amount of time (less than 1 hour, if ever possible) after scheduling the process - so I cannot really just schedule it to run once at night either ...
Hangfire is what you are looking for. Best part is it comes with a built in dashboard.
You might have to write some logic on the top of it.
You can find it here.
http://hangfire.io/
First off - for several reasons - ASP.NET is imho not the solution for long-running tasks/jobs/... whatsoever.
I have had this requirement a lot of times, and always solved/separated it like:
Worker
A service with
Quartz.net (for scheduling and processing, even if you don't have a specific timestamp for execution - but the overall handling in this framework is simply superb)
a persistent job-store (to handle start/stop and completed/aborted/paused jobs)
eg ServiceStack as the interop between the two processes
Website
Simply calls some webservice-methods of the worker to enqueue/query/pause/stop/... a job. For querying jobs a call to a unified job-store might be an option (eg. db)
It might be a bit of an overkill for you though ... but this is my Swiss army knife for such scenarios.
Use the standard built-in Windows Task Scheduler like you have done, but invoke it from your web application.
Configure your task in Task Scheduler. It does not need to have a scheduled trigger. From your web application, just use Process.Start to kick it off:
SchTasks.exe /Run /TN Folder\Taskname
I have not used SchTasks.exe directly, but have used the Microsoft.Win32.TaskScheduler wrapper classes.
An ASP.NET 3.5 webapp has to start several tasks that takes hours to complete. For obvious reasons the pages which starts these tasks cannot wait for them to finish nor will anyone want to wait that long to get a response, so the tasks must be asynchronous.
There is a Helper class to handle all of these long running tasks. The main method that schedules and executes these tasks is currently the following:
public static bool ScheduleTask(TaskDescriptor task, Action action)
{
bool notAlreadyRunning = TasksAsync.TryAdd(task);
if (notAlreadyRunning)
{
Thread worker = null;
worker = new Thread(() =>
{
try { action(); }
catch(Exception e)
{
Log.LogException(e, "Worker");
}
TasksAsync.RemoveTask(task);
workers.Remove(worker);
});
workers.Add(worker);
worker.Start();
}
return notAlreadyRunning;
}
On earlier implementations we've used the ThreadPool.QueueUserWorkItem approach but the result has always been the same: after aprox. 20-30 mins a Thread was being aborted exception is thrown.
Does anyone know why is this happening? or how can it be prevented?
More Info:
IIS standard configuration.
Tasks could be anything, querys to a database and/or IO operations etc.
UPDATE: Decisions
Thank you all for your responses. Now I don´t know which question to mark as answer. All of them are valid and are possible solutions to this problem. Will wait for today and mark as answer the answer with the most up votes, in case of a draw I will choose the first shown answer, typically they are ordered by most relevance.
For anyone who want´s to know the solution I choose, again due to time restrictions, was to change the IIS recycling configuration, But what I consider to be the ideal solution, based on my research and of course the answers below, is to create a "Worker Service" and use a communication solution between the ASP.NET App and the new "Worker Service" to coordinate the long running work to be done.
You can start the long-running process in its own application domain.
In the past, when I've needed this capability, I create a Windows Service for this purpose. If you use WCF to connect to it, it doesn't even have to run on the IIS machine at all; you can run it on any machine on the network.
Chances are you can get this working, by upping the timeout, using a different app pool or a variety of other hacks, but your best bet is going to be to decouple the long running task from the ui and asp.net completely, and use either a service (wouldn't recommend it) or a scheduled task that polls for work to do; personally I would use something like aws sqs/sns to keep track of work to be done and a scheduled task in windows server that checks for things todo at whatever frequency make sense. The only thing the ui/asp.net then needs to do is log that fact that something needs to be done, not actually do it.
Another benefit of this message based approach is should the long running process become so long running, or so overworked, you'd have the opportunity to add more worker tasks or servers to complete those requests.
Perhaps more than you can implement for your immediate problem, but something to consider for a better long term solution.
Before I go into this question, I d like to say that, I have read the threading modeling for IIS 7, 7, 7.5 so I know how threads are handled.
My application starts a thread when a request comes in.
We can assume the threads as cron jobs.
GET request comes in, Lets say /Handle
in the scope of /Handle I start a thread from that action , THREAD A
I am not long polling the GET request, so it returns back to the
user right away. So thread handling the GET is returned to the POOL
Then I wait until the thread A completes to do anything else.
So No threads are running as far as I know. Both the thread that was
handling the GET and THREAD A has exited.
I make the same request a few times SEQUANTIALLY. I always wait for both threads to exit.
After a while `Thread.Start()1 function blocks.
Questions :
I know that the threads are returning and I am not leaking any ghost threads.
Why does IIS not allowing me to start new threads after a like 4-5 requests. ?
What is the right way to create application thread for the user application.
If I said Thread t= new Thread(), does this allocate a thread from the pool that handled the GETS or CLR?
I am using IIS7.
I know that I exit each thread, I call a JOIN on THREAD A , and it never blocks, and at this point I am not worried about scalability so I always have ONE user hitting the server sequentially.
So to answer your question "What is the right way to create application thread for the user application?" (i.e. ASP.NET application) - You have many options:
run on the ASP.NET thread, without any threading - ASP.NET will still handle more then one request
use async calls (see async operations) for long running operations
use CLR ThreadPool
send a message to some other server (e.g. using WCF services), so the long running processing takes place outside the Web server.
You mentioned reading about threading in ASP.NET, but in "MSDN: Performing Asynchronous Work, or Tasks, in ASP.NET Applications" there's a relatively short description of how threading in ASP.NET works. At the end of the post, there's a question:
"Q4: Should I create my own threads (new Thread)?" and the answer for that question is "A4) Please don’t (create new Threads). Or to put it a different way, no!!! (...) ".
And to answer your question: "Why does IIS not allowing me to start new threads after a like 4-5 requests"?
That's really a strange behaviour, maybe IIS knows that your are doing it wrong ;)
If I create a new thread on an ASP.NET page the IsThreadPoolThread property is true.
First question is, is it from ASP.NET pool or CLR pool ?
Second question is, if it is from ASP.NET pool then how to create a thread from CLR and don't use ASP.NET pool ?
I need a synchronous solution for long-running requests (full story).
First off, there is no difference between the ASP.NET thread pool and the CLR thread pool. ASP.NET processes pages on the CLR thread pool, so your ASP.NET pages will always have IsThreadPoolThread == true.
I'm curious as to how you are creating your thread. Are you using the System.Threading.Thread constructor, or are you using ThreadPool.QueueUserWorkItem? If you are using ThreadPool.QueueUserWorkItem, then the threads that you are getting are coming from the regular .net thread pool.
Finally, as I have posted before, it is always a bad idea to attempt long running tasks from within ASP.NET. My general suggestion is to use a background windows service to process these requests, since ASP.NET may terminate your background thread at any point. More details here, if you must do it in IIS: http://csharpfeeds.com/post/5415/Dont_use_the_ThreadPool_in_ASP.NET.aspx
While there is a point to be made about minimizing impact on transactions and catering for the unforeseen in distributed transactions, in this example its really not necessary to reinvent IIS just because the process is long running. The whole "IIS can die at any moment" meme is IMHO greatly exaggerated.
Yes, you can manually restart IIS or app pool but so can you restart any other service which does the same job for the same effect. As for auto-recycle IIS uses overlapping worker processes and will never terminate a started thread forcefully (unless time out occurs). If that were the case we would have serious problems with any hosted application (what's to stop IIS from killing a fast response thread 0.001ms after start)
In essence, let the IIS do what IIS does best and don't insist on sync operation, you'll just waste pool's thread waiting for blocking I/O which is what I believe you are trying to avoid. You already made a good choice by going to Asynchronous Handlers (ASHX), use IHttpAsyncHandler implementation to spawn your custom thread which will then block to your heart's desire without affecting the web application and its pool. Once you initiate an async operation thread the asp.net thread will go back to its own pool and will be ready to start serving a new request. With a default limit of 100 threads in a pool and given that your process is low-on-cpu-high-on-bandwidth I doub't you'll ever run out of pool threads before you run out of pipe space :). For more information on how to build async handler check this link (its an old article but valid non the less):
Use Threads and Build Asynchronous Handlers in Your Server-Side Web Code
There is in fact a difference in threads in ASP .NET: Worker threads and IO threads (system threads which i believe you state as CLR thread).
Now ASP .NET use a worker thread on each request, unless custom configurations are used, and these worker threads are limited on your CPU number; this configuration can be set in IIS.
When you start an asynchronous task inside ASP.NET by using delegate for instance, you are using another worker thread. Delegates are a quick and dirty way of starting something asynchronous in .NET :)
If you want to start a new thread which does NOT use up a worker thread, then you must explicitly start a new thread like: new Thread()....etc. Now this comes with alot code management, and does not follow the event based asynchronous pattern.
However there is one way you can start asynchronous threads safely and that is by using .NETs own asynch methods on objects. Things you normally would use asynch for like SQL commands, webservice calls, etc. All these have a BEGIN and an END method.
By using these methods you will never use a worker thread, but an IO thread.
ASP .NET has a few tricks up its sleeve when it comes to asynchronous pages.
There are two alternatives:
Asynchronous page: which lets your page cycle to be asyncronous. This basicly means that the page is requested asynchronous.
Asynchronous page tasks: which lets you define tasks that will fire asynchronous when the page starts. Its kinda like Asynch threads, just that ASP .NET does alot of things for you, and its more restricted.
I dont have all the details, but please look into the two options on MSDN Library.
Here is an article on the subject: asynch programmin
Sometimes there is a lot that needs to be done when a given Action is called. Many times, there is more that needs to be done than what needs to be done to generate the next HTML for the user. In order to make the user have a faster experience, I want to only do what I need to do to get them their next view and send it off, but still do more things afterwards. How can I do this, multi-threading? Would I then need to worry about making sure different threads don't step on each others feet? Is there any built in functionality for this type of thing in ASP.NET MVC?
As others have mentioned, you can use a spawned thread to do this. I would take care to consider the 'criticality' of several edge cases:
If your background task encounters an error, and fails to do what the user expected to be done, do you have a mechanism of report this failure to the user?
Depending on how 'business critical' the various tasks are, using a robust/resilient message queue to store 'background tasks to be processed' will help protected against a scenario where the user requests some action, and the server responsible crashes, or is taken offline, or IIS service is restarted, etc. and the background thread never completes.
Just food for though on other issues you might need to address.
How can I do this, multi-threading?
Yes!
Would I then need to worry about making sure different threads don't step on each others feet?
This is something you need to take care of anyway, since two different ASP.NET request could arrive at the same time (from different clients) and be handled in two different worker threads simultaneously. So, any code accessing shared data needs to be coded in a thread-safe way anyway, even without your new feature.
Is there any built in functionality for this type of thing in ASP.NET MVC?
The standard .net multi-threading techniques should work just fine here (manually starting threads, or using the Task features, or using the Async CTP, ...).
It depends on what you want to do, and how reliable you need it to be. If the operaitons pending after the response was sent are OK to be lost, then .Net Async calls, ThreadPool or new Thread are all going to work just fine. If the process crashes the pending work is lost, but you already accepted that this can happen.
If the work requires any reliable guarantee, for instance the work incurs updates in the site database, then you cannot use the .Net process threading, you need to persist the request to do the work and then process this work even after a process restart (app-pool recycle as IIS so friendly calls them).
One way to do this is to use MSMQ. Other way is to use the a database table as a queue. The most reliable way is to use the database activation mechanisms, as described in Asynchronous procedure execution.
You can start a background task, then return from the action. This example is using the task Parallel Library, found in .NET 4.0:
public ActionResult DoSomething()
{
Task t = new Task(()=>DoSomethingAsynchronously());
t.Start();
return View();
}
I would use MSMQ for this kind of work. Rather than spawning threads in an ASP.NET application, I'd use an Asynchronous out of process way to do this. It's very simple and very clean.
In fact I've been using MSMQ in ASP.NET applications for a very long time and have never had any issues with this approach. Further, having a different process (that is an executable in a different app domain) do the long running work is an ideal way to handle it since your web application is no being used to do this work. So IIS, the threadpool and your web application can continue to do what they need to, while other processes handle long running tasks.
Maybe you should give it a try: Using an Asynchronous Controller in ASP.NET MVC