after doing some research it seems that AppDomains are not really a tool for building a hosting server. From my understanding, the hosting server will still crash if there is an unhandled exception in a created AppDomain (if the exception is thrown from a thread in the created AppDomain). So in that case if the hosting server hosts a service which leaks exceptions this will bring down the default AppDomain as well.
So I guess from a server architecture point-of-view there is nothing better than creating child processes and monitoring them.
Is that correct or am I missing something with AppDomains?
thanks,
Christoph
If you can control the threads created in the other AppDomain, you can also handle exceptions by using catch-all blocks in the thread main method.
Other than that, as long as you use the default host, I believe that your assumption is correct. However, if you host the runtime yourself, you can also handle unhandled exceptions.
From a forum post on the topic:
Well, it is possible. You'd have to
create your own CLR host. That starts
with ICorBindToRuntimeEx(). You get
to have full control of AppDomains
that throw exceptions. And it's being
used by MSFT software like ASP.NET and
SQL Server 2005. When you write a
service, you are working with the
default CLR host implementation and it
terminates the process when any
unhandled exception is raised,
regardless of what AppDomain caused
the exception.
Problem is, hosts like ASP.NET and SQL
server have a very well defined code
execution path. In a web server,
managed code runs because of a page
request. In a dbase server, it runs
because of a query. When something
bad happens, they have the luxury of
simply aborting everything that the
request started (killing the
AppDomain) and returning a "sorry,
couldn't do it" status back to the
client. You might have seen it,
crashing the forums server on the old
web site was pretty trivial but didn't
stop it from serving other requests.
Not actually 100% sure about that.
Your service implementation is
probably not nearly as clean. I can't
tell, you didn't say anything about
it. It general, there's a problem
with aborting a thread. You always
have to abort a thread when there's an
unhandled exception. A service
typically has one thread, started by
the OnStart() method. Aborting it
kills the server until somebody stops
and starts it again.
You can definitely make it more
resilient than that, you could start a
"master" thread that launches child
threads in response to external events
that makes your service do its job.
Having a child thread terminated
because of an unhandled exception is
something you could possibly recover
from. But then, if you make that next
step, why not have the child thread
catch an exception and pass it back to
the master thread so it can make an
intelligent decision about what to do
next.
The cold hard fact of the default CLR
host is: if you are not willing to
deal with failure, it is not going to
do the job for you. And it shouldn't,
the .NET 1.x behavior to threads that
died with exceptions was a major
mistake that got corrected in .NET
2.0.
You know what to do: handle failure.
Or write you own host. Or accept that
things could be beyond your control
and log a good error message so you
can tell your customer what to do.
I'd strongly recommend the latter.
Related
Is there a way to fire an Http call to an external web API within my own web API without having to wait for results?
The scenario I have is that I really don't care whether or not the call succeeds and I don't need the results of that query.
I'm currently doing something like this within one of my web API methods:
var client = new HttpClient() { BaseAddress = someOtherApiAddress };
client.PostAsync("DoSomething", null);
I cannot put this piece of code within a using statement because the call doesn't go through in that case. I also don't want to call .Result() on the task because I don't want to wait for the query to finish.
I'm trying to understand the implications of doing something like this. I read all over that this is really dangerous, but I'm not sure why. What happens for example when my initial query ends. Will IIS dispose the thread and the client object, and can this cause problems at the other end of the query?
Is there a way to fire an Http call to an external web API within my own web API without having to wait for results?
Yes. It's called fire and forget. However, it seems like you already have discovered it.
I'm trying to understand the implications of doing something like this
In one of the links in the answers you linked above state the three risks:
An unhandled exception in a thread not associated with a request will take down the process. This occurs even if you have a handler setup via the Application_Error method.
This means that any exception thrown in your application or in the receiving application won't be caught (There are methods to get past this)
If you run your site in a Web Farm, you could end up with multiple instances of your app that all attempt to run the same task at the same time. A little more challenging to deal with than the first item, but still not too hard. One typical approach is to use a resource common to all the servers, such as the database, as a synchronization mechanism to coordinate tasks.
You could have multiple fire-and forget calls when you mean to have just one.
The AppDomain your site runs in can go down for a number of reasons and take down your background task with it. This could corrupt data if it happens in the middle of your code execution.
Here is the danger. Should your AppDomain go down, it may corrupt the data that is being sent to the other API causing strange behavior at the other end.
I'm trying to understand the implications of doing something like
this. I read all over that this is really dangerous
Dangerous is relative. If you execute something that you don't care at all if it completes or not, then you shouldn't care at all if IIS decides to recycle your app while it's executing either, should you? The thing you'll need to keep in mind is that offloading work without registration might also cause the entire process to terminate.
Will IIS dispose the thread and the client object?
IIS can recycle the AppDomain, causing your thread to abnormally abort. Will it do so depends on many factors, such as how recycling is defined in your IIS, and if you're doing any other operations which may cause a recycle.
In many off his posts, Stephan Cleary tries to convey the point that offloading work without registering it with ASP.NET is dangerous and may cause undesirable side effects, for all the reason you've read. That's also why there are libraries such as AspNetBackgroundTasks or using Hangfire for that matter.
The thing you should most worry about is a thread which isn't associated with a request can cause your entire process to terminate:
An unhandled exception in a thread not associated with a request will
take down the process. This occurs even if you have a handler setup
via the Application_Error method.
Yes, there are a few ways to fire-and-forget a "task" or piece of work without needing confirmation. I've used Hangfire and it has worked well for me.
The dangers, from what I understand, are that an exception in a fire-and-forget thread could bring down your entire IIS process.
See this excellent link about it.
I have several long-running threads in an MVC3 application that are meant to run forever.
I'm running into a problem where a ThreadAbortException is being called by some other code (not mine) and I need to recover from this gracefully and restart the thread. Right now, our only recourse is to recycle the worker process for the appDomain, which is far from ideal.
Here's some details about this code works:
A singleton service class exists for this MVC3 application. It has to be a singleton because it caches data. This service is responsible for making request to a database. A 3rd party library is used for the actual database connection code.
In this singleton class we use a collection of classes that are called "QueryRequestors". These classes identify unique package+stored_procedure names for requests to the database, so that we can queue those calls. That is the purpose of the QueryRequestor class: to make sure calls to the same package+stored_procedure (although they may have infinite different parameters) are queued, and do not happen simultaneously. This eases our database strain considerably and improves performance.
The QueryRequestor class uses an internal BlockingCollection and an internal Task (thread) to monitor its queue (blocking collection). When a request comes into the singleton service, it finds the correct QueryRequestor class via the package+stored_procedure name, and it hands the query over to that class. The query gets put in the queue (blocking collection). The QueryRequestor's Task sees there's a request in the queue and makes a call to the database (now the 3rd party library is involved). When the results come back they are cached in the singleton service. The Task continues processing requests until the blocking collection is empty, and then it waits.
Once a QueryRequestor is created and up and running, we never want it to die. Requests come in to this service 24/7 every few minutes. If the cache in the service has data, we use it. When data is stale, the very next request gets queued (and subsequent simultaneous requests continue to use the cache, because they know someone (another thread) is already making a queued request, and this is efficient).
So the issue here is what to do when the Task inside a QueryRequestor class encounters a ThreadAbortException. Ideally I'd like to recover from that and restart the thread. Or, at the very least, dispose of the QueryRequestor (it's in a "broken" state now as far as I'm concerned) and start over. Because the next request that matches the package+stored_procedure name will create a new QueryRequestor if one is not present in the service.
I suspect the thread is being killed by the 3rd party library, but I can't be certain. All I know is that nowhere do I abort or attempt to kill the thread/task. I want it to run forever. But clearly we have to have code in place for this exception. It's very annoying when the service bombs because a thread has been aborted.
What is the best way to handle this? How can we handle this gracefully?
You can stop re-throwing of ThreadAbortException by calling Thread.ResetAbort.
Note that most common case of the exception is Redirect call, and canceling thread abort may case undesired effects of execution of request code that otherwise would be ignored due to killing the thread. It is common issue in WinForms (where separation of code and rendering is less clear) than in MVC (where you can return special redirect results from controllers).
Here's what I came up with for a solution, and it works quite nicely.
The real issue here isn't preventing the ThreadAbortException, because you can't prevent it anyway, and we don't want to prevent it. It's actually a good thing if we get an error report telling us this happened. We just don't want our app coming down because of it.
So, what we really needed was a graceful way to handle this Exception without bringing down the application.
The solution I came up with was to create a bool flag property on the QueryRequestor class called "IsValid". This property is set to true in the constructor of the class.
In the DoWork() call that is run on the separate thread in the QueryRequestor class, we catch the ThreadAbortException and we set this flag to FALSE. Now we can tell other code that this class is in an Invalid (broken) state and not to use it.
So now, the singleton service that makes use of this QueryRequestor class knows to check for this IsValid property. If it's not valid, it replaces the QueryRequestor with a new one, and life moves on. The application doesn't crash and the broken QueryRequestor is thrown away, replaced with a new version that can do the job.
In testing, this worked quite well. I would intentionally call Thread.Abort() on the DoWork() thread, and watch the Debug window for output lines. The app would report that the thread had been aborted, and then the singleton service was correctly replacing the QueryRequestor. The replacement was then able to successfully handle the request.
I have a WPF monitoring application that uses a separate (internally developed) C# test
infrastructure to execute tests and monitor and log the results. I also uses a commercial package (InGear) to communicate to a PLC. As a result, the application has LOTS of threads (most of which are created by the tools I am using).
Because of the nature of the environment, it would be very difficult to use a debugger in the target environment; so, we are both using log4net to log diagnostics.
I use try/catch blocks in around my external calls and also have setup a unhandled exception handlers both at the WPF and AppDomain levels.
During our first long run the application appears to have become non-responsive and I got the standard "not responding" dialog. Looking at the log it seems like everything just stopped. Ex: I can see from the log that a DispatcherTimer was set to respond on the main thread in 1 sec; but, never did.
So.... My questions are:
How can I detect the hang or is hook into Window's detection that I am hung? Note that I am assuming that it could be a higher priority thread that is blocking my UI tread; so, I probably can't respond to a Windows Message.
Once I do tap in, how do I find out what thread is the culprit. Being able the log its call stack would be a big plus.
Maybe simplistic, but what about attaching the debugger to the process, doing a 'Break All' and then inspect the stack trace of the various threads?
Where I was unable to determine a way to detect the 'hang' before Windows does, I was able to catch the Windows timeout exception and ultimately traced the problem to unmanaged code in the Oracle .NET component.
Client application initiates a process on a server (via RIA, but the implementationis not important). When I say process, I mean business code running and not referring to an actual process running on the CPU.
Code is C#.
The client then checks to see the process status. Failed, completed, still running.
The basics are relatively easy to implement. I store a process Id statically on the server in which the client can poll the server regularily to check the process which will check the status associated with the process id.
Edge cases around this require a bit more work. Fatal and catastrophic reasons that abnormally abort the thread (process) without allowing the code to handle the exception and gracefully set the status associated with the process to fail. In this scenario, the client will continue to assume the process is still in progress.
I was thinking of running the process in a separate thread andf tracking the thread id. When the client calls the server to check process state, we could check the IsAlive property of the thread running the process.
I am wondering if there are any scenarios where this could be problematic? Maybe there could be a chance that IsAlive would return True although the thread is hung.
Another approach would be to have the process on the server periodically set a timestamp which could be used when the client checks the state. The code checking the state, could see how old the timestamp is and then based on whatever interval we chose (let's say 2 minutes) it could decide if the process is still running (2 minutes hasn't passed since last timestamp written), or process has timedout without exception (more than 2 minutes has passed since the thread wrote the last timestamp). All timestamps would be done in memory.
Are there any best practices surrounding this that could be beneficial? Does anyone have any special insight or tips on how to best approach this? I am also open to other scenarios or ideas people have?
I am sure the better approach is to have a fully controled server app which handles everything including exceptions for each business process and maintains state of all. and the even better if client receives StateChanged event rather than polling (with WCF duplex channels for example). what you do is 1999 basically. .NET gives you all the stuff almost for free. correct architecture actually faster to write and in time cheaper to support.
This is something of a sibling question to this programmers question.
Briefly, we're looking at pushing some work that's been piggy-backing on user requests into the background "properly." The linked question has given me plenty of ideas should we go the service route, but hasn't really provided any convincing arguments as to why, exactly, we should.
I will admit that, to me, the ability to do the moral equivalent of
WorkQueue.Push(delegate(object context) { ... });
is really compelling, so if its just a little difficult (rather than inherently unworkable) I'm inclined to go with the background thread approach.
So, the problems with background threads I'm aware of (in the context of an AppPool):
They can die at any time due to the AppPool being recycled
Solution: track when a task is being executed, so it can be re-run* should a new thread be needed
The ThreadPool is used to respond to incoming HTTP queries, so using it can starve IIS
Solution: build our own thread pool, capping the number of threads as well.
My question is, what am I missing, if anything? What else can go wrongǂ with background threads in ASP.NET?
* The task in questions are already safe to re-run, so this isn't a problem.
ǂ Assume we're not doing anything really dumb, like throwing exceptions in background threads.
I would stay away from launching threads from with-in your IIS AppDomain for StackOverflow. I don't have any hard evidence to support what I am going to say, but working with IIS for 10 years, I know that it works best when it is the only game in town.
There is also an alternative, I know this is going to be sort of a take off on my answer over on the programmers thread. But as I understand it you already have a solution that works by piggy-backing the work on user requests. Why not use that code, but only launch it when a special internal API is called. Then use Task Scheduler to call a CURL command that calls that API every 30 seconds or so to launch the tasks. This way you are letting IIS handle the threading and your code is handling something that it already does easily.
One danger I ran into personally is the CallContext. We were using the CallContext to set user identity data, because the same code was shared across our web application and our .NET Remoting based application services (which is designed to use the CallContext for storing call specific data) - so we weren't using the HttpContext.
We noticed that, sometimes, a new request would end up with a non-null identity in the CallContext. In other words, ASP .NET was not nulling out the data stored in the CallContext between requests...and thus an unauthenticated user might get into the application if they picked up a thread which still had the CallContext containing validated user identity info.
Let me tell you about a non-obvious danger :)
I used threads to collect Update some RSS feeds into my database for a website I was hosting with GoDaddy. The threads worked fine (if they were terminated, they would be restarted automatically due to some checks I had built in some web pages).
It was working excellently and I was very happy, until GoDaddy (my host then) first started killing the threads, and then blocked them completely. So my app just died!
If that wasn't non-obvious, what is?
One could be are you overly complicating your architecture without getting any benefits.
You program will be more expensive to write, more expensive to maintain and have a greater chance of having bugs.