As part of a large automation process, we are calling a third-party API that does some work calling services on another machine. We discovered recently that every so often when the other machine is unavailable, the API call will spin away sometimes up to 40 minutes while attempting to connect to the remote server.
The API we're using doesn't offer a way to specify a timeout and we don't want our program waiting around for that long, so I thought threads would be a nice way to enforce the timeout. The resulting code looks something like:
Thread _thread = new Thread(_caller.CallServices());
_thread.Start();
_thread.Join(timeout);
if (_thread.IsAlive)
{
_thread.Abort();
throw new Exception("Timed-out attempting to connect.");
}
Basically, I want to let APICall() run, but if it is still going after timeout has elapsed, assume it is going to fail, kill it and move on.
Since I'm new to threading in C# and on the .net runtime I thought I'd ask two related questions:
Is there a better/more appropriate mechanism in the .net libraries for what I'm trying to do, and have I committed any threading gotchas in that bit of code?
Thread.Abort() is a request for the thread to abort, and gives no guarantee that it will do so in a timely manner. It is also considered bad practice (it will throw a thread abort exception in the aborted thread, but it seems like the 3rd party API offers you no other choices.
If you know (programmatically) the address of the remote service host you should ping it before you transfer control to the 3rd party API.
If not using a backgroundworker, you could set the thread's IsBackgroundThread to true, so it doesn't keep your program from terminating.
Bad idea. Thread.Abort doesn't necessarily clean up the mess left by such an interrupted API call.
If the call is expensive, consider writing a separate .exe that makes the call, and pass the arguments to/from it using the command line or temporary files. You can kill an .exe much more safely than killing a thread.
You can also just use a delegate... Create a delegate for the method that does the work, Then call BeginInvoke on the delegate, passing it the arguments, and a callback function to handle the return values (if you want)...
Immediately after the BeginInvoke you can wait a designated time for the asynch delegate to finish, and if it does not in that specified time, move on...
public delegate [ReturnType] CallerServiceDelegate
([parameter list for_caller.CallService]);
CallerServiceDelegate callSvcDel = _caller.CallService;
DateTime cutoffDate = DateTime.Now.AddSeconds(timeoutSeconds);
IAsyncResult aR = callSvcDel.BeginInvoke([here put parameters],
AsynchCallback, null);
while (!aR.IsCompleted && DateTime.Now < cutoffDate)
Thread.Sleep(500);
if (aR.IsCompleted)
{
ReturnType returnValue = callSvcDel.EndInvoke(aR);
// whatever else you need to do to handle success
}
else
{
callSvcDel.EndInvoke(aR);
// whatever you need to do to handle timeout
}
NOTE: as written AsynchCallback could be null, as the code retrieves the return value from the EndInvoke(), but if you want to you can have the CallService() method call the AsynchCallback delegate and pass it the return values instaed...
It might work, but nobody could say for sure without an understanding of the third-party API. Aborting the thread like that could leave the component in some invalid state that it might not be able to recover from, or maybe it won't free resources that it allocated (think - what if one of your routines just stopped executing half-way through. Could you make any guarantees about the state your program would be in?).
As Cicil suggested, it might be a good idea to ping the server first.
Does your application run for long periods of time or is it more of a run-as-needed application? If it's the latter, I personally would consider using the Thread.Abort() option. While it may not be the most desirable from a purist's perspective (resource management, etc.), it is certainly straightforward to implement and may foot the bill given the way your particular application works.
The idea of a separate executable makes sense. Perhaps another option would be to use AppDomains. I'm not an expert in this area (I welcome refinements/corrections to this), but as I understand it, you'd put the API call in a separate DLL and load it into a separate AppDomain. When the API call is finished or you have to abort it, you can unload the AppDomain along with the DLL. This may have the added benefit of cleaning up resources that a straightforward Thread.Abort() will not.
Related
Is there a way to fire an Http call to an external web API within my own web API without having to wait for results?
The scenario I have is that I really don't care whether or not the call succeeds and I don't need the results of that query.
I'm currently doing something like this within one of my web API methods:
var client = new HttpClient() { BaseAddress = someOtherApiAddress };
client.PostAsync("DoSomething", null);
I cannot put this piece of code within a using statement because the call doesn't go through in that case. I also don't want to call .Result() on the task because I don't want to wait for the query to finish.
I'm trying to understand the implications of doing something like this. I read all over that this is really dangerous, but I'm not sure why. What happens for example when my initial query ends. Will IIS dispose the thread and the client object, and can this cause problems at the other end of the query?
Is there a way to fire an Http call to an external web API within my own web API without having to wait for results?
Yes. It's called fire and forget. However, it seems like you already have discovered it.
I'm trying to understand the implications of doing something like this
In one of the links in the answers you linked above state the three risks:
An unhandled exception in a thread not associated with a request will take down the process. This occurs even if you have a handler setup via the Application_Error method.
This means that any exception thrown in your application or in the receiving application won't be caught (There are methods to get past this)
If you run your site in a Web Farm, you could end up with multiple instances of your app that all attempt to run the same task at the same time. A little more challenging to deal with than the first item, but still not too hard. One typical approach is to use a resource common to all the servers, such as the database, as a synchronization mechanism to coordinate tasks.
You could have multiple fire-and forget calls when you mean to have just one.
The AppDomain your site runs in can go down for a number of reasons and take down your background task with it. This could corrupt data if it happens in the middle of your code execution.
Here is the danger. Should your AppDomain go down, it may corrupt the data that is being sent to the other API causing strange behavior at the other end.
I'm trying to understand the implications of doing something like
this. I read all over that this is really dangerous
Dangerous is relative. If you execute something that you don't care at all if it completes or not, then you shouldn't care at all if IIS decides to recycle your app while it's executing either, should you? The thing you'll need to keep in mind is that offloading work without registration might also cause the entire process to terminate.
Will IIS dispose the thread and the client object?
IIS can recycle the AppDomain, causing your thread to abnormally abort. Will it do so depends on many factors, such as how recycling is defined in your IIS, and if you're doing any other operations which may cause a recycle.
In many off his posts, Stephan Cleary tries to convey the point that offloading work without registering it with ASP.NET is dangerous and may cause undesirable side effects, for all the reason you've read. That's also why there are libraries such as AspNetBackgroundTasks or using Hangfire for that matter.
The thing you should most worry about is a thread which isn't associated with a request can cause your entire process to terminate:
An unhandled exception in a thread not associated with a request will
take down the process. This occurs even if you have a handler setup
via the Application_Error method.
Yes, there are a few ways to fire-and-forget a "task" or piece of work without needing confirmation. I've used Hangfire and it has worked well for me.
The dangers, from what I understand, are that an exception in a fire-and-forget thread could bring down your entire IIS process.
See this excellent link about it.
I have an automatic betting BOT.
I use a Windows Service and timers to set off a job every 30 seconds in its own thread that takes bets from the DB, loops through and places them.
However in certain occurrences when the job is too long (over 30 seconds) I can get the same bet being placed twice using the same BetPK (unique ID) as the job for placing it runs at the same time as a previously started thread.
I am using C#, NET 4, VS 2012.
At the moment I set a "locked" flag in a table when the job to place bets runs and then unset it on finishing. So if another job runs and the job is locked it will return ASAP. However this is relying on the DB and network traffic.
What would be the best way in C# to prevent a job started by a timer thread from clashing with a previously started thread. I am thinking I could set a flag IN the service controller that spawns the threads so if a job is running another one won't spawn.
However I would like to learn the correct way to handle multi threaed clashes like this. I just lost a couple of hundred pounds today due to 2 LAY bets being placed at exactly the same time. As only one record existed for the Bet, the last bet placed had the Betfair ID updated so I had no clue about the duplicate until I checked Betfairs own page.
I do already do checks to see if the bet has already been placed before trying to place it but in cases where the "placebet" method is running on the same Bet record at exactly the same time then this is no good.
Any help much appreciated.
Thanks
No, the best solution is to keep the locks in the database. The app should be as stateless as possible. You already have a great solution.
Locking inside of your app is error prone and the errors are catastrophic (deadlock, the app stops to work until manually restarted). Locking using the database is much easier, and errors are recoverable.
Just get the locking with the database right. Ask a new question where you post details on what you're doing. I recommend that you XLOCK any betting jobs that you're working on. That way they can only be executed once. Use the power of database locks and transactions to make this work. This is by far easier than app-level threading.
You could always try implementing a db like Redis (redis.io) that offers built in POP functions (http://redis.io/commands/lpop). Redis has a C# client and is super useful for any kind of app where speed is crucial as it keeps the entire db in memory. It's also single threaded which makes it easy to implement distributors for multi-consumer type applications.
I'd also recommend checking out http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis as it lays out the pros and cons for Redis and other dbs. Might help you make future db decisions.
Old question, I know, but I wanted to throw this out there for anybody that stumbles across it.
C# (and presumably VB.NET) offers a couple of nice options for handling thread synchronization. You can use the lock keyword to block execution until a given lock is available, or Monitor.TryEnter() if you want to specify a timeout (possibly immediately) for taking the lock.
For either of these approaches, you need an object to use for locking. Pretty much any object will do; if you aren't synchronizing access to some object itself (collection, database connection, whatever), you can even just instantiate a throwaway object. For a polling timer, the latter is typical.
First, make sure you have an object to use for synchronization:
public class DatabasePollingClass {
object PollingTimerLock = new object();
...
Now, if you want the polling threads to block indefinitely waiting for their turn, use the lock keyword:
public class DatabasePollingClass {
object PollingTimerLock = new object();
...
protected void PollingTimerCallback() {
lock (PollingTimerLock) {
//Useful stuff here
}
}
}
Only a single thread will be allowed within the lock (PollingTimerLock) block of code at a time. All other threads will wait indefinitely, then resume executing as soon as they can acquire the lock for themselves.
However, you probably don't want that behavior. If you'd rather have the subsequent threads abort immediately (or after a short wait) if another polling thread is still running, you can use Monitor.TryEnter() when taking the lock. This does require slightly more caution, however:
public class DatabasePollingClass {
object PollingTimerLock = new object();
...
protected void PollingTimerCallback() {
if (Monitor.TryEnter(PollingTimerLock)) { //Acquires lock on PollingTimerLock object
try {
//Useful stuff here
} finally {
//Releases lock.
//You MUST do this in a finally block! (See below.)
Monitor.Exit(PollingTimerLock);
}
} else {
Console.WriteLine("Warning: Polling timer overlap. Skipping.");
}
}
}
The additional caution stems from the fact that, unlike the lock keyword, Monitor.TryEnter() requires you to manually release the lock when you're finished with it. In order to guarantee that this happens, you need to wrap your whole critical section in a try block, and release the lock in the finally block. This is to ensure that the lock will be released, even if the polling method fails or returns early. If the method returned without releasing the lock, your program would effectively be hung, as no further threads would be able to acquire the lock.
Another option, which doesn't use locking mechanisms, would be to configure your Timer without a repeat period, i.e. a one-shot Timer. At the end of your polling method, you would dispose the old Timer, and set a new one (you would also need to do this within a finally block to guarantee that the Timer gets reset by the end of the method). This approach would be useful if you want to poll the database at a certain interval since the end of the previous polling. It's a subtle distinction, but it also solves the problem of concurrent polling attempts.
Note that this is a really simple thread concurrency example. As long as all of your locking is happening on threads separate from your UI thread (the message pump itself can become a point of contention), and you're only ever locking a single object, you shouldn't have to worry too much about deadlocks. Those can be really unpleasant to debug; the symptom is usually "application stops responding, and now you get to guess which threads are waiting on what".
Here's the situation, I am writing the framework for a code war contest. As the code runs, for each turn, it calls a method in the library provided by each contestant. The rules of the contest is the method must return in 1 second or we kill the task calling them. We then use a default result for that turn.
The method has no support for a cancel because we cannot trust the called code to respond to a cancel. And we need to kill the thread because if we have 10 or 20 ignored background tasks then all calls going forward will provide fewer clock cycles on each call and methods that before took less than 1 second now take more.
On the plus side, the method we're killing should have no resources open, etc. so an abort should not leave anything hanging.
Update: Two things to keep in mind here. First, this is like a game - so performance is important. Second, the worker thread is unlikely to have any resources open. If one of the called methods goes overlong, I need to abort it and move on quickly.
You should run each contestant in his own AppDomain with low privileges. This has several advantages:
It's sandboxed
It can't interact with any other code in the process
Force unloading an AppDomain is relatively clean.
Even if you prefer killing the thread over unloading the AppDomain I'd still put each contestant into an AppDomain to get the isolation.
Unfortunately Thread.Abort is not enough. It still executes finally clauses which can take as long as they want.
I would recommend that you run the code in a second process and carefully define the interface for communicating with it to ensure that it can handle not receiving a response. Most operating systems are designed to clean up fairly well after a killing a process.
For communication, you should probably avoid .NET remoting, as that seems likely to be left in an inconsistent state on the server side. Some other choices: sockets, named pipes, web service.
Thread.Interrupt() method is maybe what you are looking for.
As the MSDN documentation says, "If this thread is not currently blocked in a wait, sleep, or join state, it will be interrupted when it next begins to block."
It is not an abort, it forces the running thread to throws ThreadInterruptedException when the thread enters in a wait state.
You can then use a timer in another thread with a timeout to check if the thread don't really want to terminate, if the thread refuses to terminate in, for example, 30 seconds, you can abort it.
My web service has this code
lock(typeof MyWebServiceClass)
Well i call a 3rd party unchangeable code and it never returns. BOOM! BAM!
I am locked forever now and my web site crashes.
This never returns only happens once in a great while.
Is it possible to create a lock that times out? IE lock the code for 5 minutes then release the lock?
Instead of creating a timed lock, I would limit your third party request by putting it in a separate thread/task. Then, kick off the thread (or task if .NET 4.0 and TPL available to you) and join on the response with a timeout. If the join times out, then cancel the thread (or call the cancel token on the TPL task).
Is it possible to create a lock that times out?
Yes, this unpleasant situation is often refereed to as deadlock.
Usually it is good practice to lock on a static private object instead of locking on instance fields or the class itself:
private static object _syncRoot = new object();
and then:
lock(_syncRoot) {
}
If the 3rd party API has a cancellation mechanism then use that.
lock(typeof MyWebServiceClass)
{
if (ThirdPartyApiThatAcceptsTimeout(TimeSpan.FromMinutes(5)))
{
// The call was successful so proceed.
}
else
{
// The call timed out so bail out.
return;
}
}
However, I highly suspect that this API does not have a cancellation mechanism and so that is why you posed this question. If that is the case then this just got exponentially harder.
The naive approach would be to defer the API call to another thread. If the thread does not respond in a timely fashion then you can abort it.
lock(typeof MyWebServiceClass)
{
var thread = new Thread(
() =>
{
ThirdPartyApiThatCouldBlockIndefinitely();
});
thread.Start();
if (thread.Join(TimeSpan.FromMinutes(5))
{
// The call was successful so proceed.
}
else
{
// The call timed out so bail out.
thread.Abort();
return;
}
}
There are many problems with this though. First, there is no guarantee that the thread will accept the abort request. Since 2.0 there are special rules in the CLR that dictate when aborts can be injected into the thread. I believe the CLR will defer the injection while unmanaged code is executing. So if your API is unmanaged then the abort might not work. Also, aborts are voluntary since the thread could catch ThreadAbortException and ignore it. Second, aborting is dangerous since the abort can be injected asynchronously. This makes it very difficult to guard against corrupting shared state. That is why the AppDomain is normally terminated after an abort.
The safest way to handle this is to put the API call into a separate process. You would have to use interprocess communication protocols like .NET Remoting, WCF, pipes, etc. to transfer data to/from the call which would be a real pain to maintain. But, it would be the safest since you can safely kill the process without the risk of corrupting the AppDomain of the caller.
I really feel for you because this problem is really hard to solve correctly.
I have a function called ApiCalls() that is wrapped in a locker because the api I'm using is not multi-thread safe. Occasionally an api call fails to return and I can't think of a way to handle this kind of situation. I was thinking about creating a timer on the lock object, but it seems the locker doesn't not have something like that.
There's really no good answer for this. A bad, but probably workable, answer is to have a watchdog thread that Aborts the calling thread after a timeout. In other words, after acquiring the lock but before calling the API, you'd order the watchdog to kill you. When you get back from the call (if you get back), you'd call off the watchdog.
Again, this is not a great solution, as Abort is very messy.
I don't think you can reasonably recover from this problem. Suppose that you could timeout, you would then attempt to call the API again, but the previous call is still active and you have said that the API is not thread-safe.
You simply cannot defend yourself from fundamentally flawed dependencies of this kind.
The only really safe thing to do is to restart the process. Steven Sudit's suggestion is one way to achieve that.
This can be solved by wrapping the API calls in a separate assembly and loading that assembly into a seperate application domain by using the AppDomain class.....
Use application domains to isolate
tasks that might bring down a process.
If the state of the AppDomain that's
executing a task becomes unstable, the
AppDomain can be unloaded without
affecting the process. This is
important when a process must run for
long periods without restarting.
You can then call thread abort on the call in the separate AppDomain, signal the host domain that an abort has happened. The host domain would unload the offending domain, thus unloading the API, and start a new domain with the API reset. You would also want a watchdog on the API domain so the host could take action if the API domain freezes.
Miscellaneous links: C# Nutshell AppDomain Listings, cbrumme's WebLog, Good example of use of AppDomain, Using AppDomain to Load and Unload Dynamic Assemblies
The only safe-ish solution is probably to start another process to handle the API calls, and then kill the process if they get stuck. Even that doesn't guarantee that the API's handlers won't get into a bogus state that can only be cured via system restart, but using Thread.Abort can mortally wound a process.
If you don't want to use "untrusted" means of killing the process, you could have one thread in the process perform the API calls while another watches for a "please die" message. Watchdogs can be tricky; if a watchdog is set for 15 seconds and an action would take 17 seconds to complete, one might request an action, time out after 15 seconds, retry the action, time out after 15 seconds, etc. indefinitely. It may be good to have the watchdog time adjust after each failure (e.g. try an action, letting it have up to 15 seconds; if that doesn't work, and nobody's complaining, try again and let it go 30 seconds; if that's still no good, give it 60 seconds.)