in a dotnet core http application we have a call that performs work, makes several http calls out to backing services, and returns. It was all async await. which meant that the call was waiting for the backing services to perform their work before returning the call to the client. this was causing the call to timeout.
the solution that was presented was to remove the async await all the way down as low as we could then essentially just wrap the http calls (still async task methods) in pragma tags that suppress the warning/compiler error.
This makes me nervous because you don't guarantee that the final (or any) of http requests are made before the calling thread returns and the async machines associated with that thread are cleaned up.
Am I missing something? is this one of those weird but usable situations? or would it be more appropriate to spin off threads to handle those http calls?
You're talking about fire-and-forget, and you're right to be worried. There are several problems with fire-and-forget. This is because "forget" means "forget", and it's almost always the wrong decision to have your application just forget about something.
If there's an exception calling one of those inner HTTP requests, that exception will be ignored, unless you have special logic handling that situation. And remember, there's no outer request anymore, so returning an error isn't possible (there's nowhere for it to return to). So you have the possibility of silently-swallowed errors. This should make you nervous.
Also, you have the problem of not informing ASP.NET that you have ongoing background work. By returning early, you're telling ASP.NET to send the response and that everything is fine, even though in reality, the work isn't done and you have no idea when it will be done or even whether it will succeed. The point here is that nothing upstream of your code (including ASP.NET, IIS/Kestrel, proxies, load balancers) has any idea that your code is still working - after all, your code did just tell all those things that it's done handling that request. Will ASP.NET respond to a shutdown request? Sure! Can IIS do its periodic app pool recycle? Sure! Can your proxy take that node out of rotation when doing a rolling upgrade? Sure! Will your load balancer send it more work since it's not doing anything? Sure! As far as any of those systems know, your app isn't actually handling that request, and that can cause problems, like your "fire and forget" work suddenly disappearing - again, with no exceptions or logs or anything. This should make you nervous.
I'd say the best approach is to fix downstream calls, if possible. Also look into asynchronous concurrency, e.g., starting several calls and then await Task.WhenAll. If these approaches aren't sufficient, then I'd recommend a proper distributed architecture: have the API write to a persistent queue, and have the background work done by a separate application that processes that queue.
Related
I am a longtime user of Azure's Application Insights, and I use the TelemetryClient's TrackTrace() and TrackException() liberally in every enterprise application I write.
One thing that has always bothered me slightly is that these methods are synchronous. Since these methods communicate with an external API, it would seem there is an ever-present risk of blocking; e.g., if the network is down/slow, or if App Insights' own API is having issues.
In such cases, it seems possible (at least in theory) that an entire application could hang. In such cases, if they ever occur, I would like my applications to continue operating despite failing to trace within a reasonable time frame.
I've done some research online, and it appears that there is no built-in way to call these methods asynchronously. Do you know of any way to accomplish this? (Or.....does the App Insights API have an under-the-hood black-box way of automatically preventing these sorts of things?)
Of course, I know I could always wrap my calls in a Task (e.g., await Task.Run(() => myTelemetryClient.TrackTrace("my message")); (or write an async extension method that does this). I could also use a timer to cancel such a request. But it would be nice if there was a more integrated way of doing this.
Can anyone enlighten me? Is this really a potential problem that I should be concerned with? Or am I merely tilting at windmills?
Update: I just now saw this, which indicates that AI does indeed handle tracking in an asynchronous manner "under the hood". But how can this be reliable, given the truism that asynchronous operations really need to be made async all the way up and down the call stack in order to be blocking-proof?
Is this really a potential problem that I should be concerned with?
No. None of the TrackABC() methods communicate with any external API or do anything which would take a long time. Track() runs all telemetry initializers, and then queues the item into an in-memory queue.
While the built-in telemetry initializers are designed to finish quickly and make no I/O or HttpCalls, if a user adds a telemetryinitializer which makes an http call or something similar, then Yes, it'll affect you Track() calls. But with normal usage of TelemetryInitializers, this should not be a concern.
If it's anything like the JS API, the tracking events are placed in a queue then dequeued and sent (possibly in batches at configurable intervals) independently of the TrackXXX methods. Enqueuing an event can be synchronous, but the sending end of the process can operated asynchronously. The queue decouples the two from one another. –
spender
I think #spender answered my question! Thanks!
We are developing a monolithic web application – very stateful. It handles both HTTP requests and long lived SignalR connections. (In ASP.NET Core 3.1 – we will upgrade to .NET 5 later.)
We do a redirect from a login page to our “main page”. The main page takes a while to load and initialize, after that it connects with SignalR. We also have a lot of work to do at the server side. Doing the server work in the login request (before redirecting to the main page) would slow down the login.
“Oh, let’s use a Task then!”, I thought. That is, put the server work in a Task, save that in the user state, and let it execute in parallel with the loading of the main page. Something like this (simplified):
public static async Task ServerSideInit()
{
// do a lot of init work
}
// at the end of the controller handling the login page POST:
UserState.BackgroundTask = ServerSideInit();
Redirect(UrlToTheMainPage);
// when the main page connects via SignalR:
try {
await UserState.BackgroundTask;
}
catch {
// handle errors in the init work
}
This would really speed things up. It won’t matter if the page loading or the init work finishes first – we await the Task. And the work in ServerSideInit() isn’t critical. If something happens and the main page never connects, the UserState (and the Task) will be destroyed after a timeout – and that’s perfectly OK. (There are some caveats. We would e.g. have to use IServiceProvider to create/dispose a scope in ServerSideInit(), so we get a scoped DbContext outside of the controller. But that’s OK.)
But then I read that there is a risk the ASP.NET Core framework shuts down the Task when wrapping up the POST request! (Do you have to await async methods?) The simple HostingEnvironment.QueueBackgroundWorkItem isn’t available any longer. There is a new BackgroundService class, though. (https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-3.1&tabs=visual-studio) But registering a service and queueing jobs seems like a very cumbersome solution… We just want to fire a task that will take a couple of seconds to complete, and let that continue to run after ASP.NET Core has finished handling the POST request.
I’m not very experienced with ASP.NET Core… So I’d be very grateful for some input! Will my simple solution not work? Will the task be terminated by the framework? Is there some easier way to tell the framework “please don’t touch this Task”? Or is BackgroundService the way to go?
Doing the server work in the login request (before redirecting to the main page) would slow down the login. “Oh, let’s use a Task then!”, I thought. That is, put the server work in a Task, save that in the user state, and let it execute in parallel with the loading of the main page.
So, you have a need for request-extrinsic work. I.e., work that your server does that is outside the scope of a request.
The first question you need to ask yourself is "does this work need to be done?" In other words, "am I OK with occasionally losing work?". If this work must be done for correctness reasons, then there is only one real solution: asynchronous messaging. If you're OK with occasionally losing work (e.g., if the main page will detect that the ServerSideInit is not done and will do it at that time), then what you're really talking about is a cache, and that's fine to have an in-memory solution for.
But then I read that there is a risk the ASP.NET Core framework shuts down the Task when wrapping up the POST request!
The first thing to recognize is that shutdowns are normal. Rolling updates during regular deployments, OS patches, etc... Your web server will voluntarily shut down sooner or later, and any code that assumes it will run forever is inherently buggy.
ASP.NET Core by default will consider itself "safe to shut down" when all requests have been responded to. This is the reasonable behavior for any HTTP service, and this logic extends to every HTTP framework, regardless of language or runtime. However, this is clearly a problem for request-extrinsic code.
So, if your code just starts a task by calling the method directly (or by Task.Run, another sadly popular option), then it is living dangerously: ASP.NET has no idea that request-extrinsic code even exists, and will happily exit when requested, abruptly terminating that code.
There are stopgap solutions like HostingEnvironment.QueueBackgroundWorkItem (pre-Core) and IHostedService / IHostApplicationLifetime (Core). These register the request-extrinsic code so that ASP.NET is aware of it, and will not shut down until that code completes. However, those solutions only go partway; since they are in-memory, they are also dangerous: ASP.NET is now aware of the request-extrinsic code, but HTTP proxies, load balancers, and deployment scripts are not.
Is there some easier way to tell the framework “please don’t touch this Task”?
Back to the question at the beginning of this answer: "does this work need to be done?"
If it's just an optimization and doesn't need to be done, then just firing off the work with a Task.Run (or IHostedService) should be sufficient. I wouldn't keep it in UserState, though, since Tasks aren't serializable.
If the work needs to be done, then build an asynchronous messaging solution.
I am using SendGrid (cloud based SMTP service) to send emails from a web api project. I want my application to wait/block (for say 30 secs) until I have a response from SendGrid before returning the response to the client, rather than returning immediately. The SendGrid library has a DeliverAsync method which returns a Task.
I have been looking at how I might Wait on the task.
I have read endless articles about how one might do this and understand that if it was my own code I would use the ConfigureAwait(false) on the task to prevent a deadlock and allow me to Wait. The problem here is that the code is not mine! It doesn't look like SendGrid have a synchronous Send method.
I do not have async controllers wired up, although appreciate this would be a way to do this, but I'd like to know if there is another way I could do this.
Hope this makes sense!!
If you can await all the way up to and including the controller action, you should, and as that's your code it should be achievable. In that case, at most you might want to consider ConfigureAwait(true) for the call from the controller method only, and have the rest (downwards) as ConfigureAwait(false) (as library methods should be). Most of the time you don't even need the context preserved in the controller action - it depends what you do there - and in that case use ConfigureAwait(false) there too.
You use "wait/block" as though they're the same, but in the TAP world they're quite different. Using await will wait for the SendGrid() call to complete before continuing, while not blocking the calling thread.
If you can't do that, it's far less-preferable to use the blocking .Wait() or .Result, or as others mention GetAwaiter().GetResult(). All 3 will block the caller as well. In Web API you can often get away with this; but in other contexts - e.g. WinForms - you probably won't.
As it's your code, use await.
Is there a way to fire an Http call to an external web API within my own web API without having to wait for results?
The scenario I have is that I really don't care whether or not the call succeeds and I don't need the results of that query.
I'm currently doing something like this within one of my web API methods:
var client = new HttpClient() { BaseAddress = someOtherApiAddress };
client.PostAsync("DoSomething", null);
I cannot put this piece of code within a using statement because the call doesn't go through in that case. I also don't want to call .Result() on the task because I don't want to wait for the query to finish.
I'm trying to understand the implications of doing something like this. I read all over that this is really dangerous, but I'm not sure why. What happens for example when my initial query ends. Will IIS dispose the thread and the client object, and can this cause problems at the other end of the query?
Is there a way to fire an Http call to an external web API within my own web API without having to wait for results?
Yes. It's called fire and forget. However, it seems like you already have discovered it.
I'm trying to understand the implications of doing something like this
In one of the links in the answers you linked above state the three risks:
An unhandled exception in a thread not associated with a request will take down the process. This occurs even if you have a handler setup via the Application_Error method.
This means that any exception thrown in your application or in the receiving application won't be caught (There are methods to get past this)
If you run your site in a Web Farm, you could end up with multiple instances of your app that all attempt to run the same task at the same time. A little more challenging to deal with than the first item, but still not too hard. One typical approach is to use a resource common to all the servers, such as the database, as a synchronization mechanism to coordinate tasks.
You could have multiple fire-and forget calls when you mean to have just one.
The AppDomain your site runs in can go down for a number of reasons and take down your background task with it. This could corrupt data if it happens in the middle of your code execution.
Here is the danger. Should your AppDomain go down, it may corrupt the data that is being sent to the other API causing strange behavior at the other end.
I'm trying to understand the implications of doing something like
this. I read all over that this is really dangerous
Dangerous is relative. If you execute something that you don't care at all if it completes or not, then you shouldn't care at all if IIS decides to recycle your app while it's executing either, should you? The thing you'll need to keep in mind is that offloading work without registration might also cause the entire process to terminate.
Will IIS dispose the thread and the client object?
IIS can recycle the AppDomain, causing your thread to abnormally abort. Will it do so depends on many factors, such as how recycling is defined in your IIS, and if you're doing any other operations which may cause a recycle.
In many off his posts, Stephan Cleary tries to convey the point that offloading work without registering it with ASP.NET is dangerous and may cause undesirable side effects, for all the reason you've read. That's also why there are libraries such as AspNetBackgroundTasks or using Hangfire for that matter.
The thing you should most worry about is a thread which isn't associated with a request can cause your entire process to terminate:
An unhandled exception in a thread not associated with a request will
take down the process. This occurs even if you have a handler setup
via the Application_Error method.
Yes, there are a few ways to fire-and-forget a "task" or piece of work without needing confirmation. I've used Hangfire and it has worked well for me.
The dangers, from what I understand, are that an exception in a fire-and-forget thread could bring down your entire IIS process.
See this excellent link about it.
In my scenario when ever user changes some fields in the program and does a SAVE, a webserivce request is sent to save some logging information into the database. I did some search on the website and found this solution for Async calls:
ThreadPool.QueueUserWorkItem(delegate
{
// Create an instance of my webservice object.
// call its Log webmethod.
});
But since I don't have much experience with webservices and Async calls so I wanted to show you the scenario I have and the way I am handling it to get your opinion about it and if it the right way to do this. Thanks for suggestions.
Can you tolerate the logging work to be lost? Only then should you start "background work" in an ASP.NET app.
QueueUserWorkItem will work. The more modern version is Task.Run. Make sure you catch errors that happen on that thread-pool thread. If not, you'll never find out about bugs and silently lose work.
If you expect a high volume of such calls, or expect them to take a long time, consider using async IO. It does not use any thread while in progress (not even a background thread).