I have a .NET Core service. We want to implement fire and forget call for one of its endpoint. When request comes to this end point it should fire a method and immediately returns response as Ok. The method then processes that requests by calling other services, and some DB operations.
The above works fine. But, when I tried to do perf testing with thousands of requests, sent back to back, we noticed that some requests are not getting processed at all. It is working fine up to 4000 requests but more than that it is not processing some(approx 20+) requests. There are no exceptions in the logs
How can I identify the issue?
await Task.Factory.StartNew(() => FireAndForgetMethod());
return Ok();
"Fire and forget" literally means that you don't care if the completes and you're perfectly fine with ignoring exceptions. Since you do care that it completes and you do want to see exceptions, then you do not have a situation that is appropriate for "fire and forget".
When request comes to this end point it should fire a method and immediately returns response as Ok. The method then processes that requests by calling other services, and some DB operations.
Your API endpoint should serialize the work to be done into a message and place it on a reliable message queue (e.g., Azure Queue, Amazon Simple Queue, etc), after which it may return an HTTP result to its caller. This queue should then be read and each message executed by a background processor (e.g., ASP.NET Core Worker Service).
Related
We are trying to Consume REST API, for message processor which has some operation which might take more than configured timeout.
Would like to know, if the timeout of Http call to API, will stop execution of API, or API will keep executing?
Idea is that, we can fire and forget API, we are not worried if API does not return 404 or 503. But would like to hear if API will continue to execute?
Any input or suggestion appreciated.
You should use some kind of background processing to handle the process.
I recommend using Hangfire for it.
https://www.hangfire.io/
Use Hangfire to enqueue a job, it will return a job id. You can return this job id to client side.
Expose another API to check for the status of this job.
Great way is to handle this with callback/observer pattern. First of all, understand that there are two types of timeout, server and client. You can explicitly specify client timeout, server timeout is handled by server itself.
So, you will need to implement algorithm such that,
you identify each request unique way and mark it before firing into
memory or file/db.
Fire request with associated callback method.
Hence on response you have control to do stuff like, mark request
fulfilled or failed or what ever it is.
Mark/delete request data.
As you'll most likely figure it out, I'm not very experienced with async operations in general (only used Android's AsyncTask).
This is the outline of a WCF REST POST method; I'll use this image to hopefully explain what I'm trying to achieve.
The FirstJob saves some stuff to the database.
SecondJob reads what was saved in the database and does some work with the data.
The client does not care about what happens in SecondJob and just wants to receive the response from FirstJob.
So the two jobs don't need to run in parallel as SecondJob depends on FirstJob; the SecondJob would ideally run in a separate thread/context(?) or similar.
From what I've noticed, the second job does start in a separate thread, the execution reaches the return statement while the 2nd job is running, but the request does not end until SecondJob finishes.
I'd personally treat the second job as a separate POST operation and call the second job POST from the controller. The controller is the controller for the first job and can return the correct status from the first job; it just happens to call a POST out to a second endpoint while doing it.
The benefit of this approach is that the second job doesn't even need to be on the same IIS (in an NLB farm it could be anywhere) so you get load balancing thrown in for free. Alternatively the "second job server" can be on a specific URL reserved just for this kind of background processing task.
I suggest you not to rely on the IIS to handle your background task as it can shut down it without waiting. I suggest you to create a windows service application, which will accept the requests for a second jobs, via another WCF binding or database requests or something else.
You can get the results of the second jobs with another request from your controller, as #PhillipH stated.
The thing I was trying to do was actually working in the first place, but the visual studio debugger fooled me. I tested again without the debugger, but with a Tread.Sleep(60000) and it looks like it behaves as expected. The SecondJob keeps running in the background after the api call returned the response.
If I need to call a method that in turn calls some async method internally, as a fire and forget operation, how can I prevent this call from forcing the "async" to need to be used up the call stack to say...an MVC controller?
For example: My MVC Controller (not async) calls a business layer method, which in turn calls the Windows Azure Service Bus QueueClient.SendAsync(BrokeredMessage), to drop a message in a queue, but doesn't need to wait for it to complete.
Typically, when this controller action is invoked, the compiler will throw an error that the async operation cannot be started at this time.
I know that instead of awaiting or just invoking the SendAsync() method, I could follow it up with ContinueWith(), in order to execute code on the callback of the async operation, but I've been told this not a correct solution. (see responses to Calling async method in controller)
Would someone care to enlighten me on the best way to fix this scenario? And tell me why the ContinueWith() approach is not correct?
calls the Windows Azure Service Bus QueueClient.SendAsync(BrokeredMessage), to drop a message in a queue, but doesn't need to wait for it to complete.
First, I'd reconsider this assumption. If you want a fully reliable system, the action should wait for the message to be sent to the bus. Note that this is usually a fast operation (100ms).
Typically, when this controller action is invoked, the compiler will throw an error that the async operation cannot be started at this time.
Ensure that you do not have any async void methods or EAP method calls. I would be very surprised if the Azure storage library causes that exception.
Would someone care to enlighten me on the best way to fix this scenario? And tell me why the ContinueWith() approach is not correct?
The best solution is to embrace async. ContinueWith could work but is dangerous to use (it has lots of parameters, some of which have unsafe default values); await is practically the same as ContinueWith but without the dangerous defaults.
However, if 100ms is truly unbearable, and you are willing to give up reliability (in this case, that means you accept the fact that some messages may not get sent to the bus even though the action completed successfully so the client thinks that they did), then you can use the BackgroundTaskManager from my blog to minimize the chance of lost messages.
An MVC controller method handles an HTTP request and sends back the corresponding HTTP response to the client. If I understood your question correctly, you want to call a fire-and-forget method from your async MVC controller method, then proceed with the HTTP response delivery, so the fire-and-forget method doesn't hold the response.
Indeed, you cannot fire it from with your controller method this way, without awaiting its results. I.e., you could, but if your ASP.NET application would be restarted, or the IIS server taken out of the farm, your ContinueWith callback would never be called. So, you woudn't know if the request had ever reached the Windows Azure Service.
One approach to solve this is to run a helper Web API or WCF service on the same host, or on another host on the same network (so the turn-around for a service call would be really quick). It can be a self-hosted service. You'd call this helper service from your MVC controller just to queue the fire-and-forget operation. You'd await the result of this call, but it's not a problem in this case, because both the caller and the callee of this operation would exist on the same network.
This way, the original MVC HTTP response won't be put on hold. Inside the helper service, you'd then do await QueueClient.SendAsync() to call Windows Azure Bus and process the result of this operation accordingly.
I'm using ServiceStack v4, and making an async request like:
var client = new JsonServiceClient(url);
Task reqTask = client.GetAsync(request);
I'll be making a lot of requests simultaneously to different servers, and need to support cancellation to immediately cancel all outstanding calls. The underlying AsyncServiceClient has a method CancelAsync, but it doesn't seem to do anything.
How can I cancel the request such that it aborts the web request immediately?
Short version, it seems that CancelAsync does nothing indeed.
Looking at the code, it seems that ServiceStack uses the old way (v3) of asynchronously executing requests and notifying you with with callbacks, all built on top of WebRequest and the APM-style calls (BeginXXX,EndXXX).
The GetAsync, SendAsync methods are actually wrappers that use a TaskCompletionSource to provide a Task based interface over the actual asynchronous operation.
CancelAsync actually calls a CancelAsyncFn delegate that isn't set by the classes in JsonServiceClient's hierarchy, so calling it actually does nothing. I suspect this is provided to create custom clients and actions.
In any case, cancelling an HTTP request isn't so easy as it sounds. You need to understand what it is that you are trying to cancel.
You can't actually cancel the call to the server once it's left your application.
You can't tell the server or the intervening devices to stop processing, as HTTP doesn't provide such a mechanism. The server has to provide such an API, which is actually a new HTTP call.
You can't prevent the server from sending a response
You can cancel sending the entire request to the server. This will only work for requests that take a detectable amount of time to serialize
You can ignore the server's response and refuse to open a stream to read the body
You can stop reading the response from the server
You can simply ignore the results of the operation
Looking at the code, it seems that you can't do #4, #5 or #6, because AsyncServiceClient doesn't expose the IAsyncResult used internally. The only thing you can do is simply ignore the response and pay the price of deserialization. Note, the same applies if you use HttpClient's GetStringAsync methods instead of executing each individual step.
So, if cancel means abandoning the request, you can do this by simply ignoring the return value. If it means stop server processing, you can't.
Cancellable requests is a feature.
http://www.github.com/ServiceStack/ServiceStack/wiki/Cancellable-Requests
You basically tag your requests and the feature adds service calls to cancel using the tag. You also have to wrap logic in your service to handle this feature, much like the caching feature.
I have a class library I developed that is rather processing intensive that I currently call through a WCF REST service.
The REST service directly accesses the DLLs for the class library and more or less the WCF rest service is an interface for the system.
Let's say the following methods are defined:
Create Request
Starts a thread that takes five minutes, but immediately returns a session ID that the process generates and the thread uses to report when it is completed to the database.
Check Status
Accepts a session id and checks the database to see if the process has completed.
I have to think that there is a better way to "manage" the threads running, however, my requirements state that the user should receive an immediate response from the REST service upon issuing a request.
I am using the WCF Message property to return XML to the browser and as this application can be called from any programming language I can't use classic WCF and callbacks (I think, correct me if I am wrong).
Sometimes I run into an issue where an error occurs and the iscomplete event never gets written to the database and therefore the "Check Status" method says it's processing forever.
Does anyone have any ideas about what is normally done and what can be done in this situation?
Thanks!
Jeffrey Kevin Pry
Your service should return a 202 Accepted at the initial request with a way for the client to check the current status, either through the Location header or as part of the content.
As you indicate the client then polls the URL indicated to check the current status. I would also suggest adding a bit of cache time to this response in case a client just starts looping.
How you handle things on the server is up to you and in no way related to REST. For one thing I would put all logic that executes as the background thread in a try/catch to you can return an error status back if an error occurs and possibly retry the action depending on the circumstances.
I implemented a similiar process for importing/processing of large files and to be honest, I have never had a problem. Perhaps resolving the reason that the IsComplete never gets set will make this more resilient.
Not much of an answer, but still..