I have deployed an ASP .net MVC web app to Azure App service.
I do a GET request from my site to some controller method which gets data from DB(DbContext). Sometimes the process of getting data from DB may take more than 4 minutes. That means that my request has no action more than 4 minutes. After that Azure kills the connection - I get message:
500 - The request timed out. The web server failed
to respond within the specified time.
This is a method example:
[HttpGet]
public async Task<JsonResult> LongGet(string testString)
{
var task = Task.Delay(360000);
await task;
return Json("Woke", JsonRequestBehavior.AllowGet);
}
I have seen a lot of questions like this, but I got no answer:
Not working 1
Cant give other link - reputation is too low.
I have read this article - its about Azure Load Balancer which is not available for webapps, but its written that common way of handling my problem in Azure webapp is using TCP Keep-alive. So I changed my method:
[HttpGet]
public async Task<JsonResult> LongPost(string testString)
{
ServicePointManager.SetTcpKeepAlive(true, 1000, 5000);
ServicePointManager.MaxServicePointIdleTime = 400000;
ServicePointManager.FindServicePoint(Request.Url).MaxIdleTime = 4000000;
var task = Task.Delay(360000);
await task;
return Json("Woke", JsonRequestBehavior.AllowGet);
}
But still get same error.
I am using simple GET request like
GET /Home/LongPost?testString="abc" HTTP/1.1
Host: longgetrequest.azurewebsites.net
Cache-Control: no-cache
Postman-Token: bde0d996-8cf3-2b3f-20cd-d704016b29c6
So I am looking for the answer what am I doing wrong and how to increase request timeout time in Azure Web app. Any help is appreciated.
Azure setting on portal:
Web sockets - On
Always On - On
App settings:
SCM_COMMAND_IDLE_TIMEOUT = 3600
WEBSITE_NODE_DEFAULT_VERSION = 4.2.3
230 seconds. That's it. That's the in-flight request timeout in Azure App Service. It's hardcoded in the platform so TCP keep-alives or not you're still bound by it.
Source -- see David Ebbo's answer here:
https://social.msdn.microsoft.com/Forums/en-US/17305ddc-07b2-436c-881b-286d1744c98f/503-errors-with-large-pdf-file?forum=windowsazurewebsitespreview
There is a 230 second (i.e. a little less than 4 mins) timeout for requests that are not sending any data back. After that, the client gets the 500 you saw, even though in reality the request is allowed to continue server side.
Without knowing more about your application it's difficult to suggest a different approach. However what's clear is that you do need a different approach --
Maybe return a 202 Accepted instead with a Location header to poll for the result later?
I just changed my Azure Web Site from Shared Enviroment to Standard, and it works.
Related
I am working on a .NET Core 3.1 web api application and am having a terrible time with something closing my long running API request.
I have a POST endpoint that looks like this (slightly simplified):
[HttpPost]
[IgnoreAntiforgeryToken]
[ActionName("LoadDataIntoCache")]
public async Task<IActionResult> LoadDataIntoCache([FromQuery] string filter)
{
//long running process (15-20 mins)
var success = await _riskService.LoadDataIntoCache(filter);
if (success == false)
{
return StatusCode(StatusCodes.Status500InternalServerError);
}
return Ok();
}
This endpoint works fine when I test it locally via Postman. However, when I push this to the server (IIS), and hit the endpoint via Postman it produces an error after 5 minutes: Error: read ECONNRESE.
No more details are produced that this. Checking the logs of the application, it does not throw an exception, in fact it appears that the long running processes continues to run as if nothing is wrong. Its as if the connection itself is just being closed by something, but that the application is working fine.
I have also tried calling this endpoint via C# instead of Postman. My calling code produced the following exception message Processing of the HTTP request resulted in an exception. and additionally The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
I have checked the IIS timeout, which is set to 120s, which does not align with the 5 minute time I am seeing. I have checked a bunch of timeout settings on the .NET side, but my understanding is the .NET Core 3.1 does not need this settings because it will wait forever by default? This application is also set to run inProcess if that is significant...
I am really scratching my head on this one. Any pointers would be much appreciated.
I have two webapps running .net core 2.1 self hosted (it's Azure, but the behavior applies also to when I'm running it locally) with NancyFx.
App1 is just a proxy that collects client requests, does authentication, and forwards them to App2 and than sends response back to clients. This architecture is not likely to change due to some internal limitations.
App1 uses HttpClient and HttpClientFactory for forwarding.
Now the problem. I run a load test which generates a lot of requests. In logs I see that App2 is in general fast enough with handling them, but when the load increases App1 is still running into timeouts. I investigated faling requests and see roughly the following:
8:30:00 Client request came to App1
8:30:01 HttpClient in App1 posted
request to App2
8:30:31 Request accepted by App2 (What? 30 seconds
when both apps are localhost?!)
8:32:00 In App2:
ERROR|Microsoft.AspNetCore.Server.Kestrel|Connection id
"0HLFE17I2N0C2", Request id "0HLFE17GH3G4R:00000001": An unhandled
exception was thrown by the application.
System.OperationCanceledException: The operation was canceled. In
App1: 503
Some research:
In App2 the CPU and RAM remain low
In App2 database access is fast and queries are all executed in much under 1 sec
In dotTrace I see that lots of time is spent in Nancy.NancyEngine.HandleRequest
I added ServicePointManager.DefaultConnectionLimit = 100; to both apps
I added <connectionManagement><add address = "http://localhost:5001" maxconnection = "100"
/></connectionManagement>
I tried with HttpRequest without factory and passed in HttpRequestHandler with MaxConnections = 100
I added this to App2:
builder.UseKestrel(options =>
{
options.Listen(IPAddress.Loopback, 5001, listenOptions =>
{
listenOptions.KestrelServerOptions.Limits.MaxConcurrentConnections = 400;
listenOptions.KestrelServerOptions.Limits.MaxConcurrentUpgradedConnections = 400;
});
});
Nothing helps, it looks to me like throttling, but where? IP of App1 doesn't change of course and probably App2 doesn't like so many requests. But the behavior is the same locally, so it's not some Azure protection.
I have built an api to handle my application and i'm using a controller to upload files with FlowJS. It works with no problems on my local machine and also on my development server but when i use it on my client's homologation server i get a problem with the requisitions.
After some requests, it's a random number or requests sometimes 5 sometimes 10, the next request just freezes and won't respond and if i try to duplicate it using Chrome dev console it works.
This is a print of the requests:
The controller and service that are called only have IO operations and i'm logging everything that is done there after the request reaches the controller and when i get that pending status then request isn't logged.
I have also tried to work around my DNS to check if it was a connection problem due to the amount of requisitions but the problem remais the same. What else could be happening here ?
I got an Owin self-hosted web-api server, and I'm wondering if I need to change timeout settings when there are huge file downloads?
The client I'm using reads the response withHttpCompletionOption.ResponseHeadersRead.
During debugging, after I stopped for some time in a breakpoint, I got an exception on client side while trying to read from a received stream:
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
While debugging I can reproduce this issue. It happens after around 30 seconds waiting in a breakpoint, after the Get-Request to the server returned.
Is this due to some kind of idle timeout, because I hold in a breakpoint and do not work on the received stream? Or can it also happen while I'm reading from the stream when my collection is slow and it takes too long?
Very old question but may help whoever hits the same wall.
I had the same problem with a streaming content and found the initial clue inside HTTPERR folder (C:\Windows\System32\LogFiles\HTTPERR)
2016-08-12 09:17:52 ::1%0 60095 ::1%0 8000 HTTP/1.1 GET
/endpoint/audiostream/0/0/streamer.mp3 - - - Timer_MinBytesPerSecond -
2016-08-12 09:18:19 ::1%0 60118 ::1%0 8000 HTTP/1.1 GET
/endpoint/audiostream/0/0/streamer.mp3 - - - Request_Cancelled -
Owin HttpListener has a TimeOutManager property that allows you to change most timeout/limits. The only way I found to get my webapp HttpListener instance was by accessing its properties
var listener = (OwinHttpListener);
app.Properties["Microsoft.Owin.Host.HttpListener.OwinHttpListener" ]);
listener.Listener.TimeoutManager.MinSendBytesPerSecond = uint.MaxValue;
According to owin codebase, uint.MaxValue as MinSendBytesPerSecond will just disable the flag.
I'm writing code to send a request to my web service in windows azure, and it turned out I cannot get response until timeout. What's confusing is that i'm sure everything inside my service has been done, and yet the response did not come back.
My service code looks like this:
public virtual JsonResult ServiceMethod()
{
// 1. do the work
// 2. write something to database
return Json(result, JsonRequestBehavior.AllowGet);
}
And my client code looks like this:
HttpWebRequest webRequest = WebRequest.Create(new Uri(httpAddress)) as HttpWebRequest;
webRequest.Timeout = 1000 * 1000;
webRequest.ServicePoint.ConnectionLeaseTimeout = 40 * 60 * 1000;
webRequest.ServicePoint.MaxIdleTime = 40 * 60 * 1000;
webRequest.ServicePoint.SetTcpKeepAlive(true, 50 * 1000, 1000);
webRequest.Method = "GET";
using (HttpWebResponse response = webRequest.GetResponse() as HttpWebResponse)
{
// handle the response
}
Now I'm pretty sure my code in my service has completed because i tried to write something to database and that happened. I also checked the iis log on the virtual machine where the service is hosted and it showed http 200 is returned. But the call GetResponse() in my client code hanged until the 1000 seconds timeout is reached.
Update
There is a parameter to my web API which will affect how long the method runs.(I didn't show it in the above code for simplicity). If the service method runs for sufficiently long time (like 6 or 7 minutes), then the hanging problem will happen. Otherwise, the web response can successfully return. So i guess there might be some problem within the timeout setting. But there are several timeout properties in the HttpWebRequest and its base classes, i don't know what combination of them can result in or solve this problem.
Any ideas of what can the problem be?
Thanks a lot.
From your code, and since you're returning JsonResult from your action, it seems that you have a RESTful architecture. One of the ways to debug RESTful architectures is to use browser, specifically in your case where HTTP method is GET.
You can simply copy/paste your URL in the address bar of a browser to see if you get any response. If you see the result, then your client code is the problem, otherwise, the problem is in the server side (azure server).
To monitor HTTP traffic, a good utility is Fiddler.
Run your project locally in the Windows Azure emulator. Then, check the output of the compute emulator. It should show unhandled exceptions there.
It seems that the call to your external service fails, and the exception is not handled properly. We've had a similar case before, and it would even cause the IIS worker process to crash.