How to efficiently limit request length timeout on server side ? I'm using Microsoft.Owin.Host.HttpListener and there are cases when (due to call to external service) serving request takes ridiculous amount of time. This is not a problem - but web server should give up sooner than - well never (I did some tests, but after 5 minutes I stopped it).
Is there a way how to limit time for serving single request (similar to <httpRuntime maxRequestLength="..." /> in IIS ecosystem) ?
Sample controller code:
public async Task<HttpResponseMessage> Get() {
// ... calls to 3pty services here
await Task.Delay(TimeSpan.FromMinutes(5));
}
Starting web server:
WebApp.Start(this.listeningAddress, new Action<IAppBuilder>(this.Build));
Note: I've read about limiting http listener, but that just limits incoming request properties, it doesn't cancel request that is slow due to slow server processing:
var listener = appBuilder.Properties[typeof(OwinHttpListener).FullName] as OwinHttpListener;
var timeoutManager = listener.Listener.TimeoutManager;
timeoutManager.DrainEntityBody = TimeSpan.FromSeconds(20);
timeoutManager.EntityBody = TimeSpan.FromSeconds(20);
timeoutManager.HeaderWait = TimeSpan.FromSeconds(20);
timeoutManager.IdleConnection = TimeSpan.FromSeconds(20);
timeoutManager.RequestQueue = TimeSpan.FromSeconds(20);
Related:
https://github.com/aspnet/AspNetKatana/issues/152
Conceptually "older" web server solutions - i.e. IIS are using one-thread-per-request separation and ThreadAbortException to kill slow requests. Owin is using different philosophy - i.e. it fires new task per request and forcibly cancelling task is best avoided. There are two sides of this problem:
shus client away if it takes too long
cancel server processing if it takes too long
Both can be achieved using middleware component. There also is a cancellation token provided directly by owin infrastructure for cases when client disconnects (context.Request.CallCancelled where context is IOwinContext)
If you're interested only in cancelling server flow ASAP when it takes to long, I'd recommend something like
public class MyMiddlewareClass : OwinMiddleware
{
// 5 secs is ok for testing, you might want to increase this
const int WAIT_MAX_MS = 5000;
public MyMiddlewareClass(OwinMiddleware next) : base(next)
{
}
public override async Task Invoke(IOwinContext context)
{
using (var source = CancellationTokenSource.CreateLinkedTokenSource(
context.Request.CallCancelled))
{
source.CancelAfter(WAIT_MAX_MS);
// combined "client disconnected" and "it takes too long" token
context.Set("RequestTerminated", source.Token);
await Next.Invoke(context);
}
}
}
And then in controller
public async Task<string> Get()
{
var context = this.Request.GetOwinContext();
var token = context.Get<CancellationToken>("RequestTerminated");
// simulate long async call
await Task.Delay(10000, token);
token.ThrowIfCancellationRequested();
return "Hello !";
}
Shusing the client away is more complex. The middleware will look like this:
public static async Task ShutDownClientWhenItTakesTooLong(IOwinContext context,
CancellationToken timeoutToken)
{
await Task.Delay(WAIT_MAX_MS, timeoutToken);
if (timeoutToken.IsCancellationRequested)
{
return;
}
context.Response.StatusCode = (int)HttpStatusCode.ServiceUnavailable;
}
public async Task ExecuteMainRequest(IOwinContext context,
CancellationTokenSource timeoutSource, Task timeoutTask)
{
try
{
await Next.Invoke(context);
}
finally
{
timeoutSource.Cancel();
await timeoutTask;
}
}
public override async Task Invoke(IOwinContext context)
{
using (var source = CancellationTokenSource.CreateLinkedTokenSource(
context.Request.CallCancelled))
using (var timeoutSource = new CancellationTokenSource())
{
source.CancelAfter(WAIT_MAX_MS);
context.Set("RequestTerminated", source.Token);
var timeoutTask = ShutDownClientWhenItTakesTooLong(context, timeoutSource.Token);
await Task.WhenAny(
timeoutTask,
ExecuteMainRequest(context, timeoutSource, timeoutTask)
);
}
}
Related
I try to understand solution of this channel CancellationTokenSource callback is called for all request
In dotnet 5
I have the code bellow
public class AppTimeout
{
...
public AppTimeout(RequestDelegate next, IOptions<ServerOptions> options)
{
...
}
public async Task Invoke(HttpContext httpContext)
{
var cancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(httpContext.RequestAborted);
cancellationTokenSource.CancelAfter(_timeout);
httpContext.RequestAborted = cancellationTokenSource.Token;
var requestPath = httpContext.Request.Path;
cancellationTokenSource.Token.Register(() =>
{
logger.Message("... requestPath...");
});
await _next(httpContext);
}
}
After on request have been in timeout, The callback will be triggers for all request
But it is not case and the callback will be called only for the request in timeout just if I replace
cancellationTokenSource.Token.Register(() =>
by
using var registration = cancellationTokenSource.Token.Register(() =>
My question is why ?
Could you explain me please?
Without using statements both cancellationTokenSource and registration will continue to be active after the requests ends and eventually you will get a timeout where the callback is invoked even though the request already ended. The timeout is not a timeout of the request. It's a result of a bug in the code. I'm not sure I understand why this only starts to occur after a "real" timeout but I wouldn't waste energy trying to understand that. You have to dispose registration and I believe also cancellationTokenSource when the requests ends (e.g. by using using statements).
I have an Angular application running on top of the .NET MVC application (.NET Framework 4.8) which sends the http requests to the API (also .NET Framework 4.8) and from there, the requests are sent to other APIs. I want to be able to pass the cancellation token to all those APIs, when the request is cancelled in the browser.
In Angular I'm using switchMap operator but I also tried simple unsubscribe and in both cases it works the same way - cancellation token is correct in the MVC app, but it doesn't seem to be passed to the APIs. Here is my code (simplified):
Angular
private request$ = new Subject();
private response$ = this.request$.pipe(switchMap(() => this.apiService.getData()));
// in other part of the code, I'm subscribing response$ and triggering the next request
// when needed (sometimes previous request is not completed yet).
// This successfully cancels request in browser.
this.request$.next()
MVC
[HttpGet]
public async Task<SomeClass> Get(string url, CancellationToken ct)
{
using (HttpClient client = new HttpClient())
{
var request = new HttpRequestMessage
{
Method = HttpMethod.Get,
RequestUri = new System.Uri(url)
};
// this part works fine i.e. IsCancellationRequested is set
// to true when request is cancelled
if(!cancellationToken.IsCancellationRequested) {
var response = await httpClient.SendAsync(proxyRequest, HttpCompletionOption.ResponseContentRead, ct);
// ...
// more code here
// ...
}
}
}
API
[HttpGet]
public async Task<IHttpActionResult> GetData()
{
CancellationToken cancellationToken = Request.GetOwinContext().Request.CallCancelled;
if(!cancellationToken.IsCancellationRequested) {
// get data, make request to another API, process and return data
}
}
I have also tried following in the API
[HttpGet]
public async Task<IHttpActionResult> GetData(CancellationToken cancellationToken)
{
if(!cancellationToken.IsCancellationRequested) {
// get data, make request to another API, process and return data
}
}
But the cancellationToken.IsCancellationRequested is always false in APIs (unless the 45s timeout happens). Any ideas how to resolve this?
I created a timeout middleware that works basically like this:
public async Task InvokeAsync(HttpContext httpContext)
{
var stopwatch = Stopwatch.StartNew();
using (var timeoutTS = CancellationTokenSource.CreateLinkedTokenSource(httpContext.RequestAborted))
{
var delayTask = Task.Delay(config.Timeout);
var res = await Task.WhenAny(delayTask, _next(httpContext));
Trace.WriteLine("Time taken = " + stopwatch.ElapsedMilliseconds);
if (res == delayTask)
{
timeoutTS.Cancel();
httpContext.Response.StatusCode = 408;
}
}
}
In order to test it, I created a controller action:
[HttpGet]
public async Task<string> Get(string timeout)
{
var result = DateTime.Now.ToString("mm:ss.fff");
if (timeout != null)
{
await Task.Delay(2000);
}
var rng = new Random();
result = result + " - " + DateTime.Now.ToString("mm:ss.fff");
return result;
}
The configured timeout to 500ms and the Time Taken reported is usually 501-504 ms (which is a very acceptable skid).
The problem is that every now and then I was seeing an error on the output windows saying that the response had already started. And I thought to myself: this cant be! this is happening 1 second earlier than the end of the Task.Delay on the corresponding controller.
So I opened up fiddler and (to my surprise) several requests are returning in 1.3-1.7 seconds WITH A FULL RESPONSE BODY.
By comparing the reported time written on the response body with the timestamp on fiddler "statistic" tab I can guarantee that the response im looking at does not belong to that request at hand!
Does anyone knows what's going on? Why is this "jumbling" happening?
Frankly, you're not using middleware in the way it is designed for.
You might want to read this middleware docs.
The ASP.NET Core request pipeline consists of a sequence of request delegates, called one after the other.
In your case, your middleware is running in parallel with the next middleware.
When a middleware short-circuits, it's called a terminal middleware because it prevents further middleware from processing the request.
If I understand you correctly, you might want to create such terminal middleware, but clearly your current one is not.
In your case, you have invoked the _next middleware, which means the request has already handed off to the next middleware in the request pipeline. The subsequent middleware components can start the response before the timeout has elapsed. i.e. a race condition between your middleware and a subsequent middleware.
To avoid the race condition, you should always check HasStarted before assigning the status code. And if the response has started, all you can do might only be aborting the request if you don't want the client to wait for too long.
static void ResetOrAbort(HttpContext httpContext)
{
var resetFeature = httpContext.Features.Get<IHttpResetFeature>();
if (resetFeature is not null)
{
resetFeature.Reset(2);
}
else
{
httpContext.Abort();
}
}
app.Use(next =>
{
return async context =>
{
var nextTask = next(context);
var t = await Task.WhenAny(nextTask, Task.Delay(100));
if (t != nextTask)
{
var response = context.Response;
// If response has not started, return 408
if (!response.HasStarted)
{
// NOTE: you will still get same exception
// because the above check does not eliminate
// any race condition
try
{
response.StatusCode = StatusCodes.Status408RequestTimeout;
await response.StartAsync();
}
catch
{
ResetOrAbort(context);
}
}
// Otherwise, abort the request
else
{
ResetOrAbort(context);
}
}
};
});
What I'm wanting to implement is a HttpClient that can get a new bearer token whenever a 401 is thrown by the server. The token expires every 30 minutes and I would like to just update that, but modifying the headers isn't threadsafe (as far as I'm aware) and will cause issues if one thread tries to update the authorisation whilst another is trying to make a request.
public class MyCustomClient : HttpClient
{
public MyCustomClient(Configuration config)
:base()
{
this.BaseAddress = new Uri(config.ServiceUrl);
this.Timeout = new TimeSpan(0, 2, 0);
this.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
}
public async Task UpdateBearerTokenAsync()
{
var token = "123" // assume this was retrieved from server successfully
this.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
}
}
What I ideally need is for only one re-authorization to be going on at once, and for all other calls to wait if the client is currently trying to re-authenticate.
The first part I think I can do by using a SemaphoreSlim
private readonly SemaphoreSlim _bearerTokenSemaphore;
public MyCustomClient(Configuration config)
:base()
{
_bearerTokenSemaphore = new SemaphoreSlim(0, 1);
_bearerTokenSemaphore.Release();
this.BaseAddress = new Uri(config.ServiceUrl);
this.Timeout = new TimeSpan(0, 2, 0);
this.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
}
public async Task<bool> UpdateBearerTokenAsync()
{
if (!_bearerTokenSemaphore.Wait(0))
{
return false;
}
try
{
var token = "123" // assume this was retrieved from server successfully using an asynchronous method
this.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
return true;
}
finally { _bearerTokenSemaphore.Release(); }
}
However I'm not sure of a good way of making all other calls wait if authorisation is currently occurring. I tried something like
public new async Task<HttpResponseMessage> GetAsync(string uri)
{
await _bearerTokenSemaphore.WaitAsync();
_bearerTokenSemaphore.Release();
return await base.GetAsync(uri);
}
But this just seemed to deadlock
EDIT:
The above didn't work because I never released the sempahore after instantiating it (woops). So this does seem to work now, but I am still open to better suggestions as I'm really not happy with this implementation
I create a bot, called picturesaver, using Microsoft's Bot Framework, I added a GroupMe channel, and I have it hosted in Azure. The bot works perfectly, saving pictures to Google Drive.
However, the bot gives an error saying "Service Error:POST to picturesaver timed out after 15s" Is it possible to extend the timeout time? Or even stop the bot from posting anything at all. Could this be an Azure issue or is it a GroupMe issue?
If your bot performs an operation that takes longer than 15 seconds to process a message, you can process the message on another thread, and acknowledge the call right away. Something like:
public async Task<HttpResponseMessage> Post([FromBody]Activity activity)
{
if (activity.Type == ActivityTypes.Message)
{
if ([determine if this will take > 15s])
{
// process the message asyncronously
Task.Factory.StartNew(async () => await Conversation.SendAsync(activity, () => new Dialogs.RootDialog()));
}
else
{
//process the message normally
await Conversation.SendAsync(activity, () => new Dialogs.RootDialog());
}
}
return Request.CreateResponse(HttpStatusCode.OK); //ack the call
}
This will avoid the 15 second timeout between connector and bot.
Edit: the above will not scale, and is just using a Task.Factory. Please refer to https://learn.microsoft.com/en-us/azure/bot-service/bot-builder-howto-long-operations-guidance for the recommended guidance on processing long operations from a bot.
The Bot Connector service has a 15s timeout so you need to make sure any async API calls are handled in that timeframe, or make sure your bot responds with some kind of message if it's waiting for some other operation to complete. Currently the 15s timeout cannot be modified.
The solution to process the message on another thread, and acknowledge the call right away is good only for a bot on an App Service.
But as for a Functions Bot doing so will finish the Azure Function if I immediately return from this method.
I tried it. The Azure Function stops running, and the real response to the chat never comes. So it's not a solution at all for the Function Bots.
I ended up with this code for a Functions Bot, which resolves this problem.
Using Azure Queues
public static class Functions
{
[FunctionName("messages")]
[return: Queue("somequeue")]
public static async Task<MessagePayload> Messages([HttpTrigger
(WebHookType = "genericJson")]HttpRequestMessage req) =>
// return from this Azure Function immediately to avoid timeout warning message
// in the chat.
// just put the request into "somequeue".
// We can't pass the whole request via the Queue, so pass only what we need for
// the message to be processed by Bot Framework
new MessagePayload
{
RequestUri = req.RequestUri,
Content = await req.Content.ReadAsStringAsync(),
AuthScheme = req.Headers.Authorization.Scheme,
AuthParameter = req.Headers.Authorization.Parameter
};
// Do the actual message processing in another Azure Function, which is
// triggered by a message enqueued in the Azure Queue "somequeue"
[FunctionName("processTheMessage")]
public static async Task ProcessTheMessage([QueueTrigger("somequeue")]
MessagePayload payload, TraceWriter logger)
{
// we don't want the queue to process this message 5 times if it fails,
// so we won't throw any exceptions here at all, but we'll handle them properly.
try
{
// recreate the request
var request = new HttpRequestMessage
{
Content = new StringContent(payload.Content),
RequestUri = payload.RequestUri
};
request.Headers.Authorization = new
AuthenticationHeaderValue(payload.AuthScheme, payload.AuthParameter);
// initialize dependency injection container, services, etc.
var initializer = new SomeInitializer(logger);
initializer.Initialize();
// handle the request in a usual way and reply back to the chat
await initializer.HandleRequestAsync(request);
}
catch (Exception ex)
{
try
{
// TODO: handle the exception
}
catch (Exception anotherException)
{
// swallow any exceptions in the exceptions handler?
}
}
}
}
[Serializable]
public class MessagePayload
{
public string Content { get; set; }
public string AuthParameter { get; set; }
public string AuthScheme { get; set; }
public Uri RequestUri { get; set; }
}
(Be sure to use different Azure Queues for local development with Bot Framework emulator and for a cloud-deployed Function App. Otherwise, the messages sent to your bot from real customers may be processed locally while you are debugging on your machine)
Using an HTTP request
Of course, the same can be done without using an Azure Queue with a direct call to another Azure Function's public URL - https://<my-bot>.azurewebsites.net/api/processTheMessage?code=<function-secret>. This call has to be done on another thread, without waiting for the result in the messages function.
[FunctionName("messages")]
public static async Task Run([HttpTrigger(WebHookType = "genericJson")]
HttpRequestMessage req)
{
// return from this Azure Function immediately to avoid timeout warning message
// in the chat.
using (var client = new HttpClient())
{
string secret = ConfigurationManager.AppSettings["processMessageHttp_secret"];
// change the RequestUri of the request to processMessageHttp Function's
// public URL, providing the secret code, stored in app settings
// with key 'processMessageHttp_secret'
req.RequestUri = new Uri(req.RequestUri.AbsoluteUri.Replace(
req.RequestUri.PathAndQuery, $"/api/processMessageHttp?code={secret}"));
// don't 'await' here. Simply send.
#pragma warning disable CS4014
client.SendAsync(req);
#pragma warning restore CS4014
// wait a little bit to ensure the request is sent. It will not
// send the request at all without this line, because it would
// terminate this Azure Function immediately
await Task.Delay(500);
}
}
[FunctionName("processMessageHttp")]
public static async Task ProcessMessageHttp([HttpTrigger(WebHookType = "genericJson")]
HttpRequestMessage req,
Microsoft.Extensions.Logging.ILogger log)
{
// first and foremost: initialize dependency
// injection container, logger, services, set default culture/language, etc.
var initializer = FunctionAppInitializer.Initialize(log);
// handle the request in a usual way and reply back to the chat
await initializer.HandleRequest(req);
}