I've created a simple webapi .net core 3.1 app.
I want to catch all unhandled exceptions.So I put this code according to the docs :
app.UseExceptionHandler(c => c.Run(async context =>
{
var exception = context.Features
.Get<IExceptionHandlerPathFeature>()
.Error;
var response = new { error = exception.Message };
log.LogDebug(exception.Message);
}));
This is my action:
[HttpGet]
public IActionResult Get()
{
throw new Exception("this is a test");
}
When this code runs, I do see that UseExceptionHandler is working.
But when my code in the action is :
[HttpGet]
public IActionResult Get()
{
Task.Run(async () =>
{
await Task.Delay(4000);
throw new Exception("this is a test");
});
return Ok();
}
Then UseExceptionHandler is NOT working.
However - the following code does catch the task's exception :
AppDomain.CurrentDomain.FirstChanceException += (sender, eventArgs) =>
{
Debug.WriteLine(eventArgs.Exception.ToString());
};
Question:
Why does the task exception isn't recognized by UseExceptionHandler?
How can I catch ALL types of exceptions? Should I rely only on AppDomain.CurrentDomain.FirstChanceException?
nb , I did disabled app.UseDeveloperExceptionPage();
To answer your questions.
Why does the task exception isn't recognized by UseExceptionHandler?
As already suggested in the comments, you cannot use UseExceptionHandler to catch exceptions initiated inside non-awaited tasks. UseExceptionHandler wraps your request in ASP.NET Core middleware. Once the action returns OK to the client, the middleware is no longer able to catch any exceptions happening in tasks started from within the action.
How can I catch ALL types of exceptions? Should I rely only on AppDomain.CurrentDomain.FirstChanceException?
You can catch exceptions globally and log them this way if you'd like. But I wouldn't recommend you to do it this way. The only reason you need to implement this event, is that you are starting tasks/threads inside your web requests. You have no way of knowing if these tasks are kept running (application restart, recycle, etc.). If you are looking to launch background tasks with ASP.NET Core, you should use Worker Services which is the intended way of doing this:
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<MyWorker>();
});
public class MyWorker : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
// Do work
}
catch (Exception e)
{
// Log it?
}
await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
}
}
}
The cause of this particular symptom is that Get is starting a fire-and-forget task that the server knows nothing about. The request will complete before the task even has a chance to execute, so the UseExceptionHandler middleware will never see any exceptions. This is no longer a fire-and-forget task.
The real problem though, is executing a long running task in the background. The built-in way to do this is using a Background Service. The docs show how to create timed and queued background service, that act as job queues.
It's equally easy, if not easier, to publish messages with the desired data from, eg a controller to the background service using, eg Channels. No need to create our own queue, when the BCL already has an asynchronous one.
The service could look like this :
public class MyService: BackgroundService
{
private readonly ChannelReader<T> _reader;
public QueuedBspService(MessageQueue<T> queue)
{
_reader = queue.Reader;
}
protected internal async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
await foreach (var msg in _reader.ReadAllAsync(stoppingToken))
{
try
{
//Process the message here
}
catch (Exception exc)
{
//Handle message-specific errors
}
}
}
catch (Exception exc)
{
//Handle cancellations and other critical errors
}
}
}
The MessageQueue<T> wraps the Channel, making it easier to inject it to both the BackgroundService and any publishers like eg, a Controller action:
public class MessageQueue<T>
{
private readonly Channel<T> _channel;
public ChannelReader<T> Reader => _channel;
public ChannelWriter<T> Writer => _channel;
public MessageChannel()
{
_channel = Channel.CreateBounded<T>(1);
}
}
I adjusted this code from a service that only allows a single operation at a time. That's a quick&dirty way of preventing controllers from making requests that can't be handled.
On the contolle side, this action will post a request to the queue if possible, and return a Busy response otherwise :
public class MyController
{
private readonly ChannelWriter<T> _writer;
public MyController(MessaggeQueue<T> queue)
{
_writer = queue.Writer;
}
[HttpPost]
[ProducesResponseType(StatusCodes.Status201Created)]
[ProducesResponseType(StatusCodes.Status503ServiceUnavailable)]
public async Task<ActionResult> Post(....)
{
var jobName="SomeJob";
var id=Guid.NewGuid();
var jobMsg=CreateMessage(id,...);
try
{
if (_writer.TryWrite(msg))
{
return CreatedAtAction("GetItem","Jobs",new {id});
}
else
{
return Problem(statusCode:(int) HttpStatusCode.ServiceUnavailable,detail:"Jobs in progress",title:"Busy");
}
}
catch (Exception exc)
{
_logger.LogError(exc,"Queueing {job} failed",jobName);
throw;
}
}
}
The Post action first checks if it can even post a job message. If it succeeds, it returns a 201 - Created response with a URL that could be checked eg to check the status of the jobs. return Created() could be used instead, but once you create a long running job, you also want to check its status.
If the channel is at capacity, the core returns 503 with an explanation
Related
I have a Background Worker implementing the BackgroundService (provided by MS).
See this simple implementation:
public class MyService : BackgroundService {
private readonly MyDbContext _context;
public MyService(MyDbContext context) {
//...
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try {
while (true)
{
stoppingToken.ThrowIfCancellationRequested();
// Do some work
}
} catch(OperationCancelledException) {
_context.Add(new MyLogMessage(){ Error = "MyService cancelled!" });
_context.SaveChanges();
}
// ...
}
}
When the graceful shutdown (in console: CTRL+C) is requested the catch block is triggered, and also the SaveChanges() seems to be executed. But, sometimes the error is stored into the database and the most of the time it is not. Also the EntityFramework is printing an insert statement on the console, but the log is not in the db.
I assume that the shutdown is happening faster then writting the data to the DB?
Can anyone give me a hint how to handle this situation and store the error into the database?
It seems like the stoppingToken isn't cancelled as expected when the application shuts down. I managed to get around this using IHostApplicationLifetime and a new field where I can store if a shutdown is in progress.
public class TestService : BackgroundService {
private readonly IHostApplicationLifetime _lifetime;
private readonly ILogger<TestService> _logger;
private bool _shutownRequested;
public TestService(IHostApplicationLifetime lifetime, ILogger<TestService> logger) {
_lifetime = lifetime;
_logger = logger;
}
public override Task StartAsync(CancellationToken cancellationToken) {
_lifetime.ApplicationStopping.Register(OnShutdown);
return Task.CompletedTask;
}
private void OnShutdown() {
_shutdownRequested = true;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
try {
while(true) {
stoppingToken.ThrowIfCancellationRequested();
if(_shutdownRequested) {
throw new OperationCanceledException();
}
await Task.Delay(100, CancellationToken.None);
}
} catch(OperationCanceledException) {
_logger.LogWarning("TestService canceled");
}
}
}
Now it might be better to now throw a new exception there, but as an example it will do.
The reason why the log entry doesn't appear in the database is that the host shutdown period is lower than what it takes to process a task in a while loop and send a log to the database. The default timeout is 5 seconds.
What you could do, is to increase the timeout to a larger value, for example a minute a two:
services.Configure<HostOptions>(
opts => opts.ShutdownTimeout = TimeSpan.FromMinutes(2));
Make sure to let enough time for a service to finish the iteration inside a while loop and log the message.
Please check Extending the shutdown timeout setting to ensure graceful IHostedService shutdown for more details.
I'm trying to start a background task on demand, whenever I receive a certain request from my api end point. All the task does is sending an email, delayed by 30 seconds. So I though BackgroundService would fit. But the problem is it looks like the BackgroundService is mostly for recurring tasks, and not to be executed on demand per this answer.
So what other alternatives I have, im hoping not to have to rely on 3rd parties libraries like Hangfire? I'm using asp.net core 3.1.
This is my background service.
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace ProjectX.Services {
public class EmailOfflineService : BackgroundService {
private readonly ILogger<EmailOfflineService> log;
private readonly EmailService emailService;
public EmailOfflineService(
ILogger<EmailOfflineService> log,
EmailService emailService
) {
this.emailService = emailService;
this.log = log;
}
protected async override Task ExecuteAsync(CancellationToken stoppingToken)
{
log.LogDebug("Email Offline Service Starting...");
stoppingToken.Register(() => log.LogDebug("Email Offline Service is stopping."));
while(!stoppingToken.IsCancellationRequested)
{
// wait for 30 seconds before sending
await Task.Delay(1000 * 30, stoppingToken);
await emailService.EmailOffline();
// End the background service
break;
}
log.LogDebug("Email Offline Service is stoped.");
}
}
}
You could try to combine an async queue with BackgroundService.
public class BackgroundEmailService : BackgroundService
{
private readonly IBackgroundTaskQueue _queue;
public BackgroundEmailService(IBackgroundTaskQueue queue)
{
_queue = queue;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var job = await _queue.DequeueAsync(stoppingToken);
_ = ExecuteJobAsync(job, stoppingToken);
}
}
private async Task ExecuteJobAsync(JobInfo job, CancellationToken stoppingToken)
{
try
{
await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
// todo send email
}
catch (Exception ex)
{
// todo log exception
}
}
}
public interface IBackgroundTaskQueue
{
void EnqueueJob(JobInfo job);
Task<JobInfo> DequeueAsync(CancellationToken cancellationToken);
}
This way you may inject IBackgroundTaskQueue inside your controller and enqueue jobs into it while JobInfo will contain some basic information for executing the job in background, e.g.:
public class JobInfo
{
public string EmailAddress { get; set; }
public string Body { get; set; }
}
An example background queue (inspired by the ASP.NET Core documentation):
public class BackgroundTaskQueue : IBackgroundTaskQueue
{
private ConcurrentQueue<JobInfo> _jobs = new ConcurrentQueue<JobInfo>();
private SemaphoreSlim _signal = new SemaphoreSlim(0);
public void EnqueueJob(JobInfo job)
{
if (job == null)
{
throw new ArgumentNullException(nameof(job));
}
_jobs.Enqueue(job);
_signal.Release();
}
public async Task<JobInfo> DequeueAsync(CancellationToken cancellationToken)
{
await _signal.WaitAsync(cancellationToken);
_jobs.TryDequeue(out var job);
return job;
}
}
I think the simplest approach is to make a fire-and-forget call in the code of handling the request to send a email, like this -
//all done, time to send email
Task.Run(async () =>
{
await emailService.EmailOffline(emailInfo).ConfigureAwait(false); //assume all necessary info to send email is saved in emailInfo
});
This will fire up a thread to send email.
The code will return immediately to the caller.
In your EmailOffline method, you can include time-delay logic as needed.
Make sure to include error logging logic in it also, otherwise exceptions from EmailOffline may be silently swallowed.
P.S. -
Answer to Coastpear and FlyingV -
No need to concern the end of calling context. The job will be done on a separate thread, which is totally independent of the calling context.
I have used similar mechanism in production for a couple of years, zero problem so far.
If your site is not supper busy, and the work is not critical, this is the easiest solution.
Just make sure you catch and log error inside your worker (EmailOffline, in this example).
If you need more reliable solution, I'd suggest using a mature queue product like AWS SQS, do not bother to create one by yourself. It is not an easy job to create a really good queue system.
Use Hangfire, it's Background Methods functionality is great, and provides you with a nice dashboard for free:
https://docs.hangfire.io/en/latest/background-methods/index.html
I work with some WIFI devices such as cameras.
The basic fellow that I implemented:
Someone presses a button.
The button calls my Web API endpoint.
My Web API end point calls one of the API's of camera (by HttpRequest).
Processing each request takes 5 second. And between each request should be 1 second delay. For instance, If you press the button 2 times with one second delay after each: First we expect 5 second for processing the first press, then one second delay and in the end we expect 5 second for the last process (second press).
To do that I am using Queued background tasks based on Fire and Forgot manner in .NetCore 3.1 project and it works fine when I am dealing with just one camera.
But the new requirement of the project is, The background task should handle multiple cameras. It means one queue per camera, and queues should work parallel based on the fellow that I described above.
For example if we have 2 devices camera-001 and camera-002 and 2 connected buttons btn-cam-001 and btn-cam-002, And the order of pressing(0.5sec delay after each press) : 2X btn-cam-001 and 1X btn-cam-002.
What really happens is FIFO. First the requests of btn-cam-001 will be processed and then btn-cam-002.
What I expect and need: Camera-002 should not wait to receive the request and the first requests towards both cameras 001/002 should be processed in a same time(Based on the exmaple). Like each camera has own queue and own process.
The question is how can I achieve that in .NetCore 3.1?
Appreciate any help.
My current background service:
public class QueuedHostedService : BackgroundService
{
public IBackgroundTaskQueue TaskQueue { get; }
private readonly ILogger _logger;
public QueuedHostedService(IBackgroundTaskQueue taskQueue, ILoggerFactory loggerFactory)
{
TaskQueue = taskQueue;
_logger = loggerFactory.CreateLogger<QueuedHostedService>();
}
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("Queued Hosted Service is starting.");
while (!cancellationToken.IsCancellationRequested)
{
var workItem = await TaskQueue.DequeueAsync(cancellationToken);
try
{
await workItem(cancellationToken);
}
catch (Exception exception)
{
_logger.LogError(exception, $"Error occurred executing {nameof(workItem)}.");
}
}
_logger.LogInformation("Queued Hosted Service is stopping.");
}
}
And the current BackgroundTaskQueue:
public class BackgroundTaskQueue : IBackgroundTaskQueue
{
private readonly SemaphoreSlim _signal = new SemaphoreSlim(0);
private readonly ConcurrentQueue<Func<CancellationToken, Task>> _workItems =
new ConcurrentQueue<Func<CancellationToken, Task>>();
public void QueueBackgroundWorkItem(Func<CancellationToken, Task> workItem)
{
if (workItem is null)
{
throw new ArgumentNullException(nameof(workItem));
}
_workItems.Enqueue(workItem);
_signal.Release();
}
public async Task<Func<CancellationToken, Task>> DequeueAsync(CancellationToken cancellationToken)
{
await _signal.WaitAsync(cancellationToken);
_workItems.TryDequeue(out var workItem);
return workItem;
}
}
My current endpoint:
[HttpPost("hit")]
public ActionResult TurnOnAsync([FromBody] HitRequest request, CancellationToken cancellationToken = default)
{
try
{
var camera = ConfigurationHelper.GetAndValidateCamera(request.Device, _configuration);
_taskQueue.QueueBackgroundWorkItem(async x =>
{
await _cameraRelayService.TurnOnAsync(request.Device, cancellationToken);
Thread.Sleep(TimeSpan.FromSeconds(1));
});
return Ok();
}
catch (Exception exception)
{
_logger.LogError(exception, "Error when truning on the lamp {DeviceName}.", request.Device);
return StatusCode(StatusCodes.Status500InternalServerError, exception.Message);
}
}
Instead of a single BackgroundTaskQueue you could have one per camera. You could store the queues in a dictionary, having the camera as the key:
public IDictionary<IDevice, IBackgroundTaskQueue> TaskQueues { get; }
Then in your end-point use the queue that is associated with the requested camera:
_taskQueues[camera].QueueBackgroundWorkItem(async x =>
is it possible to implement something similar to chain of responsibility pattern in .net core middleware which catches exceptions? Because I wanted to Handle exceptions globally and take them to their handlers.
Example
try
{
}
catch(CustomException1 ex)
{
}
catch(CustomException2 ex)
{
}
...
The middleware grows really fast and it will be hard to maintain later. I wanted to try{} catch(Exception e) { Handle(e); } and make Handlers for each Exception, for example handler for NullReference etc. I though about solution to take the exception by type and handle it in the handle() method in the specified handler.
I'm toying around with middleware, so in startup:
app.UseMiddleware<ErrorHandlingMiddleware>();
Middleware, I have one general exception handler, you could add many here (sample code, Sentry is an error log service...sentry.io):
public class ErrorHandlingMiddleware
{
private readonly RequestDelegate _next;
private readonly IHub _sentry;
public ErrorHandlingMiddleware(RequestDelegate next, IHub sentry)
{
_sentry = sentry;
_next = next;
}
public async Task Invoke(HttpContext context/* other dependencies */)
{
try
{
await _next(context).ConfigureAwait(false);
}
catch (Exception ex)
{
await HandleExceptionAsync(context, ex).ConfigureAwait(false);
}
}
private Task HandleExceptionAsync(HttpContext context, Exception exception)
{
var code = HttpStatusCode.InternalServerError; // 500 if unexpected
if (exception is ValueNotAcceptedException) code = HttpStatusCode.NotAcceptable;
/*if (exception is MyNotFoundException) code = HttpStatusCode.NotFound;
else if (exception is MyUnauthorizedException) code = HttpStatusCode.Unauthorized;
else if (exception is MyException) code = HttpStatusCode.BadRequest;*/
// send to Sentry.IO
_sentry.CaptureException(exception);
var result = JsonConvert.SerializeObject(new { error = exception.Message });
context.Response.ContentType = "application/json";
context.Response.StatusCode = (int)code;
return context.Response.WriteAsync(result);
}
Note adding a dependency in the constructor will make it a singleton, last the life-cycle of the app (in my case it's fine), or else add dependency in the Invoke.
You can create multiple exception handler IExceptionFilter. After create filters you can inject thats Mvc filters at startup.
Note: mvc filters get hit later than custom middleware.
Note: first added filter get hit last.
services.AddMvc(options =>
{
options.Filters.Add<GeneralExceptionFilter>();
options.Filters.Add<DbExceptionFilter>();
});
Note: if you decide filter should not handle exception, you should not throws exception. You should set ExceptionHandled false
public void OnException(ExceptionContext context)
{
...
context.ExceptionHandled = false;
}
You can also create .net core middlewares and inject it from startup. Concepts are pretty similar to mvc filters.
I have identified a bottleneck in my TCP application that I have simplified for the sake of this question.
I have a MyClient class, that represents when a client connects; also I have a MyWrapper class, that represents a client that fulfill some conditions. If a MyClientfulfill some conditions, it qualifies for wrapper.
I want to expose an method that allows the caller to await a MyWrapper, and that method should handle the negotiation and rejection of invalid MyClients:
public static async Task StartAccepting(CancellationToken token)
{
while (!token.IsCancellationRequested)
{
var wrapper = await AcceptWrapperAsync(token);
HandleWrapperAsync(wrapper);
}
}
Therefore AcceptWrapperAsync awaits a valid wrapper, and HandleWrapperAsync handles the wrapper asynchronously without blocking the thread, so AcceptWrapperAsync can get back to work as fast as it can.
How that method works internally is something like this:
public static async Task<MyWrapper> AcceptWrapperAsync(CancellationToken token)
{
while (!token.IsCancellationRequested)
{
var client = await AcceptClientAsync();
if (IsClientWrappable(client))
return new MyWrapper(client);
}
return null;
}
public static async Task<MyClient> AcceptClientAsync()
{
await Task.Delay(1000);
return new MyClient();
}
private static Boolean IsClientWrappable(MyClient client)
{
Thread.Sleep(500);
return true;
}
This code simulates that there is a client connection every second, and that it takes half a second to checkout if the connection is suitable for a wrapper. AcceptWrapperAsync loops till a valid wrapper is generated, and then returns.
This approach, that works well, has a flaw. During the time that IsClientWrappable is executing, no further clients can be accepted, creating a bottleneck when lot of clients are trying to connect at the same time. I am afraid that in real life, if the server goes down while having lot of clients connected, the going up is not gonna be nice because all of them will try to connect at the same time. I know that is very difficult to connect all of them at the same time, but I would like to speed up the connection process.
Making IsClientWrappable async, would just ensure that the executing thread is not blocked till the negotiation finishes, but the execution flow is blocked anyway.
How could I improve this approach to continuously accept new clients but still be able of awaiting a wrapper using AcceptWrapperAsync?
//this loop must never be blocked
while (!token.IsCancellationRequested)
{
var client = await AcceptClientAsync();
HandleClientAsync(client); //must not block
}
Task HandleClientAsync(Client client) {
if (await IsClientWrappableAsync(client)) //make async as well, don't block
await HandleWrapperAsync(new MyWrapper(client));
}
This way you move the IsClientWrappable logic out of the accept loop and into the background async workflow.
If you do not wish to make IsClientWrappable non-blocking, just wrap it with Task.Run. It is essential that HandleClientAsync does not block so that its caller doesn't either.
TPL Dataflow to the rescue. I have created a "producer/consumer" object with two queues that:
accepts inputs from "producer" and stores it in the "in" queue.
a internal asynchronous task read from the "in" queue and process the input in parallel with a given maximum degree of parallelism.
put the processed item in the "out" queue afterwards. Result or Exception.
accepts a consumer to await an item. Then can check if the processing was successful or not.
I have done some testing and it seems to work fine, I want to do more testing though:
public sealed class ProcessingResult<TOut>
where TOut : class
{
public TOut Result { get; internal set; }
public Exception Error { get; internal set; }
}
public abstract class ProcessingBufferBlock<TIn,TOut>
where TIn:class
where TOut:class
{
readonly BufferBlock<TIn> _in;
readonly BufferBlock<ProcessingResult<TOut>> _out;
readonly CancellationToken _cancellation;
readonly SemaphoreSlim _semaphore;
public ProcessingBufferBlock(Int32 boundedCapacity, Int32 degreeOfParalellism, CancellationToken cancellation)
{
_cancellation = cancellation;
_semaphore = new SemaphoreSlim(degreeOfParalellism);
var options = new DataflowBlockOptions() { BoundedCapacity = boundedCapacity, CancellationToken = cancellation };
_in = new BufferBlock<TIn>(options);
_out = new BufferBlock<ProcessingResult<TOut>>(options);
StartReadingAsync();
}
private async Task StartReadingAsync()
{
await Task.Yield();
while (!_cancellation.IsCancellationRequested)
{
var incoming = await _in.ReceiveAsync(_cancellation);
ProcessThroughGateAsync(incoming);
}
}
private async Task ProcessThroughGateAsync(TIn input)
{
_semaphore.Wait(_cancellation);
Exception error=null;
TOut result=null;
try
{
result = await ProcessAsync(input);
}
catch (Exception ex)
{
error = ex;
}
finally
{
if(result!=null || error!=null)
_out.Post(new ProcessingResult<TOut>() { Error = error, Result = result });
_semaphore.Release(1);
}
}
protected abstract Task<TOut> ProcessAsync(TIn input);
public void Post(TIn item)
{
_in.Post(item);
}
public Task<ProcessingResult<TOut>> ReceiveAsync()
{
return _out.ReceiveAsync();
}
}
So the example I used on the OP would be something like this:
public class WrapperProcessingQueue : ProcessingBufferBlock<MyClient, MyWrapper>
{
public WrapperProcessingQueue(Int32 boundedCapacity, Int32 degreeOfParalellism, CancellationToken cancellation)
: base(boundedCapacity, degreeOfParalellism, cancellation)
{ }
protected override async Task<MyWrapper> ProcessAsync(MyClient input)
{
await Task.Delay(5000);
if (input.Id % 3 == 0)
return null;
return new MyWrapper(input);
}
}
And then I could add MyClient objects to that queue as fast as I get them, they would be processed in parallel, and the consumer would await for the ones that pass the filter.
As I said, I want to do more testing but any feedback will be very welcomed.
Cheers.