I have a Background Worker implementing the BackgroundService (provided by MS).
See this simple implementation:
public class MyService : BackgroundService {
private readonly MyDbContext _context;
public MyService(MyDbContext context) {
//...
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try {
while (true)
{
stoppingToken.ThrowIfCancellationRequested();
// Do some work
}
} catch(OperationCancelledException) {
_context.Add(new MyLogMessage(){ Error = "MyService cancelled!" });
_context.SaveChanges();
}
// ...
}
}
When the graceful shutdown (in console: CTRL+C) is requested the catch block is triggered, and also the SaveChanges() seems to be executed. But, sometimes the error is stored into the database and the most of the time it is not. Also the EntityFramework is printing an insert statement on the console, but the log is not in the db.
I assume that the shutdown is happening faster then writting the data to the DB?
Can anyone give me a hint how to handle this situation and store the error into the database?
It seems like the stoppingToken isn't cancelled as expected when the application shuts down. I managed to get around this using IHostApplicationLifetime and a new field where I can store if a shutdown is in progress.
public class TestService : BackgroundService {
private readonly IHostApplicationLifetime _lifetime;
private readonly ILogger<TestService> _logger;
private bool _shutownRequested;
public TestService(IHostApplicationLifetime lifetime, ILogger<TestService> logger) {
_lifetime = lifetime;
_logger = logger;
}
public override Task StartAsync(CancellationToken cancellationToken) {
_lifetime.ApplicationStopping.Register(OnShutdown);
return Task.CompletedTask;
}
private void OnShutdown() {
_shutdownRequested = true;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
try {
while(true) {
stoppingToken.ThrowIfCancellationRequested();
if(_shutdownRequested) {
throw new OperationCanceledException();
}
await Task.Delay(100, CancellationToken.None);
}
} catch(OperationCanceledException) {
_logger.LogWarning("TestService canceled");
}
}
}
Now it might be better to now throw a new exception there, but as an example it will do.
The reason why the log entry doesn't appear in the database is that the host shutdown period is lower than what it takes to process a task in a while loop and send a log to the database. The default timeout is 5 seconds.
What you could do, is to increase the timeout to a larger value, for example a minute a two:
services.Configure<HostOptions>(
opts => opts.ShutdownTimeout = TimeSpan.FromMinutes(2));
Make sure to let enough time for a service to finish the iteration inside a while loop and log the message.
Please check Extending the shutdown timeout setting to ensure graceful IHostedService shutdown for more details.
Related
I have an example for Windows service in .NET core 3.1.
I am generating an intentional error to see its behavior, my idea is that when the exception occurs I want to call the StopAsync method to stop the service, however once the StopAsync is executed the ExecuteAsync method is executed again and the service does not stop, entering like this in an infinite loop.
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await Task.CompletedTask;
while (!stoppingToken.IsCancellationRequested)
{
try
{
int num = 233;
int result = num / 0;
//todo
}
catch (Exception ex)
{
_logger.LogError(ex.Message);
await StopAsync(stoppingToken); //this execute, but it keeps running this ExecuteAsync method
}
}
}
public override async Task StopAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("stop service");
await base.StopAsync(cancellationToken);
}
the ExecuteAsync method is executed again and the service does not stop, entering like this in an infinite loop.
Are you sure about that? StopAsync will cancel the token provided to ExecuteAsync, and it looks like your ExecuteAsync method will complete in that case.
The host application itself will continue running, though; perhaps that's what you're seeing. To shut down the host application when the background service exits, you need to explicitly stop the application:
private readonly IHostApplicationLifetime _hostApplicationLifetime;
public MyBackgroundService(IHostApplicationLifetime hostApplicationLifetime) =>
_hostApplicationLifetime = hostApplicationLifetime;
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
...
}
finally
{
_hostApplicationLifetime.StopApplication();
}
}
Recently I asked a question on StackOverflow about long running background Tasks in asp.net core. Since then I have tried everything from here https://learn.microsoft.com/cs-cz/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-5.0&tabs=visual-studio and if i do what I do it will just stop at some time. And it will stop even if I wrap it in IServiceProvider.CreateScope and await it. The only thing I still didn't try and I'm trying to avoid it, is creating dedicated .net application that would just read queue and do what it's supposed to do. And also I thing that it's overkill to create queue for it, I just want to run it in background asynchronously but it just stops. Sorry if it's some stupid bug but this is my first asp.net project and I'm fixing this problem for week now.
This is Queue version
public class QueuedHostedService : BackgroundService
{
private readonly IServiceProvider _serviceProvider;
private readonly IBackgroundTaskQueue _queue;
private readonly ILogger<QueuedHostedService> _logger;
public QueuedHostedService(IServiceProvider serviceProvider, IBackgroundTaskQueue queue, ILogger<QueuedHostedService> logger)
{
_serviceProvider = serviceProvider;
_queue = queue;
_logger = logger;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await BackgroundProcessing(stoppingToken);
}
private async Task BackgroundProcessing(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var workItem = await _queue.DequeueAsync(stoppingToken);
try
{
var scope = _serviceProvider.CreateScope();
var scrapeUrl = scope.ServiceProvider.GetRequiredService<IScopedScrapeUrl>();
// The scrape Sound Cloud Task is taking hours
await scrapeUrl.ScrapeSoundCloud(workItem);
}catch(Exception ex)
{
_logger.LogError($"Error occurred executing {nameof(workItem)},\n{ex}");
}
}
}
public override async Task StopAsync(CancellationToken stoppingToken)
{
_logger.LogInformation("Consume Scoped Scrape Url Hosted Service is stopping.");
await base.StopAsync(stoppingToken);
}
public class BackgroundScrapeQueue : IBackgroundTaskQueue
{
private readonly Channel<Scrape> _queue;
private readonly ILogger<BackgroundScrapeQueue> _logger;
public BackgroundScrapeQueue(ILogger<BackgroundScrapeQueue> logger)
{
var options = new BoundedChannelOptions(100)
{
FullMode = BoundedChannelFullMode.Wait
};
_queue = Channel.CreateBounded<Scrape>(options);
_logger = logger;
}
public async ValueTask<Scrape> DequeueAsync(CancellationToken stoppingToken)
{
var workItem = await _queue.Reader.ReadAsync(stoppingToken);
return workItem;
}
public async ValueTask QueueBackgroundWorkItemAsync(Scrape scrape)
{
if(scrape == null)
{
_logger.LogError("Invalid Scrape for queue");
return;
}
await _queue.Writer.WriteAsync(scrape);
}
}
public interface IBackgroundTaskQueue
{
ValueTask QueueBackgroundWorkItemAsync(Scrape scrape);
ValueTask<Scrape> DequeueAsync(CancellationToken stoppingToken);
}
it will just stop at some time. And it will stop even if I wrap it in IServiceProvider.CreateScope and await it.
Yes. That's the problem with in-memory background services. They can be stopped at any time, because they're hosted in an ASP.NET process that determines it's safe to shut down when requests are complete. ASP.NET will actually request a shutdown and then wait for a while for the services to complete, but there's also a timer where they'll be forced out if they don't complete within 10 minutes or so.
The bottom line is that shutdowns are normal. Any code that assumes it can run indefinitely in ASP.NET is inherently buggy.
The only thing I still didn't try and I'm trying to avoid it, is creating dedicated .net application that would just read queue and do what it's supposed to do.
That is the only reliable solution.
I'm trying to start a background task on demand, whenever I receive a certain request from my api end point. All the task does is sending an email, delayed by 30 seconds. So I though BackgroundService would fit. But the problem is it looks like the BackgroundService is mostly for recurring tasks, and not to be executed on demand per this answer.
So what other alternatives I have, im hoping not to have to rely on 3rd parties libraries like Hangfire? I'm using asp.net core 3.1.
This is my background service.
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace ProjectX.Services {
public class EmailOfflineService : BackgroundService {
private readonly ILogger<EmailOfflineService> log;
private readonly EmailService emailService;
public EmailOfflineService(
ILogger<EmailOfflineService> log,
EmailService emailService
) {
this.emailService = emailService;
this.log = log;
}
protected async override Task ExecuteAsync(CancellationToken stoppingToken)
{
log.LogDebug("Email Offline Service Starting...");
stoppingToken.Register(() => log.LogDebug("Email Offline Service is stopping."));
while(!stoppingToken.IsCancellationRequested)
{
// wait for 30 seconds before sending
await Task.Delay(1000 * 30, stoppingToken);
await emailService.EmailOffline();
// End the background service
break;
}
log.LogDebug("Email Offline Service is stoped.");
}
}
}
You could try to combine an async queue with BackgroundService.
public class BackgroundEmailService : BackgroundService
{
private readonly IBackgroundTaskQueue _queue;
public BackgroundEmailService(IBackgroundTaskQueue queue)
{
_queue = queue;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var job = await _queue.DequeueAsync(stoppingToken);
_ = ExecuteJobAsync(job, stoppingToken);
}
}
private async Task ExecuteJobAsync(JobInfo job, CancellationToken stoppingToken)
{
try
{
await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
// todo send email
}
catch (Exception ex)
{
// todo log exception
}
}
}
public interface IBackgroundTaskQueue
{
void EnqueueJob(JobInfo job);
Task<JobInfo> DequeueAsync(CancellationToken cancellationToken);
}
This way you may inject IBackgroundTaskQueue inside your controller and enqueue jobs into it while JobInfo will contain some basic information for executing the job in background, e.g.:
public class JobInfo
{
public string EmailAddress { get; set; }
public string Body { get; set; }
}
An example background queue (inspired by the ASP.NET Core documentation):
public class BackgroundTaskQueue : IBackgroundTaskQueue
{
private ConcurrentQueue<JobInfo> _jobs = new ConcurrentQueue<JobInfo>();
private SemaphoreSlim _signal = new SemaphoreSlim(0);
public void EnqueueJob(JobInfo job)
{
if (job == null)
{
throw new ArgumentNullException(nameof(job));
}
_jobs.Enqueue(job);
_signal.Release();
}
public async Task<JobInfo> DequeueAsync(CancellationToken cancellationToken)
{
await _signal.WaitAsync(cancellationToken);
_jobs.TryDequeue(out var job);
return job;
}
}
I think the simplest approach is to make a fire-and-forget call in the code of handling the request to send a email, like this -
//all done, time to send email
Task.Run(async () =>
{
await emailService.EmailOffline(emailInfo).ConfigureAwait(false); //assume all necessary info to send email is saved in emailInfo
});
This will fire up a thread to send email.
The code will return immediately to the caller.
In your EmailOffline method, you can include time-delay logic as needed.
Make sure to include error logging logic in it also, otherwise exceptions from EmailOffline may be silently swallowed.
P.S. -
Answer to Coastpear and FlyingV -
No need to concern the end of calling context. The job will be done on a separate thread, which is totally independent of the calling context.
I have used similar mechanism in production for a couple of years, zero problem so far.
If your site is not supper busy, and the work is not critical, this is the easiest solution.
Just make sure you catch and log error inside your worker (EmailOffline, in this example).
If you need more reliable solution, I'd suggest using a mature queue product like AWS SQS, do not bother to create one by yourself. It is not an easy job to create a really good queue system.
Use Hangfire, it's Background Methods functionality is great, and provides you with a nice dashboard for free:
https://docs.hangfire.io/en/latest/background-methods/index.html
I have a background task that runs every few seconds in my ASP.NET Core application. It returns the following error:
An exception occurred while iterating over the results of a query for context type 'ThaiLiveApi.Data.DataContext'.
System.InvalidOperationException: A second operation started on this context before a previous operation completed.
This is usually caused by different threads using the same instance of DbContext.
I would think because im using the dbcontext in scoped that this would not be possible and only use one dbcontext for that scope. Am I missing something?
Here is my worker code:
public class FacebookWorker : IHostedService
{
private readonly IServiceScopeFactory _serviceScopeFactory;
private readonly ILogger<FacebookWorker> _logger;
public FacebookWorker(IServiceScopeFactory serviceScopeFactory, ILogger<FacebookWorker> logger)
{
_serviceScopeFactory = serviceScopeFactory;
_logger = logger;
}
private readonly int JobIntervalInSecs = 5;
private Timer _timer;
public Task StartAsync(CancellationToken cancellationToken)
{
if (cancellationToken.IsCancellationRequested)
{
cancellationToken.ThrowIfCancellationRequested();
}
// Invoke the DoWork method every 5 seconds.
_timer = new Timer(callback: async o => await DoWork(o),
state: null, dueTime: TimeSpan.FromSeconds(0),
period: TimeSpan.FromSeconds(JobIntervalInSecs));
return Task.CompletedTask;
}
private async Task DoWork(object state) {
// allow only a certain number of concurrent work. In this case,
// only allow one job to run at a time.
if (State.numberOfActiveJobs < State.maxNumberOfActiveJobs) {
// Update number of running jobs in one atomic operation.
try {
Interlocked.Increment(ref State.numberOfActiveJobs);
_logger.LogInformation("Fetching comments " + DateTime.Now);
using (var scope = _serviceScopeFactory.CreateScope()) {
var facebookService = scope.ServiceProvider.GetRequiredService<IFacebookService>();
await facebookService.FetchAndHandleComments();
}
}
finally {
Interlocked.Decrement(ref State.numberOfActiveJobs);
}
}
}
}
}
I suppose I'm getting this error because of my Worker? Is it possible this is caused in another part of my code?
It happens when a process started but last process is not complete. try to use using block when call DbContext. One more thing Check your DbContext instance is it static or not.
I'm using mvc with .net core, I need to run a task on start, and stop it when application stops. In Startup.cs I registered events for application start and stop. The problem is, I don't know how to run a task that has to be run in a specific class from startup. The task looks like this:
public void PreventStatusChange()
{
while (forceStatusChange)
{
foreach (var ext in GetExtensions())
{
ext.Status = StatusType.Available;
}
Thread.Sleep(1000);
}
}
Variable forceStatusChange is declared in the same class, so I don't see it from my Startup.cs. What is the best way to do it?
You need to create a class that implements IHostedService. This interface defines just two methods, StartAsync which is called when the application starts and StopAsync which is called when it terminates.
You need to register it as a hosted service with :
services.AddHostedService<TimedHostedService>();
Be careful to use AddHostedService, NOT AddSingleton. If you use AddSingleton the runtime won't know to call StartAsync and StopAsync when appropriate.
The article Background tasks with hosted services in ASP.NET Core shows how to implement a service using a timer :
internal class TimedHostedService : IHostedService, IDisposable
{
private readonly ILogger _logger;
private Timer _timer;
public TimedHostedService(ILogger<TimedHostedService> logger)
{
_logger = logger;
}
public Task StartAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("Timed Background Service is starting.");
_timer = new Timer(DoWork, null, TimeSpan.Zero,
TimeSpan.FromSeconds(10));
return Task.CompletedTask;
}
private void DoWork(object state)
{
_logger.LogInformation("Timed Background Service is working.");
}
public Task StopAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("Timed Background Service is stopping.");
_timer?.Change(Timeout.Infinite, 0);
return Task.CompletedTask;
}
public void Dispose()
{
_timer?.Dispose();
}
}
There's nothing particularly interesting in this code - just start a timer when StartAsync is called and stop it on StopAsync
Cancelling long-running tasks
When the runtime needs to recycle or stop, it will call the StopAsync method on all hosted services, wait a while for them to finish gracefully and then warn them to cancel immediatelly. After a short while it will go ahead and terminate the app or recycle it.
The cancellationToken parameter is used to signal that the service should stop immediatelly. Typically, that means that you'd have to write your own code to check it, warn your own task to terminate, wait for all of them to finish etc, similar to the code shown in this article
This is pretty much boilerplate though, which is why the BackgroundService class can be used to create a class that only needs to implement ExecuteAsync(CancellationToken). Starting and stopping that task is provided by BackgroundService, eg :
public class PollingService : BackgroundService
{
private readonly ILogger _logger;
public PollingService(ILogger<PollingService> logger)
{
_logger = logger;
}
protected async override Task ExecuteAsync(
CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
try
{
await DoSomething(cancellationToken);
await Task.Delay(1000,cancellationToken);
}
catch (Exception ex)
{
_logger.LogError(ex,
$"Error occurred executing {nameof(workItem)}.");
}
}
_logger.LogInformation("Queued Hosted Service is stopping.");
}
}
In this case Task.Delay() itself will be cancelled as soon as the runtime raises the cancellation token. DoSomething() itself should be implemented in a way that checks the cancellation token, eg pass it to any asynchronous method that accepts it as a parameter, test the IsCancellationRequested property on every loop and exit it it's raised.
For example :
protected async override Task ExecuteAsync(
CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
try
{
foreach (var ext in GetExtensions())
{
//Oops, time to cancel
if(cancellationToken.IsCancellationRequested)
{
break;
}
//Otherwise, keep working
ext.Status = StatusType.Available;
}
await Task.Delay(1000,cancellationToken);
}
catch (Exception ex)
{
...
}
}
_logger.LogInformation("Hosted Service is stopping.");
}
You can use BackgroundService
public class LongRunningService : BackgroundService
{
public LongRunningService()
{
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested && forceStatusChange)
{
foreach (var ext in GetExtensions())
{
ext.Status = StatusType.Available;
}
await Task.Delay(1000, stoppingToken);
}
}
protected override async Task StopAsync (CancellationToken stoppingToken)
{
// Run your graceful clean-up actions
}
}
And register it:
public void ConfigureServices(IServiceCollection services)
{
...
services.AddSingleton<IHostedService, LongRunningService>();
...
}