Develop Reference

How to keep BackgroundService task receiving mqtt messages without Console.ReadLine()

In my ASP.Net Core 6 application, a BackgroundService task called MqttClientService runs a MQTTNet client that handles incoming mqqt messages and responds with a message to indicate it was successful.
I have gotten the sample console app from the MQTTNet repo to work using Console.ReadLine(), however this feels like a hack for my use case. Is there a better way to keep the BackgroundService handling incoming messages without restarting constantly?
There is an example with Asp.Net Core and MQTTNet version 3, but it uses handles implemented by interfaces rather than async events that the library now uses: the MQTTNet's Upgrading Guide.
Any information will be appreciated, thank you.
MqttClientService.cs in Services/
using MQTTnet;
using MQTTnet.Client;
using System.Text;
namespace MqttClientAspNetCore.Services
{
public class MqttClientService : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
await Handle_Received_Application_Message();
}
}
public static async Task Handle_Received_Application_Message()
{
var mqttFactory = new MqttFactory();
using (var mqttClient = mqttFactory.CreateMqttClient())
{
var mqttClientOptions = new MqttClientOptionsBuilder()
.WithTcpServer("test.mosquitto.org")
.Build();
// Setup message handling before connecting so that queued messages
// are also handled properly.
mqttClient.ApplicationMessageReceivedAsync += e =>
{
Console.WriteLine("### RECEIVED APPLICATION MESSAGE ###");
Console.WriteLine($"+ Payload = {Encoding.UTF8.GetString(e.ApplicationMessage.Payload)}");
// Publish successful message in response
var applicationMessage = new MqttApplicationMessageBuilder()
.WithTopic("keipalatest/1/resp")
.WithPayload("OK")
.Build();
mqttClient.PublishAsync(applicationMessage, CancellationToken.None);
Console.WriteLine("MQTT application message is published.");
return Task.CompletedTask;
};
await mqttClient.ConnectAsync(mqttClientOptions, CancellationToken.None);
var mqttSubscribeOptions = mqttFactory.CreateSubscribeOptionsBuilder()
.WithTopicFilter(f =>
{
f.WithTopic("keipalatest/1/post");
f.WithAtLeastOnceQoS();
})
.Build();
await mqttClient.SubscribeAsync(mqttSubscribeOptions, CancellationToken.None);
Console.WriteLine("MQTT client subscribed to topic.");
// The line below feels like a hack to keep background service from restarting
Console.ReadLine();
}
}
}
}
Program.cs
using MqttClientAspNetCore.Services;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHostedService<MqttClientService>();
var app = builder.Build();
// To check if web server is still responsive
app.MapGet("/", () =>
{
return "Hello World";
});
app.Run();

There's no need for Console.ReadLine or even the loop. The BackgroundService application code won't terminate when ExecuteAsync returns. If you want the application to terminate when ExecuteAsync terminates you have to actually tell it to through the IApplicationLifecycle interface.
I've found this the hard way the first time I tried using a Generic host for a command line tool. Which seemed to hang forever ....
ExecuteAsync can be used to set up the MQTT client and the event handler and just let them work. The code terminates only when StopAsync is called. Even then, this is done by signaling a cancellation token, not by aborting some worker thread.
The client itself can be built in the constructor, eg using configuration settings. Only ConnectAsync needs to be called in ExecuteAsync.
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await _client.ConnectAsync(_clientOptions, CancellationToken.None);
_logger.LogInformation("Connected");
await _client.SubscribeAsync(_subscriptionOptions, CancellationToken.None);
_logger.LogInformation("Subscribed");
}
The service code stops when StopAsync is called and the cancellation token is triggered. stoppingToken.Register could be used to call _client.DisconnectAsync when that happens, but Register doesn't accept an asynchronous delegate. A better option is to override StopAsync itself :
public virtual async Task StopAsync(CancellationToken cancellationToken)
{
await _client.DisconnectAsync();
await base.StopAsync(cancellationToken);
}
The constructor can create the client and register the message handler
public class MqttClientService : BackgroundService
{
ILogger<MqttClientService> _logger;
IMqttClient _client=client;
MqttClientOptions _clientOptions;
MqttSubscriptionOptions _subscriptionOptions;
string _topic;
public MqttClientService(IOptions<MyMqttOptions> options,
ILogger<MqttClientService> logger)
{
_logger=logger;
_topic=options.Value.Topic;
var factory = new MqttFactory();
_client = factory.CreateMqttClient();
_clientOptions = new MqttClientOptionsBuilder()
.WithTcpServer(options.Value.Address)
.Build();
_subscriptionOptions = factory.CreateSubscribeOptionsBuilder()
.WithTopicFilter(f =>
{
f.WithTopic(options.Value.Topic);
f.WithAtLeastOnceQoS();
})
.Build();
_client.ApplicationMessageReceivedAsync += HandleMessageAsync;
}
Received messages are handled by the HandleMessageAsync method :
async Task HandleMessageAsync(ApplicationMessageProcessedEventArgs e)
{
var payload=Encoding.UTF8.GetString(e.ApplicationMessage.Payload);
_logger.LogInformation("### RECEIVED APPLICATION MESSAGE ###\n{payload}",payload);
var applicationMessage = new MqttApplicationMessageBuilder()
.WithTopic(_topic)
.WithPayload("OK")
.Build();
await _client.PublishAsync(applicationMessage, CancellationToken.None);
_logger.LogInformation("MQTT application message is published.");
}
Finally, since BackgroundService implements IDisposable, we can use Dispose to dispose the _client instance :
public void Dispose()
{
Dispose(true);
}
protected virtual Dispose(bool disposing)
{
if(disposing)
{
_client.Dispose();
base.Dispose();
}
_client=null;
}

If your service has nothing else useful to do, it can just wait for the CancellationToken to fire:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
await Handle_Received_Application_Message(stoppingToken);
}
catch (OperationCanceledException) { }
}
public static async Task Handle_Received_Application_Message(CancellationToken cancellationToken)
{
...
Console.WriteLine("MQTT client subscribed to topic.");
await Task.Delay(Timeout.Infinite, cancellationToken);
}

c# asp.net core awaiting long running task stops after period of time

Recently I asked a question on StackOverflow about long running background Tasks in asp.net core. Since then I have tried everything from here https://learn.microsoft.com/cs-cz/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-5.0&tabs=visual-studio and if i do what I do it will just stop at some time. And it will stop even if I wrap it in IServiceProvider.CreateScope and await it. The only thing I still didn't try and I'm trying to avoid it, is creating dedicated .net application that would just read queue and do what it's supposed to do. And also I thing that it's overkill to create queue for it, I just want to run it in background asynchronously but it just stops. Sorry if it's some stupid bug but this is my first asp.net project and I'm fixing this problem for week now.
This is Queue version
public class QueuedHostedService : BackgroundService
{
private readonly IServiceProvider _serviceProvider;
private readonly IBackgroundTaskQueue _queue;
private readonly ILogger<QueuedHostedService> _logger;
public QueuedHostedService(IServiceProvider serviceProvider, IBackgroundTaskQueue queue, ILogger<QueuedHostedService> logger)
{
_serviceProvider = serviceProvider;
_queue = queue;
_logger = logger;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await BackgroundProcessing(stoppingToken);
}
private async Task BackgroundProcessing(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var workItem = await _queue.DequeueAsync(stoppingToken);
try
{
var scope = _serviceProvider.CreateScope();
var scrapeUrl = scope.ServiceProvider.GetRequiredService<IScopedScrapeUrl>();
// The scrape Sound Cloud Task is taking hours
await scrapeUrl.ScrapeSoundCloud(workItem);
}catch(Exception ex)
{
_logger.LogError($"Error occurred executing {nameof(workItem)},\n{ex}");
}
}
}
public override async Task StopAsync(CancellationToken stoppingToken)
{
_logger.LogInformation("Consume Scoped Scrape Url Hosted Service is stopping.");
await base.StopAsync(stoppingToken);
}
public class BackgroundScrapeQueue : IBackgroundTaskQueue
{
private readonly Channel<Scrape> _queue;
private readonly ILogger<BackgroundScrapeQueue> _logger;
public BackgroundScrapeQueue(ILogger<BackgroundScrapeQueue> logger)
{
var options = new BoundedChannelOptions(100)
{
FullMode = BoundedChannelFullMode.Wait
};
_queue = Channel.CreateBounded<Scrape>(options);
_logger = logger;
}
public async ValueTask<Scrape> DequeueAsync(CancellationToken stoppingToken)
{
var workItem = await _queue.Reader.ReadAsync(stoppingToken);
return workItem;
}
public async ValueTask QueueBackgroundWorkItemAsync(Scrape scrape)
{
if(scrape == null)
{
_logger.LogError("Invalid Scrape for queue");
return;
}
await _queue.Writer.WriteAsync(scrape);
}
}
public interface IBackgroundTaskQueue
{
ValueTask QueueBackgroundWorkItemAsync(Scrape scrape);
ValueTask<Scrape> DequeueAsync(CancellationToken stoppingToken);
}

it will just stop at some time. And it will stop even if I wrap it in IServiceProvider.CreateScope and await it.
Yes. That's the problem with in-memory background services. They can be stopped at any time, because they're hosted in an ASP.NET process that determines it's safe to shut down when requests are complete. ASP.NET will actually request a shutdown and then wait for a while for the services to complete, but there's also a timer where they'll be forced out if they don't complete within 10 minutes or so.
The bottom line is that shutdowns are normal. Any code that assumes it can run indefinitely in ASP.NET is inherently buggy.
The only thing I still didn't try and I'm trying to avoid it, is creating dedicated .net application that would just read queue and do what it's supposed to do.
That is the only reliable solution.

Running background task on demand in asp.net core 3.x

I'm trying to start a background task on demand, whenever I receive a certain request from my api end point. All the task does is sending an email, delayed by 30 seconds. So I though BackgroundService would fit. But the problem is it looks like the BackgroundService is mostly for recurring tasks, and not to be executed on demand per this answer.
So what other alternatives I have, im hoping not to have to rely on 3rd parties libraries like Hangfire? I'm using asp.net core 3.1.
This is my background service.
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace ProjectX.Services {
public class EmailOfflineService : BackgroundService {
private readonly ILogger<EmailOfflineService> log;
private readonly EmailService emailService;
public EmailOfflineService(
ILogger<EmailOfflineService> log,
EmailService emailService
) {
this.emailService = emailService;
this.log = log;
}
protected async override Task ExecuteAsync(CancellationToken stoppingToken)
{
log.LogDebug("Email Offline Service Starting...");
stoppingToken.Register(() => log.LogDebug("Email Offline Service is stopping."));
while(!stoppingToken.IsCancellationRequested)
{
// wait for 30 seconds before sending
await Task.Delay(1000 * 30, stoppingToken);
await emailService.EmailOffline();
// End the background service
break;
}
log.LogDebug("Email Offline Service is stoped.");
}
}
}

You could try to combine an async queue with BackgroundService.
public class BackgroundEmailService : BackgroundService
{
private readonly IBackgroundTaskQueue _queue;
public BackgroundEmailService(IBackgroundTaskQueue queue)
{
_queue = queue;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var job = await _queue.DequeueAsync(stoppingToken);
_ = ExecuteJobAsync(job, stoppingToken);
}
}
private async Task ExecuteJobAsync(JobInfo job, CancellationToken stoppingToken)
{
try
{
await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
// todo send email
}
catch (Exception ex)
{
// todo log exception
}
}
}
public interface IBackgroundTaskQueue
{
void EnqueueJob(JobInfo job);
Task<JobInfo> DequeueAsync(CancellationToken cancellationToken);
}
This way you may inject IBackgroundTaskQueue inside your controller and enqueue jobs into it while JobInfo will contain some basic information for executing the job in background, e.g.:
public class JobInfo
{
public string EmailAddress { get; set; }
public string Body { get; set; }
}
An example background queue (inspired by the ASP.NET Core documentation):
public class BackgroundTaskQueue : IBackgroundTaskQueue
{
private ConcurrentQueue<JobInfo> _jobs = new ConcurrentQueue<JobInfo>();
private SemaphoreSlim _signal = new SemaphoreSlim(0);
public void EnqueueJob(JobInfo job)
{
if (job == null)
{
throw new ArgumentNullException(nameof(job));
}
_jobs.Enqueue(job);
_signal.Release();
}
public async Task<JobInfo> DequeueAsync(CancellationToken cancellationToken)
{
await _signal.WaitAsync(cancellationToken);
_jobs.TryDequeue(out var job);
return job;
}
}

I think the simplest approach is to make a fire-and-forget call in the code of handling the request to send a email, like this -
//all done, time to send email
Task.Run(async () =>
{
await emailService.EmailOffline(emailInfo).ConfigureAwait(false); //assume all necessary info to send email is saved in emailInfo
});
This will fire up a thread to send email.
The code will return immediately to the caller.
In your EmailOffline method, you can include time-delay logic as needed.
Make sure to include error logging logic in it also, otherwise exceptions from EmailOffline may be silently swallowed.
P.S. -
Answer to Coastpear and FlyingV -
No need to concern the end of calling context. The job will be done on a separate thread, which is totally independent of the calling context.
I have used similar mechanism in production for a couple of years, zero problem so far.
If your site is not supper busy, and the work is not critical, this is the easiest solution.
Just make sure you catch and log error inside your worker (EmailOffline, in this example).
If you need more reliable solution, I'd suggest using a mature queue product like AWS SQS, do not bother to create one by yourself. It is not an easy job to create a really good queue system.

Use Hangfire, it's Background Methods functionality is great, and provides you with a nice dashboard for free:
https://docs.hangfire.io/en/latest/background-methods/index.html

.NET Core 2.2 Scheduled Jobs Prevent Concurrency on same job

This is my class to handle CronJobs. So far it runs fine on its own. However, how can I modify this so that when a job is already running, disallow the same job from running also?
I am not using any library for this.
public abstract class CronJob : IHostedService, IDisposable
{
private System.Timers.Timer timer;
private readonly CronExpression _expression;
private readonly TimeZoneInfo _timeZoneInfo;
protected CronJob(string cronExpression, TimeZoneInfo timeZoneInfo)
{
_expression = CronExpression.Parse(cronExpression, CronFormat.IncludeSeconds);
_timeZoneInfo = timeZoneInfo;
}
protected virtual async Task ScheduleJob(CancellationToken cancellationToken)
{
var next = _expression.GetNextOccurrence(DateTimeOffset.Now, _timeZoneInfo);
if (next.HasValue)
{
var delay = next.Value - DateTimeOffset.Now;
timer = new System.Timers.Timer(delay.TotalMilliseconds);
timer.Elapsed += async (sender, args) =>
{
timer.Stop(); // reset timer
await DoWork(cancellationToken);
await ScheduleJob(cancellationToken); // reschedule next
};
timer.Start();
}
await Task.CompletedTask;
}
public virtual async Task DoWork(CancellationToken cancellationToken)
{
await Task.Delay(5000, cancellationToken); // do the work
}
public virtual async Task StartAsync(CancellationToken cancellationToken)
{
await ScheduleJob(cancellationToken);
}
public virtual async Task StopAsync(CancellationToken cancellationToken)
{
timer?.Stop();
await Task.CompletedTask;
}
public virtual void Dispose()
{
timer?.Dispose();
}
}

You don't have to because .NET Core handles HostedServices as a Singleton. Unless when it comes to hosting multiple instance of the same project that contains this HostedService, then you would have to support multiple instances for your project on your own.
In your case,
This means that ScheduleJob only has its own instance and will never
have a deep copy of its own unless you run a separate instance of the
project that contains it

Migration to ASP CORE 3: how to migrate backround services? (service added through AddHostedService freezes web application startup)

After migration to ASP Core3 this line freezes the web app startup process (with VS debug: browser had pop up but the page loading never ends)
serviceCollection.AddHostedService<BackgroundService>();
It works in Core 2.
I can't see any breaking changes related to AddHostedService in ASP Core3 documentation:
https://learn.microsoft.com/en-us/aspnet/core/migration/22-to-30?view=aspnetcore-3.0&tabs=visual-studio
It seems like blocking background server StartAsync blocks the web app startup. May be something should be configured in WebHost to make StartAsync async again?
The background service code looks like:
public class MyBackgroundService : IHostedService
{
private readonly BackgroundServiceHandle backgroundServiceHandle;
private CancellationTokenSource tokenSource;
public MyBackgroundService (BackgroundServiceHandle backgroundServiceHandle) =>
this.backgroundServiceHandle = backgroundServiceHandle;
public async Task StartAsync(CancellationToken cancellationToken)
{
tokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
while (cancellationToken.IsCancellationRequested == false)
{
try
{
await Task.Delay(TimeSpan.FromSeconds(3), cancellationToken).ConfigureAwait(false);
// IMPORTANT: it seems that next line blocks the web app startup, but why this works in CORE 2?
var taskSettings = backgroundServiceHandle.Dequeue(tokenSource.Token);
// the work
}
catch (OperationCanceledException)
{
// execution cancelled
}
}
}
public Task StopAsync(CancellationToken cancellationToken)
{
tokenSource.Cancel();
return Task.CompletedTask;
}
}
public sealed class BackgroundServiceHandle : IDisposable
{
private readonly BlockingCollection<TaskSettings> blockingCollection;
public BackgroundServiceHandle() => blockingCollection = new BlockingCollection<TaskSettings>();
public void Enqueue(TaskSettings settings) => blockingCollection.Add(settings);
public TaskSettings Dequeue(CancellationToken token) => blockingCollection.Take(token);
public void Dispose()
{
blockingCollection.Dispose();
}
}

Moving base class from IHostedService to BackgroundService and moving StartAsync to
protected override async Task ExecuteAsync(CancellationToken cancellationToken) solves the problem