I am working on a Realtime Application where I have to consume messages from Kafka and process the message and create a status dictionary to display on webpage. The problem is that while Kafka is running as BackgroundService in my Application, The ControllerBase class is not working or say my app doesn't launch localhost:5000 or so.
using (var consumer = new ConsumerBuilder<string, string>(
(IEnumerable<KeyValuePair<string, string>>)configuration).Build())
{
consumer.Subscribe(topic);
try
{
var message = consumer.Consume(cts.Token);
string consumedMessage = result.Message.Value.ToString();
}
catch (OperationCanceledException)
{
// Ctrl-C was pressed.
}
finally
{
consumer.Close();
}
}
}
return Task.CompletedTask;
}
Running this following service class in Background as soon as i comment out the consume part the localhost:5000 launches and if consume is present it doesn't.
Your consumer needs to run in separate thread or it will block the main thread and thus the startup process.
Run your consumer in a function of the service, which you call asynchronously in the ExecuteAsync function.
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await Task.Factory.StartNew(
() =>
{
while (!stoppingToken.IsCancellationRequested)
{
YourConsumeFunction();
}
}
);
}
Related
In my ASP.Net Core 6 application, a BackgroundService task called MqttClientService runs a MQTTNet client that handles incoming mqqt messages and responds with a message to indicate it was successful.
I have gotten the sample console app from the MQTTNet repo to work using Console.ReadLine(), however this feels like a hack for my use case. Is there a better way to keep the BackgroundService handling incoming messages without restarting constantly?
There is an example with Asp.Net Core and MQTTNet version 3, but it uses handles implemented by interfaces rather than async events that the library now uses: the MQTTNet's Upgrading Guide.
Any information will be appreciated, thank you.
MqttClientService.cs in Services/
using MQTTnet;
using MQTTnet.Client;
using System.Text;
namespace MqttClientAspNetCore.Services
{
public class MqttClientService : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
await Handle_Received_Application_Message();
}
}
public static async Task Handle_Received_Application_Message()
{
var mqttFactory = new MqttFactory();
using (var mqttClient = mqttFactory.CreateMqttClient())
{
var mqttClientOptions = new MqttClientOptionsBuilder()
.WithTcpServer("test.mosquitto.org")
.Build();
// Setup message handling before connecting so that queued messages
// are also handled properly.
mqttClient.ApplicationMessageReceivedAsync += e =>
{
Console.WriteLine("### RECEIVED APPLICATION MESSAGE ###");
Console.WriteLine($"+ Payload = {Encoding.UTF8.GetString(e.ApplicationMessage.Payload)}");
// Publish successful message in response
var applicationMessage = new MqttApplicationMessageBuilder()
.WithTopic("keipalatest/1/resp")
.WithPayload("OK")
.Build();
mqttClient.PublishAsync(applicationMessage, CancellationToken.None);
Console.WriteLine("MQTT application message is published.");
return Task.CompletedTask;
};
await mqttClient.ConnectAsync(mqttClientOptions, CancellationToken.None);
var mqttSubscribeOptions = mqttFactory.CreateSubscribeOptionsBuilder()
.WithTopicFilter(f =>
{
f.WithTopic("keipalatest/1/post");
f.WithAtLeastOnceQoS();
})
.Build();
await mqttClient.SubscribeAsync(mqttSubscribeOptions, CancellationToken.None);
Console.WriteLine("MQTT client subscribed to topic.");
// The line below feels like a hack to keep background service from restarting
Console.ReadLine();
}
}
}
}
Program.cs
using MqttClientAspNetCore.Services;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHostedService<MqttClientService>();
var app = builder.Build();
// To check if web server is still responsive
app.MapGet("/", () =>
{
return "Hello World";
});
app.Run();
There's no need for Console.ReadLine or even the loop. The BackgroundService application code won't terminate when ExecuteAsync returns. If you want the application to terminate when ExecuteAsync terminates you have to actually tell it to through the IApplicationLifecycle interface.
I've found this the hard way the first time I tried using a Generic host for a command line tool. Which seemed to hang forever ....
ExecuteAsync can be used to set up the MQTT client and the event handler and just let them work. The code terminates only when StopAsync is called. Even then, this is done by signaling a cancellation token, not by aborting some worker thread.
The client itself can be built in the constructor, eg using configuration settings. Only ConnectAsync needs to be called in ExecuteAsync.
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await _client.ConnectAsync(_clientOptions, CancellationToken.None);
_logger.LogInformation("Connected");
await _client.SubscribeAsync(_subscriptionOptions, CancellationToken.None);
_logger.LogInformation("Subscribed");
}
The service code stops when StopAsync is called and the cancellation token is triggered. stoppingToken.Register could be used to call _client.DisconnectAsync when that happens, but Register doesn't accept an asynchronous delegate. A better option is to override StopAsync itself :
public virtual async Task StopAsync(CancellationToken cancellationToken)
{
await _client.DisconnectAsync();
await base.StopAsync(cancellationToken);
}
The constructor can create the client and register the message handler
public class MqttClientService : BackgroundService
{
ILogger<MqttClientService> _logger;
IMqttClient _client=client;
MqttClientOptions _clientOptions;
MqttSubscriptionOptions _subscriptionOptions;
string _topic;
public MqttClientService(IOptions<MyMqttOptions> options,
ILogger<MqttClientService> logger)
{
_logger=logger;
_topic=options.Value.Topic;
var factory = new MqttFactory();
_client = factory.CreateMqttClient();
_clientOptions = new MqttClientOptionsBuilder()
.WithTcpServer(options.Value.Address)
.Build();
_subscriptionOptions = factory.CreateSubscribeOptionsBuilder()
.WithTopicFilter(f =>
{
f.WithTopic(options.Value.Topic);
f.WithAtLeastOnceQoS();
})
.Build();
_client.ApplicationMessageReceivedAsync += HandleMessageAsync;
}
Received messages are handled by the HandleMessageAsync method :
async Task HandleMessageAsync(ApplicationMessageProcessedEventArgs e)
{
var payload=Encoding.UTF8.GetString(e.ApplicationMessage.Payload);
_logger.LogInformation("### RECEIVED APPLICATION MESSAGE ###\n{payload}",payload);
var applicationMessage = new MqttApplicationMessageBuilder()
.WithTopic(_topic)
.WithPayload("OK")
.Build();
await _client.PublishAsync(applicationMessage, CancellationToken.None);
_logger.LogInformation("MQTT application message is published.");
}
Finally, since BackgroundService implements IDisposable, we can use Dispose to dispose the _client instance :
public void Dispose()
{
Dispose(true);
}
protected virtual Dispose(bool disposing)
{
if(disposing)
{
_client.Dispose();
base.Dispose();
}
_client=null;
}
If your service has nothing else useful to do, it can just wait for the CancellationToken to fire:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
await Handle_Received_Application_Message(stoppingToken);
}
catch (OperationCanceledException) { }
}
public static async Task Handle_Received_Application_Message(CancellationToken cancellationToken)
{
...
Console.WriteLine("MQTT client subscribed to topic.");
await Task.Delay(Timeout.Infinite, cancellationToken);
}
I've created a simple webapi .net core 3.1 app.
I want to catch all unhandled exceptions.So I put this code according to the docs :
app.UseExceptionHandler(c => c.Run(async context =>
{
var exception = context.Features
.Get<IExceptionHandlerPathFeature>()
.Error;
var response = new { error = exception.Message };
log.LogDebug(exception.Message);
}));
This is my action:
[HttpGet]
public IActionResult Get()
{
throw new Exception("this is a test");
}
When this code runs, I do see that UseExceptionHandler is working.
But when my code in the action is :
[HttpGet]
public IActionResult Get()
{
Task.Run(async () =>
{
await Task.Delay(4000);
throw new Exception("this is a test");
});
return Ok();
}
Then UseExceptionHandler is NOT working.
However - the following code does catch the task's exception :
AppDomain.CurrentDomain.FirstChanceException += (sender, eventArgs) =>
{
Debug.WriteLine(eventArgs.Exception.ToString());
};
Question:
Why does the task exception isn't recognized by UseExceptionHandler?
How can I catch ALL types of exceptions? Should I rely only on AppDomain.CurrentDomain.FirstChanceException?
nb , I did disabled app.UseDeveloperExceptionPage();
To answer your questions.
Why does the task exception isn't recognized by UseExceptionHandler?
As already suggested in the comments, you cannot use UseExceptionHandler to catch exceptions initiated inside non-awaited tasks. UseExceptionHandler wraps your request in ASP.NET Core middleware. Once the action returns OK to the client, the middleware is no longer able to catch any exceptions happening in tasks started from within the action.
How can I catch ALL types of exceptions? Should I rely only on AppDomain.CurrentDomain.FirstChanceException?
You can catch exceptions globally and log them this way if you'd like. But I wouldn't recommend you to do it this way. The only reason you need to implement this event, is that you are starting tasks/threads inside your web requests. You have no way of knowing if these tasks are kept running (application restart, recycle, etc.). If you are looking to launch background tasks with ASP.NET Core, you should use Worker Services which is the intended way of doing this:
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<MyWorker>();
});
public class MyWorker : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
// Do work
}
catch (Exception e)
{
// Log it?
}
await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
}
}
}
The cause of this particular symptom is that Get is starting a fire-and-forget task that the server knows nothing about. The request will complete before the task even has a chance to execute, so the UseExceptionHandler middleware will never see any exceptions. This is no longer a fire-and-forget task.
The real problem though, is executing a long running task in the background. The built-in way to do this is using a Background Service. The docs show how to create timed and queued background service, that act as job queues.
It's equally easy, if not easier, to publish messages with the desired data from, eg a controller to the background service using, eg Channels. No need to create our own queue, when the BCL already has an asynchronous one.
The service could look like this :
public class MyService: BackgroundService
{
private readonly ChannelReader<T> _reader;
public QueuedBspService(MessageQueue<T> queue)
{
_reader = queue.Reader;
}
protected internal async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
await foreach (var msg in _reader.ReadAllAsync(stoppingToken))
{
try
{
//Process the message here
}
catch (Exception exc)
{
//Handle message-specific errors
}
}
}
catch (Exception exc)
{
//Handle cancellations and other critical errors
}
}
}
The MessageQueue<T> wraps the Channel, making it easier to inject it to both the BackgroundService and any publishers like eg, a Controller action:
public class MessageQueue<T>
{
private readonly Channel<T> _channel;
public ChannelReader<T> Reader => _channel;
public ChannelWriter<T> Writer => _channel;
public MessageChannel()
{
_channel = Channel.CreateBounded<T>(1);
}
}
I adjusted this code from a service that only allows a single operation at a time. That's a quick&dirty way of preventing controllers from making requests that can't be handled.
On the contolle side, this action will post a request to the queue if possible, and return a Busy response otherwise :
public class MyController
{
private readonly ChannelWriter<T> _writer;
public MyController(MessaggeQueue<T> queue)
{
_writer = queue.Writer;
}
[HttpPost]
[ProducesResponseType(StatusCodes.Status201Created)]
[ProducesResponseType(StatusCodes.Status503ServiceUnavailable)]
public async Task<ActionResult> Post(....)
{
var jobName="SomeJob";
var id=Guid.NewGuid();
var jobMsg=CreateMessage(id,...);
try
{
if (_writer.TryWrite(msg))
{
return CreatedAtAction("GetItem","Jobs",new {id});
}
else
{
return Problem(statusCode:(int) HttpStatusCode.ServiceUnavailable,detail:"Jobs in progress",title:"Busy");
}
}
catch (Exception exc)
{
_logger.LogError(exc,"Queueing {job} failed",jobName);
throw;
}
}
}
The Post action first checks if it can even post a job message. If it succeeds, it returns a 201 - Created response with a URL that could be checked eg to check the status of the jobs. return Created() could be used instead, but once you create a long running job, you also want to check its status.
If the channel is at capacity, the core returns 503 with an explanation
I have an example for Windows service in .NET core 3.1.
I am generating an intentional error to see its behavior, my idea is that when the exception occurs I want to call the StopAsync method to stop the service, however once the StopAsync is executed the ExecuteAsync method is executed again and the service does not stop, entering like this in an infinite loop.
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await Task.CompletedTask;
while (!stoppingToken.IsCancellationRequested)
{
try
{
int num = 233;
int result = num / 0;
//todo
}
catch (Exception ex)
{
_logger.LogError(ex.Message);
await StopAsync(stoppingToken); //this execute, but it keeps running this ExecuteAsync method
}
}
}
public override async Task StopAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("stop service");
await base.StopAsync(cancellationToken);
}
the ExecuteAsync method is executed again and the service does not stop, entering like this in an infinite loop.
Are you sure about that? StopAsync will cancel the token provided to ExecuteAsync, and it looks like your ExecuteAsync method will complete in that case.
The host application itself will continue running, though; perhaps that's what you're seeing. To shut down the host application when the background service exits, you need to explicitly stop the application:
private readonly IHostApplicationLifetime _hostApplicationLifetime;
public MyBackgroundService(IHostApplicationLifetime hostApplicationLifetime) =>
_hostApplicationLifetime = hostApplicationLifetime;
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
...
}
finally
{
_hostApplicationLifetime.StopApplication();
}
}
I'm trying to start a background task on demand, whenever I receive a certain request from my api end point. All the task does is sending an email, delayed by 30 seconds. So I though BackgroundService would fit. But the problem is it looks like the BackgroundService is mostly for recurring tasks, and not to be executed on demand per this answer.
So what other alternatives I have, im hoping not to have to rely on 3rd parties libraries like Hangfire? I'm using asp.net core 3.1.
This is my background service.
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace ProjectX.Services {
public class EmailOfflineService : BackgroundService {
private readonly ILogger<EmailOfflineService> log;
private readonly EmailService emailService;
public EmailOfflineService(
ILogger<EmailOfflineService> log,
EmailService emailService
) {
this.emailService = emailService;
this.log = log;
}
protected async override Task ExecuteAsync(CancellationToken stoppingToken)
{
log.LogDebug("Email Offline Service Starting...");
stoppingToken.Register(() => log.LogDebug("Email Offline Service is stopping."));
while(!stoppingToken.IsCancellationRequested)
{
// wait for 30 seconds before sending
await Task.Delay(1000 * 30, stoppingToken);
await emailService.EmailOffline();
// End the background service
break;
}
log.LogDebug("Email Offline Service is stoped.");
}
}
}
You could try to combine an async queue with BackgroundService.
public class BackgroundEmailService : BackgroundService
{
private readonly IBackgroundTaskQueue _queue;
public BackgroundEmailService(IBackgroundTaskQueue queue)
{
_queue = queue;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var job = await _queue.DequeueAsync(stoppingToken);
_ = ExecuteJobAsync(job, stoppingToken);
}
}
private async Task ExecuteJobAsync(JobInfo job, CancellationToken stoppingToken)
{
try
{
await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
// todo send email
}
catch (Exception ex)
{
// todo log exception
}
}
}
public interface IBackgroundTaskQueue
{
void EnqueueJob(JobInfo job);
Task<JobInfo> DequeueAsync(CancellationToken cancellationToken);
}
This way you may inject IBackgroundTaskQueue inside your controller and enqueue jobs into it while JobInfo will contain some basic information for executing the job in background, e.g.:
public class JobInfo
{
public string EmailAddress { get; set; }
public string Body { get; set; }
}
An example background queue (inspired by the ASP.NET Core documentation):
public class BackgroundTaskQueue : IBackgroundTaskQueue
{
private ConcurrentQueue<JobInfo> _jobs = new ConcurrentQueue<JobInfo>();
private SemaphoreSlim _signal = new SemaphoreSlim(0);
public void EnqueueJob(JobInfo job)
{
if (job == null)
{
throw new ArgumentNullException(nameof(job));
}
_jobs.Enqueue(job);
_signal.Release();
}
public async Task<JobInfo> DequeueAsync(CancellationToken cancellationToken)
{
await _signal.WaitAsync(cancellationToken);
_jobs.TryDequeue(out var job);
return job;
}
}
I think the simplest approach is to make a fire-and-forget call in the code of handling the request to send a email, like this -
//all done, time to send email
Task.Run(async () =>
{
await emailService.EmailOffline(emailInfo).ConfigureAwait(false); //assume all necessary info to send email is saved in emailInfo
});
This will fire up a thread to send email.
The code will return immediately to the caller.
In your EmailOffline method, you can include time-delay logic as needed.
Make sure to include error logging logic in it also, otherwise exceptions from EmailOffline may be silently swallowed.
P.S. -
Answer to Coastpear and FlyingV -
No need to concern the end of calling context. The job will be done on a separate thread, which is totally independent of the calling context.
I have used similar mechanism in production for a couple of years, zero problem so far.
If your site is not supper busy, and the work is not critical, this is the easiest solution.
Just make sure you catch and log error inside your worker (EmailOffline, in this example).
If you need more reliable solution, I'd suggest using a mature queue product like AWS SQS, do not bother to create one by yourself. It is not an easy job to create a really good queue system.
Use Hangfire, it's Background Methods functionality is great, and provides you with a nice dashboard for free:
https://docs.hangfire.io/en/latest/background-methods/index.html
I work with some WIFI devices such as cameras.
The basic fellow that I implemented:
Someone presses a button.
The button calls my Web API endpoint.
My Web API end point calls one of the API's of camera (by HttpRequest).
Processing each request takes 5 second. And between each request should be 1 second delay. For instance, If you press the button 2 times with one second delay after each: First we expect 5 second for processing the first press, then one second delay and in the end we expect 5 second for the last process (second press).
To do that I am using Queued background tasks based on Fire and Forgot manner in .NetCore 3.1 project and it works fine when I am dealing with just one camera.
But the new requirement of the project is, The background task should handle multiple cameras. It means one queue per camera, and queues should work parallel based on the fellow that I described above.
For example if we have 2 devices camera-001 and camera-002 and 2 connected buttons btn-cam-001 and btn-cam-002, And the order of pressing(0.5sec delay after each press) : 2X btn-cam-001 and 1X btn-cam-002.
What really happens is FIFO. First the requests of btn-cam-001 will be processed and then btn-cam-002.
What I expect and need: Camera-002 should not wait to receive the request and the first requests towards both cameras 001/002 should be processed in a same time(Based on the exmaple). Like each camera has own queue and own process.
The question is how can I achieve that in .NetCore 3.1?
Appreciate any help.
My current background service:
public class QueuedHostedService : BackgroundService
{
public IBackgroundTaskQueue TaskQueue { get; }
private readonly ILogger _logger;
public QueuedHostedService(IBackgroundTaskQueue taskQueue, ILoggerFactory loggerFactory)
{
TaskQueue = taskQueue;
_logger = loggerFactory.CreateLogger<QueuedHostedService>();
}
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("Queued Hosted Service is starting.");
while (!cancellationToken.IsCancellationRequested)
{
var workItem = await TaskQueue.DequeueAsync(cancellationToken);
try
{
await workItem(cancellationToken);
}
catch (Exception exception)
{
_logger.LogError(exception, $"Error occurred executing {nameof(workItem)}.");
}
}
_logger.LogInformation("Queued Hosted Service is stopping.");
}
}
And the current BackgroundTaskQueue:
public class BackgroundTaskQueue : IBackgroundTaskQueue
{
private readonly SemaphoreSlim _signal = new SemaphoreSlim(0);
private readonly ConcurrentQueue<Func<CancellationToken, Task>> _workItems =
new ConcurrentQueue<Func<CancellationToken, Task>>();
public void QueueBackgroundWorkItem(Func<CancellationToken, Task> workItem)
{
if (workItem is null)
{
throw new ArgumentNullException(nameof(workItem));
}
_workItems.Enqueue(workItem);
_signal.Release();
}
public async Task<Func<CancellationToken, Task>> DequeueAsync(CancellationToken cancellationToken)
{
await _signal.WaitAsync(cancellationToken);
_workItems.TryDequeue(out var workItem);
return workItem;
}
}
My current endpoint:
[HttpPost("hit")]
public ActionResult TurnOnAsync([FromBody] HitRequest request, CancellationToken cancellationToken = default)
{
try
{
var camera = ConfigurationHelper.GetAndValidateCamera(request.Device, _configuration);
_taskQueue.QueueBackgroundWorkItem(async x =>
{
await _cameraRelayService.TurnOnAsync(request.Device, cancellationToken);
Thread.Sleep(TimeSpan.FromSeconds(1));
});
return Ok();
}
catch (Exception exception)
{
_logger.LogError(exception, "Error when truning on the lamp {DeviceName}.", request.Device);
return StatusCode(StatusCodes.Status500InternalServerError, exception.Message);
}
}
Instead of a single BackgroundTaskQueue you could have one per camera. You could store the queues in a dictionary, having the camera as the key:
public IDictionary<IDevice, IBackgroundTaskQueue> TaskQueues { get; }
Then in your end-point use the queue that is associated with the requested camera:
_taskQueues[camera].QueueBackgroundWorkItem(async x =>