Develop Reference

How to correctly trigger an Async process when receiving a message via MqttNet?

We currently developing a set of .netCore console apps which reside on a remote server fed by mqtt message via MqttNet.
Upon receipt of messages in the mqttClient we pass these along to various registered Actions. The actions being key'd on the message topic.
public void StartClient()
{
mqttClient.ApplicationMessageReceived += (s, e) =>
{
var clientId = e.ClientId;
var topic = e.ApplicationMessage.Topic;
var payload = e.ApplicationMessage.ConvertPayloadToString();
Console.WriteLine("");
Console.WriteLine($"ClientId: {clientId}");
Console.WriteLine($"Topic: {topic}");
Console.WriteLine($"Payload: {payload}");
var action = actions[topic];
action?.Invoke(payload);
};
mqttClient.StartAsync(options).Wait();
}
An example Action could be:
public void StartupCompleted(string message)
{
IsInStartup = false;
}
So when the message is received it sets a flag in one of the netCore apps.
For synchronous Actions this isn't causing an issue; however when we want to trigger an asynchronous or longer running process we start to see the netCore app crashing.
In our case we're trying to collect the mqttNet message and route them to specific EventHubs in Azure. For small numbers of message we can publish 1 by 1 direct to Azure and use a Task.Run() for them to be Fire and Forget; however as the number of messages increase Azure gets loaded too heavily and starts to reject the messages. As the publishing to Azure is wrapped in a Task.Run() we then loose the Exceptions and do not know if publishing succeeded or failed in order to schedule retries.
So in the higher level ApplicationMessageReceived method can we register Task's to be run when a Topic is received? Something like:
public void StartClient()
{
mqttClient.ApplicationMessageReceived += (s, e) =>
{
var clientId = e.ClientId;
var topic = e.ApplicationMessage.Topic;
var payload = e.ApplicationMessage.ConvertPayloadToString();
Console.WriteLine("");
Console.WriteLine($"ClientId: {clientId}");
Console.WriteLine($"Topic: {topic}");
Console.WriteLine($"Payload: {payload}");
var task = tasks[topic];
await task?.Invoke(payload);
};
mqttClient.StartAsync(options).Wait();
}
With correct syntax, etc. Or does this stall the ApplicationMessage processing until the Task has completed?
We do have two alternatives, neither of which is working particularly well...
1 - Force the async Azure functions to be synchronous as below, asyncContext is to ensure we receive any Exceptions; this is in a client wrapper for the EventHubProducerClient:
public void PublishEvents(string connectionString, IEnumerable<AzureEventMessage> azureEventMessages)
{
var task = Task.Run(() => AsyncContext.Run(() => PublishEventsAsync(connectionString, azureEventMessages)));
task.GetAwaiter().GetResult();
}
Where the Action is registered this triggered as a Fire and Forget:
public void PublishToAzure(string message)
{
Task.Run(() => eventService.PublishToAzure(message));
}
2 - ensure the async code is present as long as possible, then in the Action have:
public void PublishToAzure(string message)
{
Task.Run(async () => await eventService.PublishToAzure(message));
}
The additional problem here is that if the ApplicationMessageReceived handler needs to complete before the next message arrives we will drop messages... internal queue lengths are 250 messages, but even if we push that to 5000-10000 they will fill up over time.

C# best practice to run an independent tasks in conjunction with awaitable task

Application : Asp.Net Core Web API
I have requirement, On receiving the notification I should do 2 independent tasks. notification gets triggered as apart of web hook.
call another service and then saves the response in log.
Since this is independent task I do not want to wait the caller until we receive response. So I thought to wrap this function in Task.Run(), so that it will be run on another thread available on thread pool.
I donot care wheather it fails or success, I just need to log the response.
Task.Run(() => ProcessThroughRestClient(this.httpContext.Request, this.cspgConfiguration.Value.ApacNotificationService, this.restClient, this.logger));
Saves the request object in DB for tracing.
Since I must save the notification, I made it as awaitable task.
await this.invoiceDbInstance.UpdateItemAsync(payment);
Below is the full code.
Main Method
public async Task<IActionResult> NotifyAsync()
{
this.logger.LogTrace("{CorrelationId} - notify async method has been called.", this.CorrelationId);
Task.Run(() => ProcessThroughRestClient(this.httpContext.Request, this.Configuration.Value.NotificationService, this.restClient, this.logger));
var request = this.GetDataFeedRequest(this.httpContext.Request);
if (request != null)
{
this.logger.LogInformation("{CorrelationId} - data feed invoked with the order reference {OrderReference}", request.OrderReference, this.CorrelationId);
Func<Invoice, bool> condition = x => x.SourceTransactionId == request.OrderReference;
var payment = this.invoiceDbInstance.GetItem(condition);
if (payment != null)
{
payment.TransactionStatus = request.IsSuccessStatusCode() ? TransactionStatus.Success : TransactionStatus.Failure;
payment.TransactionId = request.PaymentReference;
payment.UpdatedAt = DateTime.UtcNow.ToLongDateString();
await this.invoiceDbInstance.UpdateItemAsync(payment);
this.logger.LogInformation("{CorrelationId} - data feed updated order reference {OrderReference} updated in Invoice with {PaymentReference}", request.OrderReference, this.CorrelationId, request.PaymentReference);
}
else
{
this.logger.LogInformation("{CorrelationId} - Payment not found.", this.CorrelationId);
}
}
this.logger.LogTrace("{CorrelationId}- notify async method ended.", this.CorrelationId);
return new OkResult();
}
Http Call function which will be invoked through Task.Run()
private static HttpResponseMessage ProcessThroughRestClient(HttpRequest request, string url, IRestClient client, ILogger<PayDollarNotificationService> log)
{
try
{
log.LogTrace("Paydollar notify async ProcessThroughRestClient method has been called.");
var parameters = request.Form?.ToDictionary(x => x.Key, x => x.Value.ToString());
if (parameters.Any())
{
var response = client.PostAsync(url, RestClientHelper.BuildFormUrlEncodedContent(parameters)).Result;
log.LogInformation("sent request to {url}.", url);
log.LogInformation($"{url} response {response.ReadContent()?.Result}");
return response;
}
else
{
log.LogInformation("No form parameters found.");
return null;
}
}
catch (Exception ex)
{
log.LogError($"An error occured: {ex.Message} {ex}");
}
return null;
}
My question is, Is there any advantage using Task.Run() as above instead awitable task? or is it is going to be blocking thread?

Azure Function : how to implement delay of retrying queue message better

My Azure function should listen Queue for messages, then get message, try to call an external service with value inside message, if the external service returns "OK" then we have to write message to another queue (for next Azure Function), if returns "Fail" we have to return to our current queue and retry by our Azure Function again after 5 minutes. How to implement it? I did it with Timer, but solution does not like me:
[FunctionName("FunctionOffice365VerificateDomain_and_AddService_and_GexMxRecord")]
public async static Task Run([TimerTrigger("0 */5 * * * *")]TimerInfo myTimer,
[Queue("domain-verificate-Office365-add-services-get-mx-record", Connection = "StorageConnectionString")]CloudQueue listenQueue,
[Queue("domain-add-mx-record-to-registrator", Connection = "StorageConnectionString")]CloudQueue outputQueue,
ILogger log)
{
while (true)
{
// do "invisible" message for next 30 sec
var message = await listenQueue.GetMessageAsync();
if (message != null)
{
DomainForRegistration domainForRegistration = JsonConvert.DeserializeObject<DomainForRegistration>(message.AsString);
try
{
await _office365DomainService.VerifyDomainAsync(domainForRegistration.DomainName);
// remove message
await listenQueue.DeleteMessageAsync(message);
await _office365DomainService.UpdateIndicateSupportedServicesDomainAsync(domainForRegistration.DomainName);
var mxRecord = await _office365DomainService.GetMxRecordForDomainAsync(domainForRegistration.DomainName);
}
catch (DomainVerificationRecordNotFoundException)
{
// thrown when VerifyDomainAsync failed
}
}
else
break;
}
}
How to do it more carefully, without these while(true), but with timeout after fail validation?

Agree with #DavidG, try to use queue trigger to achieve your goal.
W can rely on the host setting of Queue.
visibilityTimeout is The time interval between retries when processing of a message fails
maxDequeueCount is The number of times to try processing a message before moving it to the poison queue.
{
"version": "2.0",
"extensions": {
"queues": {
"visibilityTimeout" : "00:05:00",
"maxDequeueCount": 2,
}
}
}
In this way, the function should look like
public static async Task Run(
[QueueTrigger("domain-verificate-Office365-add-services-get-mx-record")]string myQueueItem, ILogger log,
[Queue("domain-add-mx-record-to-registrator", Connection = "StorageConnectionString")]IAsyncCollector<string> outputQueue
)
{
// do stuff then output message
await outputQueue.AddAsync(myQueueItem);
}
If you don't want to throw the exception to host, we can turn to initialVisibilityDelay of CloudQueue method.
specifying the interval of time from now during which the message will be invisible
public static async Task Run(
[QueueTrigger("domain-verificate-Office365-add-services-get-mx-record")]string myQueueItem, ILogger log,
[Queue("domain-add-mx-record-to-registrator", Connection = "StorageConnectionString")]IAsyncCollector<string> outputQueue,
[Queue("domain-verificate-Office365-add-services-get-mx-record", Connection = "StorageConnectionString")]CloudQueue listenQueue
)
{
try
{
// do stuff then output message
await outputQueue.AddAsync(myQueueItem);
}
catch(DomainVerificationRecordNotFoundException)
{
// add the message in current queue and can only be visible after 5 minutes
await listenQueue.AddMessageAsync(new CloudQueueMessage(myQueueItem), null, TimeSpan.FromMinutes(5), null, null);
}
}

Where To Get InstanceId From For RaiseEventAsync Method In Azure Durable Functions?

I'm learning how to use Azure Durable Functions and I've come across a problem. I'm posting a message to a queue and waiting for it, then logging it to trace, but I'm not sure where to get the InstanceId from.
Here's what I have so far:
public static class Go
{
private const string ReceivedMessage = "received-message";
[FunctionName(nameof(Go))]
public static async Task Run(
[OrchestrationTrigger] DurableOrchestrationContext context,
TraceWriter log)
{
var message = await context.WaitForExternalEvent<string>(ReceivedMessage);
log.Info($"Received message: {message}");
}
[FunctionName(nameof(GetMessageFromQueue))]
public static async Task GetMessageFromQueue(
[QueueTrigger("messages")] string message,
[OrchestrationClient] DurableOrchestrationClient client)
{
await client.RaiseEventAsync("InstanceId", ReceivedMessage, message); // <- here
}
}
For completeness, here's the rest of the code that I have. The HttpStartSingle class is in the same project as the one above and the Program class is just a normal console application that I use to start the context:
public static class HttpStartSingle
{
[FunctionName(nameof(HttpStartSingle))]
public static async Task<HttpResponseMessage> RunSingle(
[HttpTrigger(
AuthorizationLevel.Function,
"get", "post",
Route = "orchestrators/{functionName}")]
HttpRequestMessage req,
[OrchestrationClient] DurableOrchestrationClient starter,
string functionName,
TraceWriter log)
{
var eventData = await req.Content.ReadAsAsync<object>();
var instanceId = await starter.StartNewAsync(functionName, eventData);
return starter.CreateCheckStatusResponse(req, instanceId);
}
}
public class Program
{
public static async Task Main(string[] args)
{
Thread.Sleep(TimeSpan.FromSeconds(5));
var request = WebRequest.Create("http://localhost:7071/api/orchestrators/Go");
request.Timeout = Convert.ToInt32(TimeSpan.FromMinutes(1).TotalSeconds);
request.ContentLength = 0;
request.Method = "POST";
var json = string.Empty;
using (var response = await request.GetResponseAsync())
using (var stream = response.GetResponseStream())
{
if (stream != null)
{
using (var reader = new StreamReader(stream, Encoding.UTF8))
{
if (reader.Peek() > -1)
{
json = await reader.ReadToEndAsync();
}
}
}
}
Console.WriteLine(json);
var storageAccount = CloudStorageAccount.DevelopmentStorageAccount;
var queueuClient = storageAccount.CreateCloudQueueClient();
var queue = queueuClient.GetQueueReference("messages");
await queue.CreateIfNotExistsAsync();
await queue.AddMessageAsync(new CloudQueueMessage("This is a test message"));
Console.ReadKey();
}
}
Update:
The above code is just really a proof of concept to see if Durable Functions meets my requirements. I'm not sure so far, however here is what I'm looking to do:
I will have two queues.
Queue A will be seeded with a small number of A's, which I intend to recycle.
Queue B will have an indeterminate (for all intents and purposes infinite) number of B's pushed to it by an external source.
I want the orchestrator to wait for inputs from both queues, then send A and B to another function which will produce a number of B's. That function will then push the B's to Queue B and then push the A back to Queue A. The process then repeats.
I'd prefer the processing to be parallel based on the number of A's I have. As I see it, I have two options:
A) Create one Orchestrator created with an InstanceId taken from a constant. Then, the functions that listen to the A and B Queues will know what InstanceId to use for the RaiseEventAsync call.
Problem: Not parallel.
B) Wrap an instanceId in an A. Create one Orchestrator per A. A knows which Orchestrator to use and now each Orchestrator can just listen to Queue B.
Problem: B's don't know what InstanceId to use for the RaiseEventAsync call.

Whatever puts the message on the queue that is going to trigger your GetMessageFromQueue would need to understand what instance it's trying to target and I would then expect that you include the instance ID in the message itself.
In the specific scenario you laid out above, I would actually expect your HttpTrigger based function to return a body that includes the instance ID. Right now you're returning the HttpResponseMessage that's created by CreateCheckStatusResponse, but this is documented as returning a 202 status with a Location header that tells people where they can check on the status of that instance of the orchestration going forward. While this information might be useful to handback as well, you really need to be handing back the instance ID.
One way would be to create your own HttpResponseMessage that contains a strongly typed message body that includes the instance ID as a property. If you did that, then the calling program would be able to pull that ID out of the response body and then include it in the message they put onto the queue so that your GetMessageFromQueue function can pull it out of there to pass to RaiseEventAsync.

Is adding 'await' to an async method in a synchronous logic block good practice?

My Asp.Net MVC site in synchronous (fetch, display sucess, upload, display success, etc, etc), but I do have one async method to call to send a message to my Azure service bus queue, to be triggered at a later date by a Azure function.
When I call the asynchronous method to put the message on the Azure queue I want to save the returned sequence in my db so I can cancel the message at a later date if needed. But as far as I'm aware I shouldn't be waiting for an async method, especially because I need to display a success to my user fairly quickly. So is the code below that runs the async method the best way to perform this type of operation? AFAIK 'await' will wait until the returned value comes back, should I be doing this before I return to the controller and view?
ScheduleMessageAsync
This code is part of my service class that gets called by my controller
internal void SaveEvent(some parameters) {
// save event details in db
// now call up class to save message into the Azure queue
MessageQueue messageQueue = new MessageQueue();
messageQueue.SendMessage(eventId);
// do some other stuff like send the user a email and return to the controller and show the user his/her new event has been saved
}
Now here I have the message code to save the message in the Azure queue
public async void SendMessage(int eventId)
{
string serviceBusConnectionString = System.Configuration.ConfigurationManager.AppSettings["ServiceBusConnectionString"];
string queueName = System.Configuration.ConfigurationManager.AppSettings["QueueName"];
var queueClient = QueueClient.CreateFromConnectionString(serviceBusConnectionString, queueName);
BrokeredMessage message = new BrokeredMessage("some test message");
message.MessageId = eventId.ToString();
////newEvent.EventDateTime
// TimeSpan ts = new TimeSpan(1, 0, 0);
// DateTimeOffset dto = new DateTimeOffset(DateTime.Now, ts);
long sequenceToCancel = await queueClient.ScheduleMessageAsync(message, DateTimeOffset.Now.AddDays(6));
queueClient.Close();
// store sequenceToCancel in db table
}