I've been looking for examples to easily dequeue an Azure Storage queue the same way you can enqueue an item (by injecting IAsyncCollector in the Run method). but alas, no success.
The only things I've found are enqueueing items or reacting to items being added to a queue.
There is an app running on my local server that will periodically call the function (and keeps calling till the queue is empty) to get the items of the queue. I want to do this using an Azure function.
Any help is welcome.
Please check this doc about ICollector:ICollector with Azure Storage Queue.
ICollector and IAsyncCollector can be used as parameter types for Storage Queue output bindings.
For now azure function binding only supports output binding to write messages to a queue. Or You could use queue trigger to retrieve message if you don't need to call it with HTTP request.
If you have to use HTTP request, suppose you have to create a HTTP trigger function, then retrieve and delete the queue to implement a dequeue action like the below code.
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
// Parse the connection string and return a reference to the storage account.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(Environment.GetEnvironmentVariable("AzureWebJobsStorage"));
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
// Retrieve a reference to a queue
CloudQueue queue = queueClient.GetQueueReference("myqueue");
// Async dequeue the message
CloudQueueMessage retrievedMessage = await queue.GetMessageAsync();
Console.WriteLine("Retrieved message with content '{0}'", retrievedMessage.AsString);
//Process the message in less than 30 seconds, and then delete the message
await queue.DeleteMessageAsync(retrievedMessage);
return (ActionResult)new OkObjectResult(retrievedMessage.AsString);
}
Why don't you create a webhook function that your app calls then within the function you can dequeue items as necessary using the standard storage queue api for whatever language you use.
Related
I am implementing an queue which is used by an api, dequeued by an azure function and inserted into a database. I want to bulk insert into db to improve performance. However I am experiencing some issues when completing multiple messages myself.
By setting autoCompleteMessages: false the code underneath works fine.
[FunctionName("BatchProcessingFunc")]
public async Task Run(
[ServiceBusTrigger("eventbatches", Connection = "QUEUE_CONNECTIONSTRING")] Message myQueueItem,
MessageReceiver messageReceiver,
ILogger log)
{
log.LogInformation($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
await messageReceiver.CompleteAsync(myQueueItem.SystemProperties.LockToken);
}
However once i change Message myQueueItem to Message[] myQueueItem it suddenly starts to throw an Microsoft.Azure.ServiceBus.MessageLockLostException. Code which triggers an exception:
[FunctionName("BatchProcessingFunc")]
public async Task Run(
[ServiceBusTrigger("eventbatches", Connection = "QUEUE_CONNECTIONSTRING")] Message[] myQueueItem,
MessageReceiver messageReceiver,
ILogger log)
{
foreach(var message in myQueueItem)
{
log.LogInformation($"C# ServiceBus queue trigger function processed message: {message}");
await messageReceiver.CompleteAsync(message.SystemProperties.LockToken);
}
}
Complete exception:
Microsoft.Azure.ServiceBus.MessageLockLostException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue, or was received by a different receiver instance.
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.DisposeMessagesAsync(IEnumerable`1 lockTokens, Outcome outcome)
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.CompleteAsync(IEnumerable`1 lockTokens)
at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener.<>c__DisplayClass35_0.<<StartMessageBatchReceiver>b__0>d.MoveNext()
CompleteAsync seems to work the first time, however it seems like the function tries to complete it again once more. Which obviously wont work as that message already is completed.
I need to control which messages is completed or abandoned myself,as some messages may fail and some may succeed in the same bulk-write.
Any ideas to what might be wrong, fixes or eventual workarounds would be much appreciated. Thanks.
When receiving mail through MailGun they require a response within a limited time. I have two issues with this:
1) After receiving the message I need to process and record it in my CRM which takes some time. This causes MailGun to time out before I get to send a response. Then MailGun resends the message again and again as it continues to time out.
2) MailGun's post is not async but the api calls to my CRM are async.
So I need to send MailGun a 200 response and then continue to process the message. And that process needs to be in async.
The below code shows what I want to have happen. I tried using tasks and couldn't get it working. There are times when many emails can come in a once (like when initializing someone's account) if the solution requires some sort of parallel tasks or threads it would need to handle many of them.
public class HomeController : Controller
{
[HttpPost]
[Route("mail1")]
public ActionResult Mail()
{
var emailObj = MailGun.Receive(Request);
return Content("ok");
_ = await CRM.SendToEmailApp(emailObj);
}
}
Thank you for the help!
The easiest way to do what you are describing (which is not recommended, because you may lose some results if your app crash) is to use a fire & forget task:
var emailObj = MailGun.Receive(Request);
Task.Run(async () => await CRM.SendToEmailApp(emailObj));
return Content("ok");
But, I think what you really want is sort of a Message Queue, by using a message queue you put the message in the queue (which is fast enough) and return immediately, at the same time a processor is processing the message queue and saves the result in the CRM.
This is what it'll look like when you use a message queueing broker.
I need some help figuring out why the following code in the continueWith block is not being executed for a long running service call.
public static async void postServiceAsync(string json, string postServiceUrl, string callbackUrl, string clientId,
string tenant, string secret, string d365Environment, TraceWriter log)
{
HttpClient client = new HttpClient();
//Get authorization header
string authHeader = await D365Authorization.getAccessToken(clientId, tenant, secret, d365Environment);
client.DefaultRequestHeaders.Add("Authorization", authHeader);
var httpContent = new StringContent(json);
client.Timeout = TimeSpan.FromMinutes(90);
client.PostAsync(postServiceUrl, httpContent).ContinueWith(async (result) =>
{
//call callback URL
//This is not executed after a long running service that runs for 20 minutes.
}
}
The continueWith code does get run if the service execution time is short though. I thought it was a timeout issue so I added the client.Timeout value. I tried calling the service in Postman and a value is returned even after waiting for 20+ minutes. I am not using await as I want the execution to continue after calling PostAsync. I just want the continueWith callback executed after the long running service execution has completed. Thanks for your help!
The above method called postServiceAsync is called from an Azure function which is being called from an Azure Logic App http webhook action. Here is the Azure function:
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Function, "post", Route = null)]HttpRequestMessage req, TraceWriter log)
{
...
PostServiceAsync.postServiceAsync(json, shipServiceUrl, callbackUrl, clientId, tenant, secret, d365Environment, log);
var resp = req.CreateResponse(HttpStatusCode.Accepted);
return resp;
}
}
From the Azure function, I need to return the Accepted status code right away. After I've finished calling the long running service using PostAsync, I need to post to the callback URL, which is what I am doing in the continueWith block. Like I mentioned, it works if the service runtime is short. I tried Camilo's suggestion of adding await but the continueWith code did not get executed. I also tried getting rid of the continueWith and just added the code after "await client.PostAsync(...)".
It turns out that there is an Azure function 230 second timeout for http calls without a response. I might not be able to use an Azure function for my purposes.
I have a long running process which performs matches between millions of records I call this code using a Service Bus, However when my process passes the 5 minute limit Azure starts processing the already processed records from the start again.
How can I avoid this
Here is my code:
private static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
long receivedMessageTrasactionId = 0;
try
{
IQueueClient queueClient = new QueueClient(serviceBusConnectionString, serviceBusQueueName, ReceiveMode.PeekLock);
// Process the message
receivedMessageTrasactionId = Convert.ToInt64(Encoding.UTF8.GetString(message.Body));
// My Very Long Running Method
await DataCleanse.PerformDataCleanse(receivedMessageTrasactionId);
//Get Transaction and Metric details
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
}
catch (Exception ex)
{
Log4NetErrorLogger(ex);
throw ex;
}
}
Messages are intended for notifications and not long running processing.
You've got a fewoptions:
Receive the message and rely on receiver's RenewLock() operation to extend the lock.
Use user-callback API and specify maximum processing time, if known, via MessageHandlerOptions.MaxAutoRenewDuration setting to auto-renew message's lock.
Record the processing started but do not complete the incoming message. Rather leverage message deferral feature, sending yourself a new delayed message with the reference to the deferred message SequenceNumber. This will allow you to periodically receive a "reminder" message to see if the work is finished. If it is, complete the deferred message by its SequenceNumber. Otherise, complete the "reminder" message along with sending a new one. This approach would require some level of your architecture redesign.
Similar to option 3, but offload processing to an external process that will report the status later. There are frameworks that can help you with that. MassTransit or NServiceBus. The latter has a sample you can download and play with.
Note that option 1 and 2 are not guaranteed as those are client-side initiated operations.
Have been looking out for analogous to Azure service bus queue, where messaging pump use to fire OnMessage(BrokeredMessage msg) whenever a new message arrives.
Does Azure Storage Queue, not have such event supported?
// Retrieve storage account from connection string
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("StorageConnectionString"));
// Create the queue client
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
// Retrieve a reference to a queue
CloudQueue queue = queueClient.GetQueueReference("myqueue");
// Get the next message
CloudQueueMessage retrievedMessage = queue.GetMessage();
//Process the message in less than 30 seconds, and then delete the message
queue.DeleteMessage(retrievedMessage);
Natively this feature is not supported with Azure Storage Queues. A client need to poll a queue to check for new messages in that queue.
You could simulate the event based behavior by using Functions or WebJobs Triggers. However internally they will be polling the queue and invoke the function/webjob (or in other words raise an event), whenever they find a message in a queue.