Develop Reference

How to put message into dead-letter and process it after some interval in azure function

I have azure function which trigger when we ave new message into service bus topic.
In azure function I checked the customer api is available or not by calling it and checking its status code.
If status code is 200 I need to process the message else put this message into dead-letter and after some interval when customer api is available then process all dead letter message also.
public static class Function1
{
[FunctionName("Function1")]
public static void Run([ServiceBusTrigger("customer-order", "customer-order", Connection = "")]string mySbMsg, ILogger log)
{
// 1.call customer api to check it is available or not
// 2.if it is up and running then process the message else put message into dead-letter
// and after some interval when customer ai is available process dead-letter messages
log.LogInformation($"C# ServiceBus topic trigger function processed message: {mySbMsg}");
}
}
I can able to invoke customer api using HTTPClient and process message also.
but how to put message into dead-letter and How to execute dead-letter
after some interval when customer api is available ?
Proposed Flow Will be
in azure function app which will trigger if new message is there into.
topic start step - Check api is available or down if api is available
process the current message if api is down then do not process
message here we have two choices
1a.put current message into dead letter
1b.put current message back into main but if we do this then again function app will trigger as its new message trigger based and start step will continue.

Rather than putting this in dead letter queue, a better approach would be to defer the message for a certain duration.
If a message cannot be processed because a particular resource for
handling that message is temporarily unavailable but message
processing should not be summarily suspended, a way to put that
message on the side for a few minutes is to remember the
SequenceNumber in a scheduled message to be posted in a few minutes,
and re-retrieve the deferred message when the scheduled message
arrives.
See this answer for an example to how to do deferral in Azure functions v2. Note that the input binding is using message of type Message and is also using the injected MessageReceiver. Those are needed to be able to call DeferAsync. The template code for service bus trigger sets the message type to string, so you would have to change signature as described in that answer.
About deferred messages:
Deferred messages remain in the main queue along with all other active
messages (unlike dead-letter messages that live in a subqueue), but
they can no longer be received using the regular Receive/ReceiveAsync
functions. To retrieve a deferred message, its owner is responsible
for remembering the SequenceNumber as it defers it. Any receiver that
knows the sequence number of a deferred message can later receive the
message explicitly with Receive(sequenceNumber).
How to schedule messages with sequence number of deferred messages so that deferred message can be processed at a later time:
You can schedule messages either by setting the
ScheduledEnqueueTimeUtc property when sending a message through the
regular send path, or explicitly with the ScheduleMessageAsync API

Azure ServiceBus queue not retrying messages when TaskCanceledException is thrown

I have an Azure WebJob function that listens to messages on an Azure ServiceBus queue. Usually when I encounter an exception in my code, the message is abandoned as per the Azure WebJobs SDK documentation:
The SDK receives a message in PeekLock mode and calls Complete on the message if the function finishes successfully, or calls Abandon if the function fails. If the function runs longer than the PeekLock timeout, the lock is automatically renewed.
According to the Azure ServiceBus documentation this should mean that the message becomes available again, and will be retried:
If the application is unable to process the message for some reason, it can call the AbandonAsync method on the received message (instead of CompleteAsync). This method enables Service Bus to unlock the message and make it available to be received again, either by the same consumer or by another competing consumer. Secondly, there is a timeout associated with the lock and if the application fails to process the message before the lock timeout expires (for example, if the application crashes), then Service Bus unlocks the message and makes it available to be received again (essentially performing an AbandonAsync operation by default).
The behavior described above is what usually happens, but I have found an exception to this rule. If my code throws a TaskCanceledException specifically, the message is not abandoned as it should:
public void ProcessQueueMessage([ServiceBusTrigger("queue")] BrokeredMessage message, TextWriter log)
{
throw new TaskCanceledException();
}
When running this function through a web job, I see the error message printed out clear as day, but the message is consumed without any retries and without entering the dead-letter queue. If I replace the TaskCanceledException above with InvalidOperationException, the message is abondened and retried as it should (I have verified this against an actual ServiceBus queue).
I have not been able to find any explanation for this behavior. Currently I am wrapping the TaskCanceledException in another exception to work around the issue.
The question
Is what I am experiencing a bug in the Azure WebJobs SDK? Is TaskCanceledException special in this regard, or do other types of exceptions have similar behavior?
I am using the following NuGet packages:
Microsoft.Azure.WebJobs 2.3.0
Microsoft.Azure.WebJobs.ServiceBus 2.3.0

Functions are supposed to abandon the message if execution was not successful. If you're saying the message was not abandoned and retried even though it should be (assuming MaxDeliveryCount was set to larger than 1 and the receive mode was PeekLock), then it's likely to be the issue with the Functions and not Azure Service Bus. You could verify that by running a console application and performing the same, checking wherever the message is completed and removed from the queue or still on the queue and available for consumption.
Also, looks like you're using the older version of WebJobs (and Azure Service Bus). When performing verification, you'd need to use the older Azure Service Bus client (WindowsAzure.ServiceBus) and not the new one (Microsoft.Azure.ServiceBus).

Azure functions with service bus: How to keep a message in the queue if something goes wrong with its processing?

I'm new to service bus and not able to figure this out.
Basically i'm using Azure function app which is hooked onto the service bus queue. Let's say a trigger is fired from the service bus and I receive a message from the queue, and in the processing of that message something goes wrong in my code. In such cases how do I make sure to put that message back in the queue again? Currently its just disappearing into thin air and when I restart my function app on VS, the next message from the queue is taken.
Ideally only when all my data processing is done and when i hit myMsg.Success() do I want it to be removed from the queue.
public static async Task RunAsync([ServiceBusTrigger("xx", "yy", AccessRights.Manage)]BrokeredMessage mySbMsg, TraceWriter log)
{
try{ // do something with mySbMsg }
catch{ // put that mySbMsg back in the queue so it doesn't disappear. and throw exception}
}
I was reading up on mySbMsg.Abandon() but it looks like that puts the message in the dead letter queue and I am not sure how to access it? and if there is a better way to error handle?

Cloud queues are a bit different than in-memory queues because they need to be robust to the possibility of the client crashing after it received the queue message but before it finished processing the message.
When a queue message is received, the message becomes "invisible" so that other clients can't pick it up. This gives the client a chance to process it and the client must mark it as completed when it is done (Azure Functions will do this automatically when you return from the function). That way, if the client were to crash in the middle of processing the message (we're on the cloud, so be robust to random machine crashes due to powerloss, etc), the server will see the absence of the completed message, assume the client crashed, and eventually resend the message.
Practically, this means that if you receive a queue message, and throw an exception (and thus we don't mark the message as completed), it will be invisible for a few minutes, but then it will show up again after a few minutes and another client can attempt to handle it. Put another way, in Azure functions, queue messages are automatically retried after exceptions, but the message will be invisible for a few minutes inbetween retries.

If you want the message to remain on the queue to be retried, the function should not swallow exception and rather throw. That way Function will not auto-complete the message and retry it.
Keep in mind that this will cause message to be retried and eventually, if exception persists, to be moved into dead-letter queue.

As per my understanding, I think what you are for is if there is an error in processing the message it needs to retry the execution instead of swallowing it. If you are using Azure Functions V2.0 you define the message handler options in the host.json
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": false,
"maxConcurrentCalls": 1
}
}
}
prefetchCount - Gets or sets the number of messages that the message receiver can simultaneously request.
autoComplete - Whether the trigger should automatically call complete after processing, or if the function code will manually call complete.
After retrying the message n(defaults to 10) number of times it will transfer the message to DLQ.

Azure cannot Add Message to Queue

I have an Azure WebJob and I use the CloudQueue to communicate to it.
From my Web Application:
logger.Info("Writing Conversion Request to Document Queue " + JsonConvert.SerializeObject(blobInfo));                       
var queueMessage = new CloudQueueMessage(JsonConvert.SerializeObject(blobInfo));                       
documentQueue.AddMessage(queueMessage);
I verify in my log file that I see the INFO statement being written.
However, when I go to my queue:
What baffles me even more ... this Queue was full of messages before, including timestamps of this evening. 
I went and cleared out my Queue, and after clearing it, it will no longer receive messages.
Has anyone ever seen this before?

As Gaurav Mantri mentioned in the comments that the message shoud be processed by your WebJob. When it is up to max attempts then will be moved to poison queue.
We could also get more details about poison messages from azure official tutorials. The following is the snippet from the tutorials.
Messages whose content causes a function to fail are called poison messages. When the function fails, the queue message is not deleted and eventually is picked up again, causing the cycle to be repeated. The SDK can automatically interrupt the cycle after a limited number of iterations, or you can do it manually.
The SDK will call a function up to 5 times to process a queue message. If the fifth try fails, the message is moved to a poison queue. The maximum number of retries is configurable.

why does azure service bus subscription report active messages but onmessage retrieves nothing

I have an azure service bus topic/subscription. When I query the subscription, I am informed there are 11,646 active messages.
However, the message loop (as implemented with subscription.onMessage) retrieves nothing.
If I submit a message, the count will go up by one, I receive it from the loop and when I .complete the message, the count goes down by one.
And before you jump to 'deadletter', there is nothing in the deadletter queue.

Can you check to see the messages in the queue don't have an existing Lock on them?

This is likely an ACL issue. We have the same issue when working with the Service Bus Explorer.