We have some issues with messages from Azure ServiceBus being read multiple times. Previously we had the same issue, which turned out to be due to lock timeout. Then, as the lock timed out the messages were read again, and their deliveryCount increased by 1 for each time the message was read. After this, we set the max delivery count to 1 to avoid resending of messages, and also increased the lock timeout to 5 minutes.
The current issue is a lot more strange.
First, messages are read at 10:45:34. Message locks are set to 10:50:34, and deliveryCount is 1. The reading says it succeeds, at 10:45:35.0. All good so far.
But then, at 10:45:35.8, the same messages are read again! And the delivery count is still 1. Both the sequence number and message id are the same in the two receive logs. This happens for a very small percentage of messages, something like 0,02% of the messages.
From what I understand, reading a message should either result in a success where the message should be removed, or an increase of deliveryCount, which in my case should send the message to DLQ. In these cases, neither happens.
I'm using ServiceBusTrigger, like this:
[FunctionName(nameof(ReceiveMessages))]
public async Task Run([ServiceBusTrigger(queueName: "%QueueName%", Connection = "ServiceBusConnectionString")]
string[] messages,
This seems to be like a bug in either the service bus or the library, any thoughts on what it could be?
That’s not the SDK but rather the specific entity. It sounds like the entity is corrupted. Delete and recreate it. If that doesn’t help, then open a support case.
On a different note, most of the time when delivery count is set to 1 is an indicator of something off. If you truly need at-most-once delivery guarantee, use ReceiveAndDelete mode instead of PeekLock.
Related
If I have a service bus brokered message receiver configured. and it fails for any reason. I call on it
message.abandon();
however this means the message will be back again in the queue/subscription.
can i configure a timeout after which the same message is available in the queue for processing.
For example: if there is only one message in the queue. and it's failing, then it is not good to keep processing it every second or every minute. if i configure something, that can make sure, the failed/abandoned message only reappears after 30 mins . then it is useful.
Any suggestions?
When you abandon a message, you cannot supply a "cool off" time. The message will be available right away. It won't be dead-lettered until MaxDeliveryCount attempts have been exhausted. Once all those processing attempts have been used up, the message will go to the dead-letter queue.
If you need to postpone message processing, there are several options.
Deferring a message
You could defer a message using BrokeredMessage.DeferAsync(). The message will go back to the queue for future processing and a SequenceNumber of the message will be returned. The caveat with this approach is the need to hold on to the SequenceNumber in order to retrieve the message later. If you happened to lose SequenceNumber, it is still possible to browse for deferred messages and retrieve those. More information here.
Scheduling a new future message
Another option would be to clone an incoming failing message, schedule it for some time in future using BrokeredMessage.ScheduledEnqueueTimeUTC and completing the original message. With this option, I'd recommend to send the new message scheduled in future to be dispatch using send-via feature, also known as Transaction Processing, to leverage atomic operation of completing the incoming message and sending the outgoing one. This way the code will not produce "ghost" message if completion fails. More information here.
Scheduling using client, not message
Another option is to schedule using a client and not BrokeredMessage using client.ScheduleMessageAsync(). It will return aSequenceNumber` you need to hold on to, but using this API a message can be cancelled at any point in time w/o waiting for the schedule time to arrive or receiving the message. More information here.
I have an Azure WebJob and I use the CloudQueue to communicate to it.
From my Web Application:
logger.Info("Writing Conversion Request to Document Queue " + JsonConvert.SerializeObject(blobInfo));
var queueMessage = new CloudQueueMessage(JsonConvert.SerializeObject(blobInfo));
documentQueue.AddMessage(queueMessage);
I verify in my log file that I see the INFO statement being written.
However, when I go to my queue:
What baffles me even more ... this Queue was full of messages before, including timestamps of this evening.
I went and cleared out my Queue, and after clearing it, it will no longer receive messages.
Has anyone ever seen this before?
As Gaurav Mantri mentioned in the comments that the message shoud be processed by your WebJob. When it is up to max attempts then will be moved to poison queue.
We could also get more details about poison messages from azure official tutorials. The following is the snippet from the tutorials.
Messages whose content causes a function to fail are called poison messages. When the function fails, the queue message is not deleted and eventually is picked up again, causing the cycle to be repeated. The SDK can automatically interrupt the cycle after a limited number of iterations, or you can do it manually.
The SDK will call a function up to 5 times to process a queue message. If the fifth try fails, the message is moved to a poison queue. The maximum number of retries is configurable.
We need a deterministic/consistent way of knowing if a particular queue (local/remote) is backed up (i.e. can no longer receive a message) or not before sending another message to it. The best solution I've seen so far is to get a message count (in an efficient way) on the given queue & then make a determination to send the message or not.
What is the best practice here outside of catching exception X or Y & re-trying? Thanks
If a queue has reached it's quota size then new messages will be rejected. You would need to monitor the total message size for the queue, not the total message number. The number of messages is not used by MSMQ to determine whether quota is reached or not.
Alternatively, if you are using the Dead Letter Queue, you would be able to see a rejected message with a status that indicated the destination queue had reached quota. I would regard that method as more efficient than remotely checking queue stats.
I'm using an Azure Service Bus Queue with Session based messaging enabled. To consume from the queue I register an IMessageSessionAsyncHandler and then process the message in the OnMessageAsync method.
This issue I'm seeing is that if I abandon a message for whatever reason, rather than being received again immediately, I receive the next message in the session and only after processing that message, do I receive the first message again (assuming only two messages in the session).
As an example, lets say I have a queue with 2 messages on it, both with the same SessionId. The two messages have sequence numbers of 1 and 2 respectively. I start receiving and get message with sequence 1, as expected. If I then abandon this message using message.Abandon (the reason for abandoning is irrelevant), I immediately get the next message in the session (sequence number 2). Only after handling (or abandoning) this second message, do I get the first message again.
This behaviour I'm seeing isn't what I'd expect from abandoning a message and isn't consistent with other ways of using the queue. I've tested this same example in the following scenarios
without the use of an IMessageSessionAsyncHandler and instead just manually accepting a message session.
without the use of sessions and instead just having two independent messages on the queue.
In both scenarios, I see the expected bahaviour, in that when I abandon a message it is always guaranteed to be the next message received, unless the max delivery count is exceeded and it is dead-lettered.
My question is this: Is the behaviour I'm seeing with the use of an IMessageSessionAsyncHandler expected, or is this a bug in the Service Bus Library? If this is not a bug, can anyone give me an explaination for why this behaves different to the other ways of receiving?
When you Register a session handler on the Queueclient, Prefetch is turned on internally to improve latency and throughput of the receivers. Unfortunately for the IMessageSessionAsyncHandler scenario this behavior cannot be overriden. One option is to abandon the Session itself when you encounter a message in a session which needs to be abandoned, this will ensure that the messages are delivered in order.
I have an NServiceBus application for which a given message may not be processed due to some external event not having taken place. Because this other event is not an NSB event I can't implement sagas properly.
However, rather than just re-queuing the message (which would cause a loop until that external event has occurred), I'm wrapping the message in another message (DelayMessage) and queuing that instead. The DelayMessage is picked up by a different service and placed in a database until the retry interval expires. At which point, the delay service re-queues the message on the original queue so another attempt can be made.
However, this can happen more than once if that external event still hasn't taken place, and in the case where that even never happens, I want to limit the number of round trips the message takes. This means the DelayMessage has a MaxRetries property, but that is lost when the delay service queues the original message for the retry.
What other options am I missing? I'm happy to accept that there's a totally different solution to this problem.
Consider implementing a saga which stores that first message, holding on to it until the second message arrives. You might also want the saga to open a timeout as well so that your process won't wait indefinitely if that second message got lost or something.