As seen in the image below, queue items remain invisible for more than 7 minutes and no processing is being done in the meantime as Live Metrics Stream shows no processor activity until after the (roughly) 7 minutes. It is like the Queue Trigger is not being activated. The visilibityTimeout is set to '00:00:05'.
This happens very often and there are no exceptions or errors. Refreshing many times shows some of the messages appearing with a dequeue count of 1 and then instantly disappearing / getting processing (even if the QueueTrigger function is stopped).
The more servers online there are that process the queue messages the more messages are likely to get stuck in such a way.
This is how it is implemented:
[FunctionName(nameof(QueueFunc))]
public async Task QueueFuncProcess(
[QueueTrigger("%QueueFuncName%", Connection = "QueueFuncConnection")] string queueItem)
Related
We have some issues with messages from Azure ServiceBus being read multiple times. Previously we had the same issue, which turned out to be due to lock timeout. Then, as the lock timed out the messages were read again, and their deliveryCount increased by 1 for each time the message was read. After this, we set the max delivery count to 1 to avoid resending of messages, and also increased the lock timeout to 5 minutes.
The current issue is a lot more strange.
First, messages are read at 10:45:34. Message locks are set to 10:50:34, and deliveryCount is 1. The reading says it succeeds, at 10:45:35.0. All good so far.
But then, at 10:45:35.8, the same messages are read again! And the delivery count is still 1. Both the sequence number and message id are the same in the two receive logs. This happens for a very small percentage of messages, something like 0,02% of the messages.
From what I understand, reading a message should either result in a success where the message should be removed, or an increase of deliveryCount, which in my case should send the message to DLQ. In these cases, neither happens.
I'm using ServiceBusTrigger, like this:
[FunctionName(nameof(ReceiveMessages))]
public async Task Run([ServiceBusTrigger(queueName: "%QueueName%", Connection = "ServiceBusConnectionString")]
string[] messages,
This seems to be like a bug in either the service bus or the library, any thoughts on what it could be?
That’s not the SDK but rather the specific entity. It sounds like the entity is corrupted. Delete and recreate it. If that doesn’t help, then open a support case.
On a different note, most of the time when delivery count is set to 1 is an indicator of something off. If you truly need at-most-once delivery guarantee, use ReceiveAndDelete mode instead of PeekLock.
I have an HTTP-trigger Azure Function that adds a message to the queue: outputQueue.AddAsync(myMessage); Then the Queue-trigger Azure Function is triggered. It adds 100 messages to the same queue. Each one of those 100 messages is dequeued by this function and processed. This processing takes about 5-7 minutes. My functionTimeout is 10 minutes. Sometimes (in 10% of the calls) the same message is being dequeued amd processed twice and even more, thought the previous processing of this message was successful. Also I paid attantion that each such redundent dequeue happens about 10 minutes after the previous dequeue of the same massage (seems to be related to my functionTimeout of 10 minutes). So it looks like after the processing is done the function is not ended and hence not deleted from the queue, which causes another instance to dequeue it.
When I look at the Failures section of Application Insights I see that approximately for 1K operations I have about 10 WebExceptions and 2 TimeoutExceptions.
WebException:
Message: The remote server returned an error: (409) Conflict.
Failed method:
Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoExceptiond
FormattedMessage: An unhandled exception has occurred. Host is shutting down.
TimeoutException:
Message: The client could not finish the operation within specified timeout. The client could not finish the operation within specified timeout.
Failed method: Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync
FormattedMessage: An unhandled exception has occurred. Host is shutting down.
I have try..catch in my Function entry point, but probably those 2 exceptions don't go to the catch block.
My host.json is as follows:
{
"functionTimeout": "00:10:00",
"version": "2.0",
"extensions": {
"queues": {
"maxPollingInterval": 1000,
"visibilityTimeout": "01:00:00",
"batchSize": 8,
"maxDequeueCount": 5,
"newBatchThreshold": 4
}
}
}
When I set "batchSize": 2 and "newBatchThreshold": 1 I have less redundant dequeues, but more instances are created (I know this by logging the IP of the server of every Azure Function call). If I have more servers that process different messages then my static data is less re-used betwen instances.
Also note that I've set the "visibilityTimeout" to 1 hour (I tried 30 minutes as well), but looks like this value is completely ignored and the message becomes visible after 10 minutes.
Any idea how I can avoid duplicate processing of the same messages? I'm thinking about writing the message info to DB after the successful processing and on every dequeue of a message check if this message was processed, say, within 1 hour from now and if so, not to process it again. Another option I'm thinking about is setting "maxDequeueCount" to 1 (I have a restore mechanism if some messages won't be processed at all due to some real failure).
BTW, those 10% of redundant processings don't cause functionality issues, but I still want to improve the performance.
In an Azure Web Job you can have functions triggered by an Azure queue in a continuously running job.
When a message has been read from a queue, it is deleted.
But if for some reason my Job crashes (think a VM restart) the current Job that was not finished will crash and I will lose the information in the message.
Is it possible to configure Azure Web Jobs not to delete messages from queue automatically, and do it manually when my job finishes?
There are two cases:
The function failed because the message was bad and we couldn't bind it - for example, you bind to an Person object but the message body is invalid JSON. In that case, we delete the message from the queue. We are going to have a mechanism of handling poison messages in a future release. (related question)
The function failed because an exception was thrown after the message was bound - for example, your own code thrown an exception. Whenever we get a message from a queue (except in case #1) we set a lease of, I think, 10 minutes:
If the function still runs after 10 minutes, we renew the lease.
If the function completes, the message is deleted.
If the function throws for any reason, we leave the message there and not renew the lease again. The message will show up in the queue again after the lease time expires (max 10 minutes).
The answer your question, if the VM restarts you are in case #2 and the message should show up again after, at most, 10 minutes.
I'm working with Azure Service Bus Queues in a request/response pattern using two queues and in general it is working well. I'm using pretty simple code from some good examples I've found. My queues are between web and worker roles, using MVC4, Visual Studio 2012 and .NET 4.5.
During some stress testing, I end up overloading my system and some responses are not delivered before the client gives up (which I will fix, not the point of this question).
When this happens, I end up with many messages left in my response queue, all well beyond their ExpiresAtUtc time. My message TimeToLive is set for 5 minutes.
When I look at the properties for a message still in the queue, it is clearly set to expire in the past, with a TimeToLive of 5 minutes.
I create the queues if they don't exist with the following code:
namespaceManager.CreateQueue(
new QueueDescription( RequestQueueName )
{
RequiresSession = true,
DefaultMessageTimeToLive = TimeSpan.FromMinutes( 5 ) // messages expire if not handled within 5 minutes
} );
What would cause a message to remain in a queue long after it is set to expire?
As I understand it, there is no background process cleaning these up, only the act of moving the queue cursor forward with a call to Receive will cause the server to skip past and dispose of messages which are expired and actually return the first message that is not expired or none if all are expired.
I am creating a mail(not email) messaging system on a website (in the same line as Facebook). I am looking at employing a queue for creating the messages. The problem I am facing is that in terms of user experience and UI, if I create a new conversation/message, while it gets added to the queue, it may sit there for 30+ seconds while the next poll runs. As the list of messages being returned comes from the non-queue table, there are limited options to how to show that the message has been sent.
Can only think of:
- When message is created, show a "messaging sending" ajax loader, and initialize a javascript poll of the queue to run every 5 secs. When the queue item no longer exists, reload the conversation list with the updated items.
- When a message is created, or page loads, query the message table, and join against the queue table for any messages created by senderid, so to user, it essentially looks like message has truly been sent. (Only issue with this, is that is technically negates the reason for a queue).