A long time ago, Amazon introduced the long polling feature. And with that, it is possible to configure on the Queue the "Receive Message Wait Time" parameter. According to the documentation, a valid value falls in the range 0 - 20 seconds.
In the client, we can also configure this parameter on each MessageReceiveRequest. I'm using the AWS SDK for .NET.
var receiveRequest = new ReceiveMessageRequest
{
QueueUrl = "https://queue-url-goes-here.com",
MaxNumberOfMessages = 10,
VisibilityTimeout = 30,
WaitTimeSeconds = 20 // This should tell if we want long polling or not
};
Questions:
a) What is the relationship between the Receive Message Wait Time configured in the Queue VS the WaitTimeSeconds attribute set in the Message Receive Request? Will they work independently? Or the value set in the client overrides the value set in the Queue (for that single request).
b) Under certain conditions, can the C# client time out? I am thinking about setting both values to the max (20 seconds) but I'm afraid that might cause the C# long polling operation to Time Out.
c) What is the best-practice. WaitTimeSeconds > Receive Message Wait Time?
a) As noted in pastk's answer, the WaitTimeSeconds on the message will override the Receive Message Wait Time configured in the queue. See the long polling documentation for details.
b) The AWS SDK for .NET uses System.Net.HttpWebRequest under the hood - its default timeout is 100 seconds. If you're using the defaults, setting the WaitTimeSeconds to 20 seconds will not cause the operation to time out.
c) There is no best practice prescribed by Amazon on this point. Do whatever you think is best for your scenario.
Its just a different way to set wait time you need.
Request-level wait time always overrides queues value: "A value set between 1 to 20 for the WaitTimeSeconds parameter for ReceiveMessage has priority over any value set for the queue attribute ReceiveMessageWaitTimeSeconds." (http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html)
In case some of queue's consumers need to use long polling and others don't then it makes sense to use per-request wait time setting, otherwise simpler to use queue's setting.
Related
I am aiming for exactly-once-delivery in Orleans. I have taken care of at-most-once by using sequence numbers and was relying on Orleans retry mechanism for at-least-once. So, I have configured something like this - increased timeout limit to 2 min and set resendCount to 60. .Configure<SiloMessagingOptions>(options => { options.ResendOnTimeout = true; options.MaxResendCount = 60; options.ResponseTimeout = new TimeSpan(0,2,0); });
Is this enough?
Is there any way to know my SiloMessagingOptions like resendCount, after the Silo has been started?
How does Orleans determine that a message has failed? If I don't await the Task and the message fails, does Orleans still detect it and resend the message? Is there any way for the application to know that a message has failed?
What benefit do I get, in context of message reliability, by awaiting a Task (assuming I don't care about the return value of the Task)?
UPDATE: I have been told that using the at-least-once delivery of Orleans is not the best way to go and that I should use features like reminders instead. The question above is here just to get some doubts cleared.
I have read the documentation and all details about how RetryPolicy object is used in Azure Service Bus Queue Client are clear to me, except one: TimeBuffer property.
Definition in the documentation says:
"The termination time buffer associated with the retry. Retry attempts
will be abandoned if the remaining time is less than TimeBuffer."
But, what is the "remaining time"? Can someone give an example of how is this used?
Here is how I see the situation: If the request fails and error is transient, operation will be repeated until success or until MaxRetryCount is reached. In the meantime intervals between attempts will grow depending on MinimalBackoff and DeltaBackoff until MaximumBackoff reached. From that point, intervals between attempts will be constant and equal to MaximumBackoff. How is TimeBuffer used in this scenario?
One more thing I forgot, so, I edited the question. How to set DeltaBackoff and TimeBuffer when only constructor that allows setting those values is obsolete, and DeltaBackoff property has no setter and is readonly?
TerminationTimeBuffer is a property that is used to limit the total time for the operation to complete.
The termination time buffer associated with the retry. Retry attempts will be abandoned if the remaining time is less than TimeBuffer.
Consider a scenario with following configurations
MinimumBackoff = 0
MaximumBackoff = 30 sec
DeltaBackoff = 300 msec
TimeBuffer = 300 msec
MaxRetryCount = 2
It works as
Attempt 1: Delay 0 sec
Attempt 2: Delay ~300 msec
Attempt 3: Delay ~900 msec
Here the "remaining time" after the 2nd attempt would be 29.7 sec and 29.1 sec after the 3rd attempt.
I have azure functions (C# v1 functions--non scripted) that use DocumentDBAttribute bindings for both reading and writing documents. How do those bindings respond to throttling in the following situations?
Writing an item by adding it to an ICollector
Reading an item by providing an Id
This is for functions v1.
First case:
//input binding
[DocumentDB(ResourceNames.APCosmosDBName,
ResourceNames.EpisodeOfCareCollectionName,
ConnectionStringSetting = "APCosmosDB",
CreateIfNotExists = true)] ICollector<EOC> eoc,
//...
eoc.Add(new EOC()); //what happens here if throttling is occuring?
Second case:
[DocumentDB(ResourceNames.ORHCasesDBName, ResourceNames.ORHCasesCollectionName, ConnectionStringSetting = "ORHCosmosDBCases", CreateIfNotExists = true, Id = "{id}")] string closedCaseStr,
Both input and output bindings use CosmosDB SDK which has the retry mechanism in place.
By default, SDK retries 9 times on a throttled result, after that, the exception is bubbled and you Function will error. Depending on the trigger type, it will fail HTTP call, put the message back to the queue etc.
The retries respect the timing recommendation returned by Cosmos DB:
When a client is sending requests faster than the allowed rate, the service will return HttpStatusCode 429 (Too Many Request) to rate limit the client. The current implementation in the SDK will then wait for the amount of time the service tells it to wait and retry after the time has elapsed.
At the moment, there is no way to configure the bindings with a policy other than default.
When Webjobs get a message from a queue on Azure Storage via QueueTrigger, it leases the message (makes it invisible). If the triggering function (of webjob) takes a long time to process the message, is this lease automatically extended? Or should I handle that in the function?
On this link Windows Azure Queues: Improved Leases, Progress Tracking, and Scheduling of Future Work, the author states that "A lease on the message can be extended by the worker that did the original dequeue so that it can continue processing the message"
Note: I've tried a webjob (with a QueueTrigger) which waits for 20 minutes.
//Write Log
Thread.Sleep(1200000);
//Write Log
It is completed successfully. And during this time no other webjob instance try to attempt for the same queue item (It did not become visible). Therefore it seems that an auto-renew mechanism for leases exists. Anyhow I am waiting for an answer from a Microsoft employee or with an official link (msdn, azure, ...).
Yes, your lease is automatically extended. It is 10 minutes each time.
See this answer here [1] by a Microsoft employee referring to the docs and comments on azure.microsoft.com [2].
EDIT (long answer)
In addition, an examination of the source code, starting with the QueueListener class at https://github.com/Azure/azure-webjobs-sdk/blob/cfc875a7f00e595410c0603e6ca65537025490a9/src/Microsoft.Azure.WebJobs.Host/Queues/Listeners/QueueListener.cs indicates the same.
The code in QueueListener has the relevant parts on line 138, where the 10 minute visibilityTimeout variable is defined:
TimeSpan visibilityTimeout = TimeSpan.FromMinutes(10); // long enough to process the job
That variable is then passed along to ProcessMessageAsync, which starts a timer defined in the method CreateUpdateMessageVisibilityTimer with that same value. The 10 minute value is used to determine when the first and next update the visibility timeout also (by halving it and creating an instance of the LinearSpeedupStrategy class).
Eventually, in the class UpdateQueueMessageVisibilityCommand [3], you will find that the UpdateMessageAsync method on the queue is called with that same 10 minute renewal.
The LinearSpeedupStrategy will renew again after 5 minutes, unless the renewal failed in which case it will try again after 1 minute (as defined in QueueListener).
[1] Azure Storage Queue and multiple WebJobs instances: will QueueTrigger set the message lease time on triggered?
[2] https://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-webjobs-sdk-get-started/
[3] https://github.com/Azure/azure-webjobs-sdk/blob/cfc875a7f00e595410c0603e6ca65537025490a9/src/Microsoft.Azure.WebJobs.Host/Queues/Listeners/UpdateQueueMessageVisibilityCommand.cs
You can use method(Java code):
queue.retrieveMessage()
to get a message from a queue on azure storage. It will be visible after 30 seconds by default.
If you want to extend the lease, you can use code below:
CloudQueueMessage updateMessage = queue.retrieveMessage();
EnumSet<MessageUpdateFields> updateFields = EnumSet.of(MessageUpdateFields.CONTENT, MessageUpdateFields.VISIBILITY);
queue.updateMessage(updateMessage, 60, updateFields, null, null);
This means your message will be able to be processed for another 60 seconds.
Working with a Azure Service Bus Topic currently and running into an issue receiving my messages using ReceiveBatch method. The issue is that the expected results are not actually the results that I am getting. Here is the basic code setup, use cases are below:
SubscriptionClient client = SubscriptionClient.CreateFromConnectionString(connectionString, convoTopic, subName);
IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100);
foreach (BrokeredMessage message in messageList)
{
try
{
Console.WriteLine(message.GetBody<string>() + message.MessageId);
message.Complete();
}
catch (Exception ex)
{
message.Abandon();
}
}
client.Close();
MessageBox.Show("Done");
Using the above code, if I send 4 messages, then poll on the first run through I get the first message. On the second run through I get the other 3. I'm expecting to get all 4 at the same time. It seems to always return a singular value on the first poll then the rest on subsequent polls. (same result with 3 and 5 where I get n-1 of n messages sent on the second try and 1 message on the first try).
If I have 0 messages to receive, the operation takes between ~30-60 seconds to get the messageList (that has a 0 count). I need this to return instantly.
If I change the code to IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100, new Timespan(0,0,0)); then issue #2 goes away because issue 1 still persists where I have to call the code twice to get all the messages.
I'm assuming that issue #2 is because of a default timeout value which I overwrite in #3 (though I find it confusing that if a message is there it immediately responds without waiting the default time). I am not sure why I never receive the full amount of messages in a single ReceiveBatch however.
The way I got ReceiveBatch() to work properly was to do two things.
Disable Partitioning in the Topic (I had to make a new topic for this because you can't toggle that after creation)
Enable Batching on each subscription created like so:
List item
SubscriptionDescription sd = new SubscriptionDescription(topicName, orgSubName);
sd.EnableBatchedOperations = true;
After I did those two things, I was able to get the topics to work as intended using IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100, new TimeSpan(0,0,0));
I'm having a similar problem with an ASB Queue. I discovered that I could mitigate it somewhat by increasing the PrefetchCount on the client prior to receiving the batch:
SubscriptionClient client = SubscriptionClient.CreateFromConnectionString(connectionString, convoTopic, subName);
client.PrefetchCount = 100;
IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100);
From the Azure Service Bus Best Practices for Performance Improvements Using Service Bus Brokered Messaging:
Prefetching enables the queue or subscription client to load additional messages from the service when it performs a receive operation.
...
When using the default lock expiration of 60 seconds, a good value for
SubscriptionClient.PrefetchCount is 20 times the maximum processing rates of all receivers of the factory. For example, a factory creates 3 receivers, and each receiver can process up to 10 messages per second. The prefetch count should not exceed 20*3*10 = 600.
...
Prefetching messages increases the overall throughput for a queue or subscription because it reduces the overall number of message operations, or round trips. Fetching the first message, however, will take longer (due to the increased message size). Receiving prefetched messages will be faster because these messages have already been downloaded by the client.
Just a few more pieces to the puzzle. I still couldn't get it to work even after Enable Batching and Disable Partitioning - I still had to do two ReceiveBatch calls. I did find however:
Restarting the Service Bus services (I am using Service Bus for Windows Server) cleared up the issue for me.
Doing a single RecieveBatch and taking no action (letting the message locks expire) and then doing another ReceiveBatch caused all of the messages to come through at the same time. (Doing an initial ReceiveBatch and calling Abandon on all of the messages didn't cause that behavior.)
So it appears to be some sort of corruption/bug in Service Bus's in-memory cache.