I have azure functions (C# v1 functions--non scripted) that use DocumentDBAttribute bindings for both reading and writing documents. How do those bindings respond to throttling in the following situations?
Writing an item by adding it to an ICollector
Reading an item by providing an Id
This is for functions v1.
First case:
//input binding
[DocumentDB(ResourceNames.APCosmosDBName,
ResourceNames.EpisodeOfCareCollectionName,
ConnectionStringSetting = "APCosmosDB",
CreateIfNotExists = true)] ICollector<EOC> eoc,
//...
eoc.Add(new EOC()); //what happens here if throttling is occuring?
Second case:
[DocumentDB(ResourceNames.ORHCasesDBName, ResourceNames.ORHCasesCollectionName, ConnectionStringSetting = "ORHCosmosDBCases", CreateIfNotExists = true, Id = "{id}")] string closedCaseStr,
Both input and output bindings use CosmosDB SDK which has the retry mechanism in place.
By default, SDK retries 9 times on a throttled result, after that, the exception is bubbled and you Function will error. Depending on the trigger type, it will fail HTTP call, put the message back to the queue etc.
The retries respect the timing recommendation returned by Cosmos DB:
When a client is sending requests faster than the allowed rate, the service will return HttpStatusCode 429 (Too Many Request) to rate limit the client. The current implementation in the SDK will then wait for the amount of time the service tells it to wait and retry after the time has elapsed.
At the moment, there is no way to configure the bindings with a policy other than default.
Related
If the processing of an Azure Service Bus message depends on another resource, e.g. an API or a database service, and this resource is not available, not calling CompleteMessageAsync() is not an option, because the message will be immediately received again until the Max Delivery Count is reached, and then put into the DLQ. If an API is down for maintenance, we want to wait a bit before retrying.
One of the answers to this question has the general steps for deferring and receiving deferred messages. This is a little better than Microsoft's documentation, but not enough for me to understand the intent of the API, and how it is to be implemented in a hosted service that basically sits in ServiceBusProcessor.StartProcessingAsync all day long.
This is the basic structure of my service:
public class ServiceBusWatcher : IHostedService, IDisposable
{
public Task StartAsync(CancellationToken stoppingToken)
{
ReceiveMessagesAsync();
return Task.CompletedTask;
}
private async void ReceiveMessagesAsync()
{
ServiceBusClient client = new ServiceBusClient(connectionString);
processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions());
processor.ProcessMessageAsync += MessageHandler;
await processor.StartProcessingAsync();
}
async Task MessageHandler(ProcessMessageEventArgs args)
{
// a dependency is not available that allows me to process a message. so:
await args.DeferMessageAsync(args.Message);
Once the message is deferred, it is my understanding that the processor will not get to it anymore (or will it?). Instead, I have to use ReceiveDeferredMessageAsync() to receive it, along with the sequence number of the originally received message.
In my case, it will make sense to wait minutes or hours before trying again.
This could be done with a separate service that uses a timer and an explicit call to ReceiveDeferredMessageAsync(), as opposed to using a ServiceBusProcessor. I also suppose that the deferred message sequence numbers will have to be persisted in non-volatile storage so that they don't get lost.
Does this sound like a viable approach? I don't like having to remember its sequence numbers so that I can get to a message later. It goes against everything that using a message queue brings to the table in the first place.
Or, instead of deferring, I could just post a new "internal" message with the sequence number and use the ScheduledEnqueueTimeUtc property to delay receiving it. Once I receive this message, I could call ReceiveDeferredMessageAsync() with that sequence number to get to the original message. This seems elegant at the surface, but messages could quickly multiply if there is a longer outage of a dependency.
Another idea that could work without another service: I could complete and repost the payload of the message and set ScheduledEnqueueTimeUtc to a time in the future, as described in another answer to the question I mentioned earlier. Assuming that this works (Microsoft's documentation does not mention what this property is for), it seems simple and clean, and I like simple.
How have you solved this? Is there a better/preferred way that balances low complexity with high robustness without requiring a large amount of code?
Deferring a message works when you know what message you want to retrieve later and your receiver will have the message sequence number saved to retrieve the deferred message. If the receiver has no ability to save message sequence number, the delaying the message is a better option. Delaying a message will mean to copy the original message data into a newly scheduled one and completing the original message. That way the consumer doesn't have to neither hold on to the message sequence number nor initiate the retrieval of a specific message.
I have an Azure Servicebus function trigger which is supposed to call an external endpoint whenever an event is placed in its designated queue. However, when the function picks up the event and is going to make the outgoing request it fails with the following log message
Executed 'AzureServicebusTrigger' (Failed, Id=cb218eb5-300b-40c0-8a4e-81f977b9cd5c)
An attempt was made to access a socket in a way forbidden by its access permissions`
My function:
public static class AzureServicebusTrigger
{
private static HttpClient _httpClient = new HttpClient();
[FunctionName("AzureServicebusTrigger")]
public static async Task Run([ServiceBusTrigger("receiver", Connection = "ServiceBusConnectionString")]
string myQueueItem, ILogger log)
{
var request = await _httpClient.GetAsync("http://<ip>:7000");
var result = await request.Content.ReadAsStringAsync();
}
}
What is this permission?
And how do I enable such that outgoing requests from my function become possible?
I have an educated guess. I've seen this before with Web Apps. It can occur if you've exhausted the number of outbound connections available for your chosen SKU in your pricing tier.
I'm guessing you're on the Azure Function consumption plan. If so, you are limited to 600 active connections (see here). So if you're seeing this problem intermittently, try moving to the Premium tier or an App Service plan of Standard or higher.
I think Rob Reagan is correct. To help you accept his answer as correct I suggest the following.
Can you inspect how many function execution and servers (function host instances) did you have at the moment of the failure. You can do that by adding Application Insights and writing such a query.
performanceCounters
| where cloud_RoleName =~ 'YourFunctionName'
| where timestamp > ago(30d)
| distinct cloud_RoleInstance
Or by reproducing the load and watching live monitor.
I have an Azure Function with an input binding to an Event Hub.
public static async Task Run(TraceWriter log, string eventHubMessage)
When the function is triggered, how many messages does it receive per execution by default?
Is it 1 execution = 1 message?
I have read the documentation and understand you can set these properties in the function's host.json file:
"eventHub": {
// The maximum event count received per receive loop. The default is 64.
"maxBatchSize": 64,
// The default PrefetchCount that will be used by the underlying EventProcessorHost.
"prefetchCount": 256
}
Does maxBatchSize mean I will receive 64 messages in 1 execution?
By default it's going to be 1 by 1 processing, but you can do batches too. Change the signature of your function to
public static async Task Run(TraceWriter log, string[] eventHubMessages)
(if you change the name like I did, rename the binding parameter too)
Reference github issue.
#Mikhail is correct. I'd just like to add the following:
If you use the default EventHub-Trigger C# template, the Function created will process 1 message per execution.
If you need each execution to process in batches, change the following:
a. In function.json, add the property "cardinality":"many" as shown here.
b. In run.csx, modify Function signature and process messages in a loop, e.g.,
public static async Task Run(TraceWriter log, string[] eventHubMessages)
{
foreach(string message in eventHubMessages)
{
// process messages
}
}
The host.json configuration you specified in the question allows you to experiment with the correct batch size and prefetch buffer to meet the needs of your workflow.
Additional comments:
Under the Consumption Plan, a Function is currently allowed a max default 5-minute execution time (configurable up to 10 mins --Added on 11/30/2017). You should experiment with the maxBatchSize and prefetchCount setting to ensure that a typical execution of the batch will complete within the timeframe.
The prefetchCount should be 3-4 times the maxBatchSize.
Each Function host instance is backed by a single EventProcessorHost (EPH). EPH uses a checkpointing mechanism to mark the last successfully processed message. A Function execution could terminate prematurely due to uncaught exceptions in the Function code host crashing, timeout or partition lease lost, resulting in an unsuccessful checkpoint. When the Function execution restarts again, the batch retrieved will have messages from the last known checkpoint. Setting a very high value for maxBatchSize will also mean that you must re-process a large batch. EventHub guarantees at-least-once delivery but not at-most-once delivery. Azure Functions will not attempt to change that behavior. If having only unique messages is a priority, you will need to handle de-duplication in your downstream workflows.
Working with a Azure Service Bus Topic currently and running into an issue receiving my messages using ReceiveBatch method. The issue is that the expected results are not actually the results that I am getting. Here is the basic code setup, use cases are below:
SubscriptionClient client = SubscriptionClient.CreateFromConnectionString(connectionString, convoTopic, subName);
IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100);
foreach (BrokeredMessage message in messageList)
{
try
{
Console.WriteLine(message.GetBody<string>() + message.MessageId);
message.Complete();
}
catch (Exception ex)
{
message.Abandon();
}
}
client.Close();
MessageBox.Show("Done");
Using the above code, if I send 4 messages, then poll on the first run through I get the first message. On the second run through I get the other 3. I'm expecting to get all 4 at the same time. It seems to always return a singular value on the first poll then the rest on subsequent polls. (same result with 3 and 5 where I get n-1 of n messages sent on the second try and 1 message on the first try).
If I have 0 messages to receive, the operation takes between ~30-60 seconds to get the messageList (that has a 0 count). I need this to return instantly.
If I change the code to IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100, new Timespan(0,0,0)); then issue #2 goes away because issue 1 still persists where I have to call the code twice to get all the messages.
I'm assuming that issue #2 is because of a default timeout value which I overwrite in #3 (though I find it confusing that if a message is there it immediately responds without waiting the default time). I am not sure why I never receive the full amount of messages in a single ReceiveBatch however.
The way I got ReceiveBatch() to work properly was to do two things.
Disable Partitioning in the Topic (I had to make a new topic for this because you can't toggle that after creation)
Enable Batching on each subscription created like so:
List item
SubscriptionDescription sd = new SubscriptionDescription(topicName, orgSubName);
sd.EnableBatchedOperations = true;
After I did those two things, I was able to get the topics to work as intended using IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100, new TimeSpan(0,0,0));
I'm having a similar problem with an ASB Queue. I discovered that I could mitigate it somewhat by increasing the PrefetchCount on the client prior to receiving the batch:
SubscriptionClient client = SubscriptionClient.CreateFromConnectionString(connectionString, convoTopic, subName);
client.PrefetchCount = 100;
IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100);
From the Azure Service Bus Best Practices for Performance Improvements Using Service Bus Brokered Messaging:
Prefetching enables the queue or subscription client to load additional messages from the service when it performs a receive operation.
...
When using the default lock expiration of 60 seconds, a good value for
SubscriptionClient.PrefetchCount is 20 times the maximum processing rates of all receivers of the factory. For example, a factory creates 3 receivers, and each receiver can process up to 10 messages per second. The prefetch count should not exceed 20*3*10 = 600.
...
Prefetching messages increases the overall throughput for a queue or subscription because it reduces the overall number of message operations, or round trips. Fetching the first message, however, will take longer (due to the increased message size). Receiving prefetched messages will be faster because these messages have already been downloaded by the client.
Just a few more pieces to the puzzle. I still couldn't get it to work even after Enable Batching and Disable Partitioning - I still had to do two ReceiveBatch calls. I did find however:
Restarting the Service Bus services (I am using Service Bus for Windows Server) cleared up the issue for me.
Doing a single RecieveBatch and taking no action (letting the message locks expire) and then doing another ReceiveBatch caused all of the messages to come through at the same time. (Doing an initial ReceiveBatch and calling Abandon on all of the messages didn't cause that behavior.)
So it appears to be some sort of corruption/bug in Service Bus's in-memory cache.
A long time ago, Amazon introduced the long polling feature. And with that, it is possible to configure on the Queue the "Receive Message Wait Time" parameter. According to the documentation, a valid value falls in the range 0 - 20 seconds.
In the client, we can also configure this parameter on each MessageReceiveRequest. I'm using the AWS SDK for .NET.
var receiveRequest = new ReceiveMessageRequest
{
QueueUrl = "https://queue-url-goes-here.com",
MaxNumberOfMessages = 10,
VisibilityTimeout = 30,
WaitTimeSeconds = 20 // This should tell if we want long polling or not
};
Questions:
a) What is the relationship between the Receive Message Wait Time configured in the Queue VS the WaitTimeSeconds attribute set in the Message Receive Request? Will they work independently? Or the value set in the client overrides the value set in the Queue (for that single request).
b) Under certain conditions, can the C# client time out? I am thinking about setting both values to the max (20 seconds) but I'm afraid that might cause the C# long polling operation to Time Out.
c) What is the best-practice. WaitTimeSeconds > Receive Message Wait Time?
a) As noted in pastk's answer, the WaitTimeSeconds on the message will override the Receive Message Wait Time configured in the queue. See the long polling documentation for details.
b) The AWS SDK for .NET uses System.Net.HttpWebRequest under the hood - its default timeout is 100 seconds. If you're using the defaults, setting the WaitTimeSeconds to 20 seconds will not cause the operation to time out.
c) There is no best practice prescribed by Amazon on this point. Do whatever you think is best for your scenario.
Its just a different way to set wait time you need.
Request-level wait time always overrides queues value: "A value set between 1 to 20 for the WaitTimeSeconds parameter for ReceiveMessage has priority over any value set for the queue attribute ReceiveMessageWaitTimeSeconds." (http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html)
In case some of queue's consumers need to use long polling and others don't then it makes sense to use per-request wait time setting, otherwise simpler to use queue's setting.