Develop Reference

how to extend the timeout of azure fuctions using durables?

let's say I have an orchestrated function that is chaining like that:
[FunctionName("E1")] //default timeout of 5 minutes
public static async Task<List<string>> Run(
[OrchestrationTrigger] IDurableOrchestrationContext context)
{
var outputs = new List<string>();
outputs.Add(await context.CallActivityAsync<string>("E1_SayHello", "Tokyo")); //takes 5 minutes to complete
outputs.Add(await context.CallActivityAsync<string>("E1_SayHello", "Seattle")); //takes 5 minutes complete
outputs.Add(await context.CallActivityAsync<string>("E1_SayHello_DirectInput", "London")); //takes 5 minutes complete
// should return ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]
return outputs;
}
now we have three functions
let's say each one needs 5 minutes to complete (on the default azure consumptionl plan) ,people say each function has it has its own timeout so we should have a total of around 15 minutes in order to complete (5+5+5) for all ,however the top level function E1 has only a timeout of 5 minutes.Will it timeout before complete because the total of all sub-functions exceeds its limit of 5?
if E1 orchestrator timedout then does the activities or subfunctions stop if the orchestrator itself timedout?

The beauty about durable functions is that it is only active when orchestrating the function. When it reaches await context.CallActivityAsync it will start E1_SayHello but it won't wait for its completion. Instead the durable function will unload and resume once E1_SayHello is completed.
What you are doing is called the Function chaining pattern and this behavior I described above is documented there like this:
Each time the code calls await, the Durable Functions framework checkpoints the progress of the current function instance. If the process or virtual machine recycles midway through the execution, the function instance resumes from the preceding await call.
So no, the durable function won't be active the whole 15 minutes.

What happens when an Azure Timer function has an execution duration that's longer shorter than the trigger period?

If an Azure function executes for say 10 minutes, but has a period of 5 minutes, does the trigger still fire at the 5 minute mark?

The easiest way to answer this question is to test it yourself.
I created a small function that did nothing more than wait for 90 seconds after executing.
The timer is set to run every minute and therefore, the test is to see if the function still executes every minute on the minute or if it delays by the 30 seconds.
The logs show that the answer to your question is ... NO
You can see that it's queued another execution though because as soon as it's finished, it starts again.
You'll also see the start times of each invocation is delayed by the 30 seconds additional it takes for the function to run.
This is the function ...
using System;
public static async Task Run(TimerInfo myTimer, ILogger log)
{
log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
await Task.Delay(90000);
}
Monitor Log
Invocations

One of the workaround I tried is:
In host.json, given the function timeout explicitly to test locally:
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"functionTimeout": "00:05:00"
}
In Azure Function, Timer trigger is scheduled at every 10 minutes:
public void Run([TimerTrigger("0 */10 * * * *")]TimerInfo myTimer, ILogger log)
{
log.LogInformation($"C# Timer trigger function1 executed at: {DateTime.Now}");
Even Running in Azure Cloud (Consumption Mode) also, there is no timeout happened as the timer is still running every 10 minutes:
As mentioned in the MS Doc and an Azure blog article, the timeout comes on Azure Function request (it means your business logic should be completed before the function timeout occurs, so at every schedule you can run the same logic by using Timer Trigger).

What happens in long running async method before delay?

I have a method that sends request to my server every second.
public async Task StartRequestProcess(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCanecllationRequested)
{
var result = await GetDataFromServerAsync();
await Task.Delay(1000, stoppingToken);
}
}
But my GetDataFromServerAsync() method takes 10 or 15 seconds sometimes.
What does in this time (10 or 15 seconds)?
Will process wait until complete long requests? Or will new requests send every second without wait?

I have a method that sends a request to my server every second.
No, you do not. Please do not open questions with false statements; it makes them hard to answer!
Your workflow is:
Request data
Wait for the data to arrive -- that's what asynchronously wait means. await is asynchronously wait.
Once the data has arrived, pause for a second, again, asynchronously
Repeat.
That is NOT the workflow "send a request every second". That is the workflow "send a request one second after the last request succeeded".
What does in this time (10 or 15 seconds)?
Asynchronously waits. You said to asynchronously wait until the data was available, and that's what it does. During the asynchronous wait other tasks can be scheduled to execute on the thread.
Will the workflow wait until the long request is completed?
Yes. It will wait asynchronously. Again, that's what await means. It means asynchronously wait.
Will new requests send every second without wait?
No. You said to wait until the data was received and then pause for one second, so that's what happens.

The whole idea of TAP (task Async pattern) is that a single thread can service lots of things "simultaneously" because it can go back to what it was doing before, any time that an await is in progress. This is why async marking on methods tends to be on every method right the way down in a hierarchy from the first point that the code you write (your controller Get method for example) through every method you call, right the way to where you need to wait something like db or network IO.
Encountering await is a bit like throwing an uncaught exception- control flow goes right the way back up the entire stack of methods that is your code, and out of the top, back to whatever was going on before, outside of your code. The difference between a thrown exception and an awaiting state machine is that when the task being awaited is done, the thread that went off to do other things will come back to where the await is and continue on from there
What it was doing before is highly contextual - in your case it's probably "waiting for a TCP client to connect and send some data"
Now, in your your code the thread goes back to what it was doing before- you say it takes 15 seconds so the thread will busy itself with other things for 15 seconds, then it will come back and wait for your 1000ms task to complete, then it will loop round and issue another request. In practice what this means is that every 16 seconds your code will make a request; not the request every second that you were hoping for. Use a timer

When you call await GetDataFromServerAsync(), execution of your method will resume once the asynchronous operation finishes, e.g., after 10 to 15 seconds. Only then will you wait asynchronously for another second.

Must StartNewAsync be awaited on the orchestration client?

I have an Azure orchestration where the orchestration client, which triggers the orchestration, threw a timeout exception.
The orchestration client function only does two things, starting two orchestrations, awaiting each as most example code suggest to.
await orchestrationClient.StartNewAsync("TableOrchestrator", updates);
await orchestrationClient.StartNewAsync("ClientOrchestrator", clientExport);
However, as I understand then the orchestration client is not a special function like the orchestration functions, so it can only run for a max of 10 minutes.
Obviously there is a high chance that the combined run time of my two orchestrations exceeds 10 minutes in total.
Questions:
Is the orchestration client state saved like the actual orchestration functions?
Do I need to await the orchestrations they do not depend on previous orchestration results?
Update Made a complete example of what my code does, and the runtimes as shown below.
It seems that starting an orchestration will await it if there is code written after, but not if the orchestration is the last statement!
Updated Questions:
Will any code after calling StartNewAsync() make the function await till the orchestration really finishes? or will e.g. log statements not trigger this behaviour?
Is it the recommended code practice that StartNewAsync() should only be called after all other code has executed?
.
public static class testOrchestration
{
[FunctionName("Start")]
public static async Task Start([TimerTrigger("0 */30 * * * *", RunOnStartup = true, UseMonitor = false)]TimerInfo myStartTimer, [OrchestrationClient] DurableOrchestrationClient orchestrationClient, ILogger log)
{
var startTime = DateTime.Now;
log.LogInformation(new EventId(0, "Startup"), "Starting Orchestror 1 ***");
await orchestrationClient.StartNewAsync("Orchestrator", "ONE");
log.LogInformation($"Elapsed time, await ONE: {DateTime.Now - startTime}");
await Task.Delay(5000);
log.LogInformation($"Elapsed time, await Delay: {DateTime.Now - startTime}");
log.LogInformation(new EventId(0, "Startup"), "Starting Orchestror 2 ***");
await orchestrationClient.StartNewAsync("Orchestrator", "TWO");
log.LogInformation($"Elapsed time, await TWO: {DateTime.Now - startTime}");
}
[FunctionName("Orchestrator")]
public static async Task<string> TestOrchestrator([OrchestrationTrigger] DurableOrchestrationContextBase context, ILogger log)
{
var input = context.GetInput<string>();
log.LogInformation($"Running {input}");
await Task.Delay(5000);
return $"Done {input}";
}
}
Running this gives me the following output:
Starting Orchestror 1 ***
Elapsed time, await ONE: 00:00:08.5445755
Running ONE
Elapsed time, await Delay: 00:00:13.5541264
Starting Orchestror 2 ***
Elapsed time, await TWO: 00:00:13.6211995
Running TWO

StartNewAsync() just schedules the orchestrators to be started (immediately). To await those calls does not mean that your initial function will really wait for the orchestrators to run - or even to actually start and finish its work.
The StartNewAsync (.NET) or startNew (JavaScript) method on the
orchestration client binding starts a new instance. Internally, this
method enqueues a message into the control queue, which then triggers
the start of a function with the specified name that uses the
orchestration trigger binding.
This async operation completes when the orchestration process is
successfully scheduled
Source
This async operation completes when the orchestration process is successfully scheduled.
So yes: You should await those calls (can also be done in parallel as Miguel suggested). But it will not take longer than a few milliseconds.

If they don't depend on each other, you can run them in parallel using:
var t1 = orchestrationClient.StartNewAsync("TableOrchestrator", updates);
var t2 = orchestrationClient.StartNewAsync("ClientOrchestrator", clientExport);
await Task.WhenAll(t1, t2);

Using Azure WebJob Timeout on Continuous Job with TimerTrigger

I have a continuous WebJob with a function using the TimerTrigger to run a process every 30 seconds. A particular call in the function occasionally and seemingly randomly hangs, causing the webjob to wait indefinitely. Current solution is notice the service has stopped, then log into the Azure Dashboard and abort it manually.
Note that I know the correct course of action is to identify the root cause and fix it. Trust me, we're working on that. In the mean time, I want to treat the symptom, and need help doing so.
I'm attempting to have the WebJob detect if status using the Timeout decorator as described in this post on the Azure WebJobs SDK: https://github.com/Azure/azure-webjobs-sdk/issues/590. Implementing the suggestion, I'm able to see that when the problematic call hangs, the Timeout is detected, but the WebJob still doesn't die. What I doing wrong here that won't kill the function to allow subsequent invocations?
Program.cs
static void Main()
{
var config = new JobHostConfiguration();
config.UseTimers();
config.FunctionTimeout = new TimeSpan(0, 15, 0);
var host = new JobHost(config);
Functions.Initialize();
host.RunAndBlock();
}
Functions.cs
[Singleton]
[Timeout("00:05:00")]
public async static Task PeriodicProcess([TimerTrigger("00:00:30", RunOnStartup = true)] TimerInfo timer, CancellationToken cancelToken, TextWriter log)
{
log.WriteLine("-- Processing Begin --");
List<Emails> cases = GetEmailsAndWhatNot();
foreach (Email e in Emails)
{
try
{
ProblematicFunction_SendEmail(e, log);
}
catch(Exception e)
{
// do stuff
}
}
log.WriteLine("-- Processing End -- ");
}
public static void ProblematicFunction_SendEmail(Email e, TextWriter log)
{
// send email
}
WebJob Output During Issues
-- Processing Begin --
Timeout value of 00:05:00 exceeded by function 'Functions.PeriodicProcess' (Id: '0f7438bd-baad-451f-95a6-9461f35bfb2d'). Initiating cancellation.
Despite the webjob initiating cancellation, the function doesn't die. Do I need to monitor the CancellationToken? How far down do I need to propogate asynchronous calling? What am I missing here that will actually abort the process?

As TimerTrigger states about TimerTrigger:
Singleton Locks
TimerTrigger uses the Singleton feature of the WebJobs SDK to ensure that only a single instance of your triggered function is running at any given time.
Scheduling
If your function execution takes longer than the timer interval, another execution won't be triggered until after the current invocation completes. The next execution is scheduled after the current execution completes.
Here is my test for this scenario, you could refer to it:
Use CancellationToken.None and never propogate the cancellation token
Note: The function PeriodicProcess would be time out after 30 s, but the Time-consuming job is still running, and after the long-running job has done, the Processing End log would be printed.
Propogate the cancellation token
Note: If we propogate the cancellation token, the Time-consuming job would be cancelled immediately.
Code snippet
[Timeout("00:00:30")]
[Singleton]
public async static Task PeriodicProcess([TimerTrigger("00:00:10", RunOnStartup = true)] TimerInfo timer, CancellationToken cancelToken, TextWriter log)
{
log.WriteLine($"-- [{DateTime.Now.ToString()}] Processing Begin --");
try
{
await longRunningJob(log, cancelToken);
}
catch (Exception e)
{
// do stuff
}
log.WriteLine($"-- [{DateTime.Now.ToString()}] Processing End -- ");
}
private async static Task longRunningJob(TextWriter log, CancellationToken cancelToken)
{
log.WriteLine($"-- [{DateTime.Now.ToString()}] Begin Time-consuming jobs --");
await Task.Delay(TimeSpan.FromMinutes(1), cancelToken);
log.WriteLine($"-- [{DateTime.Now.ToString()}] Complete Time-consuming jobs --");
}