If an Azure function executes for say 10 minutes, but has a period of 5 minutes, does the trigger still fire at the 5 minute mark?
The easiest way to answer this question is to test it yourself.
I created a small function that did nothing more than wait for 90 seconds after executing.
The timer is set to run every minute and therefore, the test is to see if the function still executes every minute on the minute or if it delays by the 30 seconds.
The logs show that the answer to your question is ... NO
You can see that it's queued another execution though because as soon as it's finished, it starts again.
You'll also see the start times of each invocation is delayed by the 30 seconds additional it takes for the function to run.
This is the function ...
using System;
public static async Task Run(TimerInfo myTimer, ILogger log)
{
log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
await Task.Delay(90000);
}
Monitor Log
Invocations
One of the workaround I tried is:
In host.json, given the function timeout explicitly to test locally:
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"functionTimeout": "00:05:00"
}
In Azure Function, Timer trigger is scheduled at every 10 minutes:
public void Run([TimerTrigger("0 */10 * * * *")]TimerInfo myTimer, ILogger log)
{
log.LogInformation($"C# Timer trigger function1 executed at: {DateTime.Now}");
Even Running in Azure Cloud (Consumption Mode) also, there is no timeout happened as the timer is still running every 10 minutes:
As mentioned in the MS Doc and an Azure blog article, the timeout comes on Azure Function request (it means your business logic should be completed before the function timeout occurs, so at every schedule you can run the same logic by using Timer Trigger).
Related
let's say I have an orchestrated function that is chaining like that:
[FunctionName("E1")] //default timeout of 5 minutes
public static async Task<List<string>> Run(
[OrchestrationTrigger] IDurableOrchestrationContext context)
{
var outputs = new List<string>();
outputs.Add(await context.CallActivityAsync<string>("E1_SayHello", "Tokyo")); //takes 5 minutes to complete
outputs.Add(await context.CallActivityAsync<string>("E1_SayHello", "Seattle")); //takes 5 minutes complete
outputs.Add(await context.CallActivityAsync<string>("E1_SayHello_DirectInput", "London")); //takes 5 minutes complete
// should return ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]
return outputs;
}
now we have three functions
let's say each one needs 5 minutes to complete (on the default azure consumptionl plan) ,people say each function has it has its own timeout so we should have a total of around 15 minutes in order to complete (5+5+5) for all ,however the top level function E1 has only a timeout of 5 minutes.Will it timeout before complete because the total of all sub-functions exceeds its limit of 5?
if E1 orchestrator timedout then does the activities or subfunctions stop if the orchestrator itself timedout?
The beauty about durable functions is that it is only active when orchestrating the function. When it reaches await context.CallActivityAsync it will start E1_SayHello but it won't wait for its completion. Instead the durable function will unload and resume once E1_SayHello is completed.
What you are doing is called the Function chaining pattern and this behavior I described above is documented there like this:
Each time the code calls await, the Durable Functions framework checkpoints the progress of the current function instance. If the process or virtual machine recycles midway through the execution, the function instance resumes from the preceding await call.
So no, the durable function won't be active the whole 15 minutes.
I have 1 function app project, with a Timer trigger and a http trigger. (using DI or not, does not seem to matter)
The timertrigger:
public static class Timer
{
[FunctionName("Timer")]
public static void Run([TimerTrigger("0 */5 * * * *", RunOnStartup =true)]TimerInfo myTimer, ILogger log)
{
log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
Thread.Sleep(10000);//this is just for demonstrating the behavior!!
log.LogInformation($"C# Timer trigger function executed done");
}
}
the http trigger:
public static class Function1
{
[FunctionName("test2")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req,
ILogger log)
{
return new OkObjectResult("Done");
}
}
The issue is, that when the timer runs, all requests are blocked. Resulting in "Error: connect ECONNREFUSED 127.0.0.1:7071"
I read here and there (mostly in comments) that multiple triggers are (were) not supported, but that doesn't make sense to me as everything seems prepared for it anyway (i mean: i could not find any conclusive documentation about it, VS doesnt throw ANY resistance to add multiple, and even the debug interface and azure interface nicely sums up all the functions it could find...)
Is this behaviour expected, or is there some setting i can change to make it run in parallel or so??
This behavior is expected, since you're completely halting the Azure Functions process by running a Thread.Sleep(). As long as your Function App hasn't scaled to multiple instances, there's only one process running your Functions. This is where working asynchronously comes in.
If you change your timer triggered function to this, it should work fine.
[FunctionName("Timer")]
public static async Task Run([TimerTrigger("0 */5 * * * *", RunOnStartup=true)] TimerInfo myTimer, ILogger log)
{
log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
await Task.Delay(10000);
log.LogInformation($"C# Timer trigger function executed done");
}
Next to this, please be advised:
It is not recommended to have your timer triggered Function RunOnStartup set to true
It is not advised to explicitly build in delays in Functions
Multiple triggers for one Function are not supported, having multiple Functions each with their own trigger within one Function App is supported.
A Function App is a collection of one or more Functions
Triggers are what cause a function to run. A trigger defines how a function is invoked and a function must have exactly one trigger. Triggers have associated data, which is often provided as the payload of the function.
Source: Azure Functions triggers and bindings concepts
EDIT:
Tested it, and this also occurs when the Function is async, and also when it's running on Azure. This means the initial run of the Function with runAtStartup=true always blocks any other incoming requests, and with runOnStartup enabled the trigger is invoked whenever your function app is scaled.
This turns out to be because of RunOnStartup. Any scheduled runs after the initial one do not have this issue.
However, if RunOnStartup = false, the trigger still might be called on startup and then it does have the issue.
As stated in the comments, async does not change anything (I actually use async and DI in the real scenario, but simplified the example here.)
Why the "runtimeStatus" is set to "Completed" only after 52 seconds not 30 as I set in context.CreateTimer() function when checking it with statusQueryGetUri http request?
The documentation that I used
My Code
[FunctionName("H")]
public static async Task<HttpResponseMessage> Start([HttpTrigger(AuthorizationLevel.Anonymous, "get","post",Route = "route/{route}")] HttpRequestMessage req, [DurableClient] IDurableOrchestrationClient client, string route)
{
string id = await client.StartNewAsync("Or1");
return client.CreateCheckStatusResponse(req, id);
}
[FunctionName("Or1")]
public static async Task<string> Or1([OrchestrationTrigger] IDurableOrchestrationContext context, ILogger logger)
{
using (CancellationTokenSource cts = new CancellationTokenSource())
{
DateTime endTime = context.CurrentUtcDateTime.AddSeconds(30);
logger.LogInformation($"*********time now {context.CurrentUtcDateTime}");
logger.LogInformation($"*********end Time {endTime}");
await context.CreateTimer(endTime, cts.Token);
logger.LogInformation($"*********end Time finish {context.CurrentUtcDateTime}");
return "timer finished";
}
}
[FunctionName("Activity1")]
public static async Task A1([ActivityTrigger] IDurableActivityContext context)
{
//Do something
}
The Log
Functions:
H: [GET,POST] http://localhost:7071/api/route/{route}
Activity1: activityTrigger
Or1: orchestrationTrigger
For detailed output, run func with --verbose flag.
[2021-01-13T16:17:06.841Z] Host lock lease acquired by instance ID '000000000000000000000000EB8F9C93'.
[2021-01-13T16:17:24.767Z] Executing 'H' (Reason='This function was programmatically called via the host APIs.', Id=0aeee0e1-6148-4c21-9aa9-d17a43bce8d1)
[2021-01-13T16:17:24.925Z] Executed 'H' (Succeeded, Id=0aeee0e1-6148-4c21-9aa9-d17a43bce8d1, Duration=164ms)
[2021-01-13T16:17:24.995Z] Executing 'Or1' (Reason='(null)', Id=6aa97b04-d526-41b1-9532-afb21c088b18)
[2021-01-13T16:17:25.006Z] *********time now 1/13/2021 4:17:24 PM
[2021-01-13T16:17:25.007Z] *********endTime 1/13/2021 4:17:54 PM
[2021-01-13T16:17:25.017Z] Executed 'Or1' (Succeeded, Id=6aa97b04-d526-41b1-9532-afb21c088b18, Duration=23ms)
[2021-01-13T16:18:16.476Z] Executing 'Or1' (Reason='(null)', Id=9749d719-5789-419a-908f-6523cf497cca)
[2021-01-13T16:18:16.477Z] *********time now 1/13/2021 4:17:24 PM
[2021-01-13T16:18:16.478Z] *********endTime 1/13/2021 4:17:54 PM
[2021-01-13T16:18:16.481Z] *********endTime finish 1/13/2021 4:18:16 PM
[2021-01-13T16:18:16.485Z] Executed 'Or1' (Succeeded, Id=9749d719-5789-419a-908f-6523cf497cca, Duration=9ms)
The azure Orchestrater works on Queue polling which is implemented as a random exponential back-off algorithm to reduce the effect of idle-queue polling on storage transaction costs. When a message is found, the runtime immediately checks for another message; when no message is found, it waits for a period of time before trying again. After subsequent failed attempts to get a queue message, the wait time continues to increase until it reaches the maximum wait time, which defaults to 30 seconds.
If see your logs, you can notice that Orchestrater has triggered Timer at 16:17:24 and when it was finished at 16:17:54 a message was added in the storage queue. As mentioned above due to queue polling it seems that the message was picked at 16:18:16 to resume the orchestration execution.
I believe if you trigger the durable function multiple times then you will notice the total time to finish the orchestration would be different for each instance.
You can read about Azure function orchestration queue polling at here.
You can also check the history table to understand when a message was queued and when picked. Read about it at here.
To show how queuing works you can stop the function as soon timer is triggered. Following is the output in my local environment emulator queue which displays that a message is queued when timer is triggered
Now when Orchestrator function resumes again then it polls the message from queue and pick it to process further.
Note - in my local environment, I tried couple of time with your code as I noticed all instances finishes in ~30 secs.
I have an azure service bus queue set up. Within the host.json file I have maxConcurrentCalls set to 1 however when i debug the app locally there are 8-10 worker threads running and its making the app impossible to debug. How do i have only 1 thread running when i debug my app locally?
Initially my host.json was empty and i found the below code that was supposed to have in order to have only 1 thread execution see https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus#host-json
{
"version": "2.0",
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": false,
"maxConcurrentCalls": 1,
"maxAutoRenewDuration": "00:55:00"
}
}
}
}
I added the host.json settings to my local.json settings to no avail, i still get 10 worker threads starting up on launch.
I also attempted to make the function call a singleton see
Make Azure Function run sequential on ServiceBusTrigger - maxConcurrentCalls not working
however, this just locked up the function completely and didnt solve any issues at all.
[FunctionName("ReadFromXCagoIssueQueueAndRetrieveFiles")]
public static void Run([ServiceBusTrigger(QueueName, AccessRights.Manage, Connection = "SBConnection")]BrokeredMessage myQueueItem, TextWriter log)
{
log.WriteLine($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
var propIssueId = myQueueItem.Properties["IssueId"].ToString();
var issueId = int.Parse(propIssueId);
log.WriteLine($"begin grab files for issue: {issueId}");
retrieveIssueEPub(issueId);
}
The execution of retrieveIssueEPub is performed over several worker threads which means that the message ends up in deadletter queue after 1 run of the function due to it splitting the same message across all the threads and not the 1 thread that i expect due to the max concurrent calls. am i missing something? Is there a difference between concurrently pulling out 1 message at a time and executing the function over several threads?
A colleague of mine has found the error.
the application is only firing one thread at a time. However on azure itself the lock duration was set to 30 seconds.
This means that the app would start new a new thread after 30 seconds etc because the message becomes unlocked.
Extending the lock has solved the issue.
I have a continuous WebJob with a function using the TimerTrigger to run a process every 30 seconds. A particular call in the function occasionally and seemingly randomly hangs, causing the webjob to wait indefinitely. Current solution is notice the service has stopped, then log into the Azure Dashboard and abort it manually.
Note that I know the correct course of action is to identify the root cause and fix it. Trust me, we're working on that. In the mean time, I want to treat the symptom, and need help doing so.
I'm attempting to have the WebJob detect if status using the Timeout decorator as described in this post on the Azure WebJobs SDK: https://github.com/Azure/azure-webjobs-sdk/issues/590. Implementing the suggestion, I'm able to see that when the problematic call hangs, the Timeout is detected, but the WebJob still doesn't die. What I doing wrong here that won't kill the function to allow subsequent invocations?
Program.cs
static void Main()
{
var config = new JobHostConfiguration();
config.UseTimers();
config.FunctionTimeout = new TimeSpan(0, 15, 0);
var host = new JobHost(config);
Functions.Initialize();
host.RunAndBlock();
}
Functions.cs
[Singleton]
[Timeout("00:05:00")]
public async static Task PeriodicProcess([TimerTrigger("00:00:30", RunOnStartup = true)] TimerInfo timer, CancellationToken cancelToken, TextWriter log)
{
log.WriteLine("-- Processing Begin --");
List<Emails> cases = GetEmailsAndWhatNot();
foreach (Email e in Emails)
{
try
{
ProblematicFunction_SendEmail(e, log);
}
catch(Exception e)
{
// do stuff
}
}
log.WriteLine("-- Processing End -- ");
}
public static void ProblematicFunction_SendEmail(Email e, TextWriter log)
{
// send email
}
WebJob Output During Issues
-- Processing Begin --
Timeout value of 00:05:00 exceeded by function 'Functions.PeriodicProcess' (Id: '0f7438bd-baad-451f-95a6-9461f35bfb2d'). Initiating cancellation.
Despite the webjob initiating cancellation, the function doesn't die. Do I need to monitor the CancellationToken? How far down do I need to propogate asynchronous calling? What am I missing here that will actually abort the process?
As TimerTrigger states about TimerTrigger:
Singleton Locks
TimerTrigger uses the Singleton feature of the WebJobs SDK to ensure that only a single instance of your triggered function is running at any given time.
Scheduling
If your function execution takes longer than the timer interval, another execution won't be triggered until after the current invocation completes. The next execution is scheduled after the current execution completes.
Here is my test for this scenario, you could refer to it:
Use CancellationToken.None and never propogate the cancellation token
Note: The function PeriodicProcess would be time out after 30 s, but the Time-consuming job is still running, and after the long-running job has done, the Processing End log would be printed.
Propogate the cancellation token
Note: If we propogate the cancellation token, the Time-consuming job would be cancelled immediately.
Code snippet
[Timeout("00:00:30")]
[Singleton]
public async static Task PeriodicProcess([TimerTrigger("00:00:10", RunOnStartup = true)] TimerInfo timer, CancellationToken cancelToken, TextWriter log)
{
log.WriteLine($"-- [{DateTime.Now.ToString()}] Processing Begin --");
try
{
await longRunningJob(log, cancelToken);
}
catch (Exception e)
{
// do stuff
}
log.WriteLine($"-- [{DateTime.Now.ToString()}] Processing End -- ");
}
private async static Task longRunningJob(TextWriter log, CancellationToken cancelToken)
{
log.WriteLine($"-- [{DateTime.Now.ToString()}] Begin Time-consuming jobs --");
await Task.Delay(TimeSpan.FromMinutes(1), cancelToken);
log.WriteLine($"-- [{DateTime.Now.ToString()}] Complete Time-consuming jobs --");
}