Develop Reference

Wait for Data Factory pipeline to finish in Durable Function

I have a number of Azure Data Factory pipelines that I want to execute. The pipeline hierarchy is essentially a DAG, so I'm able to execute them in order using a topological sort.
I was thinking about doing this with Durable Functions as they seem to fit the use case. However, I'm at a loss for how I go about letting the orchestrator know, that a specific pipeline has finished. I would prefer not to make modifications to the pipelines, like calling a accept/reject function at the end.
I know I can do something like the below code, but does that go against best practices regarding Durable Functions?
var pipelineRunId = await context.CallActivityAsync<string>("StartPipeline", pipelineId);
var hasFinished = false;
while(!hasFinished)
{
var fireAt = context.CurrentUtcDateTime.AddSeconds(30);
await context.CreateTimer(fireAt, CancellationToken.None);
hasFinished = await context.CallActivityAsync<bool>("CheckPipelineStatus", pipelineRunId);
}

The major difference of Durable Functions from Azure Functions is that they run asynchronously. When you trigger a Durable Function, it creates a background process and which gives you a few URLs that you can interact with that process; including one to query its status.
You can trigger a Durable Function through an HTTP endpoint, wait for it to finish and then get the result. Here’s how it looks like:
And here’s how it looks inside the Until activity:
Here’s the process:
First, we trigger our Durable Function through an HTTP trigger using Azure Function activity.
Then with the Until activity, we check status of that function.
The Wait activity waits around 30 seconds (or different, up to you) to let function to be executed.
The Web activity makes a request to the statusQueryUrl that Azure Function activity returns, by calling **#activity('StartUntar').output.statusQueryGetUri**
Until activity checks the result of CheckStatus Web activity with expression **#not(or(equals(activity('CheckStatus').output.runtimeStatus, 'Pending'), equals(activity('CheckStatus').output.runtimeStatus, 'Running')))**
It repeats until the function is finished or failed, or until it times out (set on the Timeout property)

You can both start a Data Factory pipeline and monitor for completion within a durable function.
To use the Azure credentials passed to the data factory client you will need to register your application in Azure Active Directory, generate a secret for it and then add your application registration to the Data Factory Contributor role, which you do under your Azure Subscriptions Access Control (IAM) blade.
https://learn.microsoft.com/en-us/powerapps/developer/data-platform/walkthrough-register-app-azure-active-directory
To start a pipeline use the IPipelinesOperations interface. I have used the CreateRunWithHttpMessagesAsync method with a lot of success.
https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.management.datafactory.ipipelinesoperations.createrunwithhttpmessagesasync?view=azure-dotnet
Use DataFactoryManagementClient to get the pipelines like this:
//Set up credentials for app registration
ClientCredential cc = new ClientCredential(<Your-App-ClientId>, <Your-App-Secret>);
AuthenticationResult result = context.AcquireTokenAsync("https://management.azure.com/", cc).Result;
ServiceClientCredentials credentials = new TokenCredentials(result.AccessToken);
//Create data factory client and start the pipeline
var dataFactoryMgmtClient = new DataFactoryManagementClient(credentials)
{
SubscriptionId = "<your-Azure-SubscriptionId>"
};
Dictionary<string, object> parameters = new Dictionary<string, object>()
AzureOperationResponse<CreateRunResponse> runResponse = await
dataFactoryMgmtClient.Pipelines.CreateRunWithHttpMessagesAsync("<azure-resource-group>", "<factory-name>", "<pipeline-name>", parameters: parameters);
Use the parameters dictionary to pass any Data Factory parameters your data factory pipeline expects.
The AzureOperationResponse<CreateRunResponse> runResponse contains a run id which you'll find in runResponse.Body.RunId. You need this to monitor the execution of the pipeline.
Start the pipeline in an ActivityTrigger function. I suggest using CallActivityWithRetryAsync to call it with a retry.
To monitor the execution you will need to set up another orchestration. Something like this:
[FunctionName("AdfMonitor")]
public async Task<string> RunMonitorOrchestrator([OrchestrationTrigger] IDurableOrchestrationContext monitorContext)
{
var _replayLog = monitorContext.CreateReplaySafeLogger(_log);
string runId = monitorContext.GetInput<string>(); //pass the run id to the orchestration
string status = "Queued";
//Timeout after waiting one hour
DateTime endTime = monitorContext.CurrentUtcDateTime.AddHours(1);
while (monitorContext.CurrentUtcDateTime < endTime && (status == "InProgress" || status == "Queued"))
{
status = await monitorContext.CallActivityAsync<string>("CheckPipelineStatus", runId);
_replayLog.LogInformation($"Adf Pipeline status is {status}.");
if (status == "InProgress" || status == "Queued")
{
var nextCheckpoint = monitorContext.CurrentUtcDateTime.AddSeconds(5);
await monitorContext.CreateTimer(nextCheckpoint, CancellationToken.None);
}
}
return status;
}
Launch the monitor as a sub-orchestration immediately after calling the activity that starts your pipeline.
//Monitor pipeline progress
string pipelineStatus = await monitorContext.CallSubOrchestratorAsync<string>("AdfMonitor", runId);
Use the CreateTimer function to set the next time you want to check your pipeline status. In this example it executes and checks the pipeline status every 5 seconds, which is maybe a bit too frequent.
Notice also the use of the time out in the while loop, so if something goes wrong we just give up after an hour.
Here's the link to MS docs on monitors in durable functions:
https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-monitor?tabs=csharp
Use another ActivityTrigger function to check the pipeline status
[FunctionName("CheckPipelineStatus")]
public async Task<string> CheckRunStatus([ActivityTrigger] string runId)
{
return await dataFactoryMgmtClient.GetPipelineStatus(runId);
}
Hope that helps, I use this technique a lot where the durable function loads many small batches of data into Datalake and then triggers ADF pipelines that run in parallel handling merge, copy & transform of the data.

Send notification on specific date and time

In our App, We are storing questions with Question's startdate, enddate and resultdate. We need to send notification to app (iPhone and Andorid) once startdate of question is arrives.
Can anybody let me know how can we achieve this?
We don't want to use pull method. like in particular time interval it will check for question startdate and send notification.
I have a URL to send Notification for question. I need to call this URL when question's startdate is arrived.
Thanks.

Take a look at Quartz :
Quartz.NET is a full-featured, open source job scheduling system that can be used from smallest apps to large scale enterprise systems
Quartz Enterprise Scheduler .NET
You can create a new Quarts Job, lets call it QuestionSenderJob. Then your application can schedule a task in Quartz scheduler, jobs can have many instances of same Job with custom data - in your case QuestionId.
Additionally it supports storing Job scheduling in your SQL database (there are DDL Scripts included) so you can create some relations if you need for UI for example.
You can find table-creation SQL scripts in the "database/dbtables" directory of the Quartz.NET distribution
Lesson 9: JobStores
This way you leave firing in right moment to Quartz engine.
When you will go through Quartz .NET basics, see this code snippet I made a for your case to schedule job. Perhaps some modifications will be necessary thought.
IDictionary<string, object> jobData = new Dictionary<string, object> { { "QuestionId", questionId } };
var questionDate = new DateTime(2016, 09, 01);
var questionTriggerName = string.Format("Question{0}_Trigger", questionId);
var questionTrigger = TriggerBuilder.Create()
.WithIdentity(questionTriggerName, "QuestionSendGroup")
.StartAt(questionDate)
.UsingJobData(new Quartz.JobDataMap(jobData))
.Build();
scheduler
.ScheduleJob(questionSenderJob, questionTrigger);
Then in Job you will get your questionId through JobExecutionContext.
public class QuestionSenderJob: IJob
{
public void Execute(JobExecutionContext context)
{
JobDataMap dataMap = context.JobDetail.JobDataMap;
// Extract question Id and send message
}
}

What about using the Task Scheduler Managed Wrapper?
You do not want to use pooling, but if you write your own class that will encapsulate Timer (e.g. System.Thread.Timer) and check for the time each second, that will not take much resources. Depending on how exact you need it, you could check also less often, e.g. each minute. Maybe you should reconsider it.

If you use any third party service to manage your push notification such as Azure Notification Hub, Parse.com, ... they offer an integrated way to schedule push notifications. Either by passing in a send date or let them run a job periodically. I'm a user of the Azure service and it works very well.

The best implementation i can advice right now is for you to send the notification from a server.
All you just need is a good scheduler that can dispatch operation.
For me, my server is powered by Javascript (NodeJS) so i use "node-schedule". All i just do is
var schedule = require('node-schedule');
//Reporting rule at minute 1 every hour
var rule = new schedule.RecurrenceRule();
rule.minute = 1;
schedule.scheduleJob(rule, function () {
console.log(new Date().toTimeString() + ' Testing Scheduler! Executing Every other minute');
//sendPush()
});

How do I create a process that checks date time and performs action when elapsed time passed?

in my dll I'm serializing an object that I need to expire, and regenerate when x amount of days have passed.
How can I do this in a way that doesn't require the calling application to restart every day (in order to initiate the check for date time in my dll)?
using .net 3.5
private void updatePersistableItems()
{
if (!File.Exists(Items_FILENAME) && PersistableItems != null) //create new
{
_serializer.SerializeObject<PersistableObject>(Items_FILENAME, PersistableItems);
}
else //check if expired and replace, or update if not expired
{
PersistableObject ItemsFromStorage = new PersistableObject();
ItemsFromStorage = _serializer.DeSerializeObject<PersistableObject>(Items_FILENAME);
TimeSpan ts = DateTime.Now - ItemsFromStorage.DateItemsInitialized;
if (ts.TotalDays < this.DaysToPersistItems) //use stored Items
Items = ItemsFromStorage.Items;
}
}

Use a Timer to periodically call your function that does the checking: http://msdn.microsoft.com/en-us/library/system.timers.timer(v=vs.110).aspx

You can start new Thread with while loop. Once executed on start of the application the thread can check the time amount left till content expiry, and go to sleep for the rest of the time. Once waked up after sleep, refresh content, calculate time amount left to sleep .... : )

Sounds like a job for a the task scheduler which can be controlled via c# this wrapper. Using this you can schedule the update to get run even if your application is not running all the time.

Setting Reminders in Windows Phone 7 & 8 to Go Off Every Other Day

In my application I'm using the reminders service to supply reminders to the user, which prompts them to do something.
I am using the following code to do this:
if (date > DateTime.Now)
{
Reminder r = new Reminder(fileTitle);
r.Title = fileTitle;
r.Content = fileContent;
r.BeginTime = date;
ScheduledActionService.Add(r);
}
However this only goes off once. I have tried setting the ExpirationTime to a certain value, but this repeats the reminder every day.
Does anyone know how to set a reminder to fire every other day?
(In addition it would be good to know how to set reminders for certain days of the week, but the every other day part is the main issue at the moment.)

For your case I would suggest storing the time the alarm should go off. You would store this information in either the application settings or in a file. When the user first asks for the reminder to be scheduled, continue what you are doing, and then also save the time for the alarm. You may want to also ask the user when they want the alarm to stop and save that as well.
To ensure that the alarm goes off every other day, you will need to add a background agent to your application. In the agent is an OnInvoke method. In this method you will check to see if the alarm is scheduled. If it is then you have nothing to do. If it is not, then schedule it for the following day. Agents fire about every 30 minutes, so 99% of the time your agent fires, the alarm/reminder will already be scheduled.
Here is the code to place in your OnInvoke method
string fileTitle = "Foo";
string fileContent = "Bar";
var action = ScheduledActionService.Find(fileTitle);
if (action == null)
{
// shouldn't be null if it was already added from the app itself.
// should get the date the user actually wants the alarm to go off.
DateTime date = DateTime.Now.AddSeconds(30);
action = new Reminder(fileTitle) { Title = fileTitle, Content = fileContent, BeginTime = date };
}
else if (action.IsScheduled == false)
{
ScheduledActionService.Remove(fileTitle);
// most likely fired today, add two days to the begin time.
// best to also add some logic if BeginTime.Date == Today
action.BeginTime = action.BeginTime.AddDays(2);
}
ScheduledActionService.Add(action);

You need to set the RecurrenceType to a RecurrenceInterval value. Unfortunately for you, there is nothing currently available for a custom schedule (i.e. every other day).
Another duh! by Microsoft here really.

How does one configure Quartz.net to run a job when another job hasn't run for a period

Assume I have two Quartz.net jobs that
downloads a CSV file with a delta of changes for a period (e.g. 24h) and then imports the data (called IncrementalImportJob)
downloads a CSV file with a all the records and then imports the data (called FullImportJob)
The requirement is that IncrementalImportJob at a minimum once for the period (e.g. 24h). If that window is missed, or the job didn't complete successfully, then FullImportJob should run instead. The reason is that changes for that (missed) day would not be imported. This condition is rather exceptional.
The FullImportJob requires resources (time, CPU, database, memory) to import all the data, which may impact other systems. Further, the delta of changes are often minimal or non-existent. So the goal is to favour running the IncrementalImportJob when possible.
How does one configure quartz.net to run FullImportJob if IncrementalImportJob hasn't completed successfully in a specific time period (say 24h)?
Searching the web for "quartz.net recovery" and "quartz.net misfire" doesn't reveal whether its supported or whether its even possible.

There is native misfire handling in quartz.net, however it only goes as far as specifying whether the job should fire immediately again, or after a period of time or a number of times after misfiring.
I think one option is to handle this internally from IncrementalImportJob.
try
{
//download data
//import data
}
catch (Exception e) //something went wrong
{
//log the error
UpdateFullImportJobTrigger(sched);
}
//Reschedule FullImportJob to run at a time of your choosing.
public void UpdateFullImportJobTrigger(IScheduler sched)
{
Trigger oldTrigger = sched.getTrigger(triggerKey("oldTrigger", "group1");
TriggerBuilder tb = oldTrigger.getTriggerBuilder();
//if you want it to run based on a schedule use this:
Trigger newTrigger = tb.withSchedule(simpleSchedule()
.withIntervalInSeconds(10)
.withRepeatCount(10)
.build();
sched.rescheduleJob(oldTrigger.getKey(), newTrigger);
//or use simple trigger if you want it to run immediately and only once so that
//it runs again on schedule the next time.
}
This is one way of doing it. Another would be abstracting this logic to a maintenance job that checks the logs every so often and if it finds a failure message from IncrementalImportJob, it fires FullImportJob. However, this depends to some extent on your logging system (most people use NLog or log4net).
If on the other hand, your concern is that the job never ran in the first place because, for instance, the app/database/server was down, you could schedule FullImportJob to fire a few hours later and check if IncrementalImportJob has fired as follows:
//this is done from FullImportJob
//how you retrieve triggerKey will depend on whether
//you are using RAMJobStore or ADO.NET JobStore
public void Execute(IJobExecutionContext context)
{
ITrigger incImportJobTrigger = context.Scheduler.GetTrigger(triggerKey);
//if the job has been rescheduled with a new time quartz will set this to null
if (!incImportJobTrigger.GetPreviousFireTimeUtc().HasValue) return;
DateTimeOffset utcTime = incImportJobTrigger.GetPreviousFireTimeUtc().Value;
DateTime previousTireTime = utcTime.LocalDateTime;
if (previousTireTime.Day == DateTime.Now.Day) return;
//IncrementalImportJob has not ran today, let's run FullImportJob
}
Hope this helps.