Develop Reference

Singleton Lock for Azure Webjob's TimerTrigger accross multiple regions

I have a timer triggered webjob deployed across multiple regions and it is getting triggering concurrently from all regions on given scheduled time, how do I make sure that only one instance of the job runs at a time.
I tried applying Singleton attribute and "is_singleton": true but still it is triggering from all regions.
Is there any other way to achieve this. This link says that Singleton attribute no longer works for this purpose and also I don't see any lock file created in the azure blob storage. If its true how do we implement this to make sure only one region is triggered from multiple regions. Or If there is any other inbuilt way of achieving this with webjob sdk
that would be really helpful to me
My program.cs
var builder = new HostBuilder();
builder
.ConfigureWebJobs((context, b) =>
{
b.AddAzureStorageCoreServices();
});
var host = builder.Build();
using (host)
{
var jobHost = host.Services.GetService(typeof(IJobHost)) as JobHost;
await host.StartAsync().ConfigureAwait(false);
await jobHost.CallAsync("Run").ConfigureAwait(false);
await host.StopAsync().ConfigureAwait(false);
}
Function.cs
[Singleton]
[NoAutomaticTrigger]
public async Task Run()
{
}
settings.job
{
"schedule": "0 */5 * * * *",
"is_singleton": true
}
nuget package
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="4.0.1" />

So it sounds like you want each region to run, but just not at the same time.
One simple solution would be to stagger them so they don't all run at the same time but you give each one a slightly different cron time, such as 10m apart from each other.
But supposing you couldn't predict how long they will run or you essentially want them to run much faster then the minimum 10m apart, and supposing they could all access a database, such a Azure Cosmos Mongo, then you could add a table and some simple logic which would essentially model a "locking" pattern.
In mongodb you can use the findOneAndUpdate function to do an atomic update on a well known document which will allow only one process to "lock" a document at a time.
The database in this case contains a collection (aka table) with a single document (aka row) that looks like this:
interface ILock {
state: 'UNLOCKED' | 'LOCKED';
lockedBy: null | string;
lockedAt: null | Date;
}
psuedo code
while (true)
{
// todo: add some kind of timeout here so it doesn't run forever.
// `findOneAndUpdate` is atomic, if multiple processes
// attempt to do this modification simultaneously
// only one will succeed, the others will get an `undefined`
// result to indicate the document was not found.
var lock = await this.db.locks.findOneAndUpdate(
{ state: 'UNLOCKED' },
{
$set: {
state: 'LOCKED',
lockedBy: this.jobId,
lockedAt: new Date()
}
}
)
if (lock) {
try {
// you have the lock, do your thing...
await DoWork();
// you are done, exit the loop.
return;
} finally {
// don't forget to unlock!
await this.db.locks.findOneAndUpdate(
{ state: 'LOCKED' },
{
$set: {
state: 'UNLOCKED',
}
}
)
}
} else {
// you are not the one neo, take the blue pill...
await sleep(3000)
}
}

Reentrance method and partial synchronized calls

I do have a singleton component that manages some information blocks. An information block is a calculated information identified by some characteristics (concrete an Id and a time period). These calculations may take some seconds. All information blocks are stored in a collection.
Some other consumers are using these information blocks. The calculation should start when the first request for this Id and time period comes. I had following flow in mind:
The first consumer requests the data identified by Id and time period.
The component checks if the information block already exists
If not: Create the information block, put it into the collection and start the calculation in a background task. If yes: Take it from the collection
After that the flow goes to the information block:
When the calculation is already finished (by a former call), a callback from the consumer is called with the result of the calculation.
When the calculation is still in process, the callback is called when the calculation is finished.
So long, so good.
The critical section comes when the second (or any other subsequent) call is coming and the calculation is still running. The idea is that the calculation method holds each consumers callback and then when the calculation is finished all consumers callbacks are called.
public class SingletonInformationService
{
private readonly Collection<InformationBlock> blocks = new();
private object syncObject = new();
public void GetInformationBlock(Guid id, TimePersiod timePeriod,
Action<InformationBlock> callOnFinish)
{
InformationBlock block = null;
lock(syncObject)
{
// check out if the block already exists
block = blocks.SingleOrDefault(b => b.Id ...);
if (block == null)
{
block = new InformationBlock(...);
blocks.Add(block);
}
}
block?.BeginCalculation(callOnFinish);
return true;
}
}
public class InformationBlock
{
private Task calculationTask = null;
private CalculationState isCalculating isCalculating = CalculationState.Unknown;
private List<Action<InformationBlock> waitingRoom = new();
internal void BeginCalculation(Action<InformationBlock> callOnFinish)
{
if (isCalculating == CalculationState.Finished)
{
callOnFinish(this);
return;
}
else if (isCalculating == CalculationState.IsRunning)
{
waitingRoom.Add(callOnFinish);
return;
}
// add the first call to the waitingRoom
waitingRoom.Add(callOnFinish);
isCalculating = CalculationState.IsRunning;
calculationTask = Task.Run(() => { // run the calculation})
.ContinueWith(taskResult =>
{
//.. apply the calculation result to local properties
this.Property1 = taskResult.Result.Property1;
// set the state to mark this instance as complete
isCalculating = CalculationState.Finished;
// inform all calls about the result
waitingRoom.ForEach(c => c(this));
waitingRoom.Clear();
}, TaskScheduler.FromCurrentSynchronizationContext());
}
}
Is that approach a good idea? Do you see any failures or possible deadlocks? The method BeginCalculation might be called more than once while the calculation is running. Should I await for the calculationTask?

To have deadlocks, you'll need some cycles: object A depends of object B, that depends on object A again (image below). As I see, that's not your case, since the InformationBlock class doesn't access the service, but is only called by it.
The lock block is also very small, so probably it'll not put you in troubles.
You could look for the Thread-Safe Collection from C# standard libs. This could simplify your code.
I suggest you to use a ConcurrentDictionary, because it's fastest then iterate over the collection every request.

Block Controller Method while already running

I have a controller which returns a large json object. If this object does not exist, it will generate and return it afterwards. The generation takes about 5 seconds, and if the client sent the request multiple times, the object gets generated with x-times the children. So my question is: Is there a way to block the second request, until the first one finished, independent who sent the request?
Normally I would do it with a Singleton, but because I am having scoped services, singleton does not work here

Warning: this is very oppinionated and maybe not suitable for Stack Overflow, but here it is anyway
Although I'll provide no code... when things take a while to generate, you don't usually spend that time directly in controller code, but do something like "start a background task to generate the result, and provide a "task id", which can be queried on another different call).
So, my preferred course of action for this would be having two different controller actions:
Generate, which creates the background job, assigns it some id, and returns the id
GetResult, to which you pass the task id, and returns either different error codes for "job id doesn't exist", "job id isn't finished", or a 200 with the result.
This way, your clients will need to call both, however, in Generate, you can check if the job is already being created and return an existing job id.
This of course moves the need to "retry and check" to your client: in exchange, you don't leave the connection to the server opened during those 5 seconds (which could potentially be multiplied by a number of clients) and return fast.
Otherwise, if you don't care about having your clients wait for a response during those 5 seconds, you could do a simple:
if(resultDoesntExist) {
resultDoesntExist = false; // You can use locks for the boolean setters or Interlocked instead of just setting a member
resultIsBeingGenerated = true;
generateResult(); // <-- this is what takes 5 seconds
resultIsBeingGenerated = false;
}
while(resultIsBeingGenerated) { await Task.Delay(10); } // <-- other clients will wait here
var result = getResult(); // <-- this should be fast once the result is already created
return result;
note: those booleans and the actual loop could be on the controller, or on the service, or wherever you see fit: just be wary of making them thread-safe in however method you see appropriate
So you basically make other clients wait till the first one generates the result, with "almost" no CPU load on the server... however with a connection open and a thread from the threadpool used, so I just DO NOT recommend this :-)
PS: #Leaky solution above is also good, but it also shifts the responsability to retry to the client, and if you are going to do that, I'd probably go directly with a "background job id", instead of having the first (the one that generates the result) one take 5 seconds. IMO, if it can be avoided, no API action should ever take 5 seconds to return :-)

Do you have an example for Interlocked.CompareExchange?
Sure. I'm definitely not the most knowledgeable person when it comes to multi-threading stuff, but this is quite simple (as you might know, Interlocked has no support for bool, so it's customary to represent it with an integral type):
public class QueryStatus
{
private static int _flag;
// Returns false if the query has already started.
public bool TrySetStarted()
=> Interlocked.CompareExchange(ref _flag, 1, 0) == 0;
public void SetFinished()
=> Interlocked.Exchange(ref _flag, 0);
}
I think it's the safest if you use it like this, with a 'Try' method, which tries to set the value and tells you if it was already set, in an atomic way.
Besides simply adding this (I mean just the field and the methods) to your existing component, you can also use it as a separate component, injected from the IOC container as scoped. Or even injected as a singleton, and then you don't have to use a static field.
Storing state like this should be good for as long as the application is running, but if the hosted application is recycled due to inactivity, it's obviously lost. Though, that won't happen while a request is still processing, and definitely won't happen in 5 seconds.
(And if you wanted to synchronize between app service instances, you could 'quickly' save a flag to the database, in a transaction with proper isolation level set. Or use e.g. Azure Redis Cache.)
Example solution
As Kit noted, rightly so, I didn't provide a full solution above.
So, a crude implementation could go like this:
public class SomeQueryService : ISomeQueryService
{
private static int _hasStartedFlag;
private static bool TrySetStarted()
=> Interlocked.CompareExchange(ref _hasStartedFlag, 1, 0) == 0;
private static void SetFinished()
=> Interlocked.Exchange(ref _hasStartedFlag, 0);
public async Task<(bool couldExecute, object result)> TryExecute()
{
if (!TrySetStarted())
return (couldExecute: false, result: null);
// Safely execute long query.
SetFinished();
return (couldExecute: true, result: result);
}
}
// In the controller, obviously
[HttpGet()]
public async Task<IActionResult> DoLongQuery([FromServices] ISomeQueryService someQueryService)
{
var (couldExecute, result) = await someQueryService.TryExecute();
if (!couldExecute)
{
return new ObjectResult(new ProblemDetails
{
Status = StatusCodes.Status503ServiceUnavailable,
Title = "Another request has already started. Try again later.",
Type = "https://tools.ietf.org/html/rfc7231#section-6.6.4"
})
{ StatusCode = StatusCodes.Status503ServiceUnavailable };
}
return Ok(result);
}
Of course possibly you'd want to extract the 'blocking' logic from the controller action into somewhere else, for example an action filter. In that case the flag should also go into a separate component that could be shared between the query service and the filter.
General use action filter
I felt bad about my inelegant solution above, and I realized that this problem can be generalized into basically a connection number limiter on an endpoint.
I wrote this small action filter that can be applied to any endpoint (multiple endpoints), and it accepts the number of allowed connections:
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class ConcurrencyLimiterAttribute : ActionFilterAttribute
{
private readonly int _allowedConnections;
private static readonly ConcurrentDictionary<string, int> _connections = new ConcurrentDictionary<string, int>();
public ConcurrencyLimiterAttribute(int allowedConnections = 1)
=> _allowedConnections = allowedConnections;
public override async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
{
var key = context.HttpContext.Request.Path;
if (_connections.AddOrUpdate(key, 1, (k, v) => ++v) > _allowedConnections)
{
Close(withError: true);
return;
}
try
{
await next();
}
finally
{
Close();
}
void Close(bool withError = false)
{
if (withError)
{
context.Result = new ObjectResult(new ProblemDetails
{
Status = StatusCodes.Status503ServiceUnavailable,
Title = $"Maximum {_allowedConnections} simultaneous connections are allowed. Try again later.",
Type = "https://tools.ietf.org/html/rfc7231#section-6.6.4"
})
{ StatusCode = StatusCodes.Status503ServiceUnavailable };
}
_connections.AddOrUpdate(key, 0, (k, v) => --v);
}
}
}

Quartz Scheduler's job fired but JobListener not picking up event?

I have been struggling to make this work for a couple of days now. I am attempting to use the quartz scheduler to fire an event on my main form, from which a method can be called. Code on my main form is as follows:
void StartSchedule()
{
string AuctionTime = "0 0 0 ? * MON-FRI *";
JobKey jobkey = new JobKey("Auction", "G");
IJobDetail job = JobBuilder.Create<AuctionJob>()
.WithIdentity(jobkey)
.Build();
CronScheduleBuilder csb = CronScheduleBuilder.CronSchedule(new CronExpression(AuctionTime)).InTimeZone(TimeZoneInfo.Local);
ICronTrigger trigger = (ICronTrigger)TriggerBuilder.Create()
.WithIdentity("Auction-Trigger", "G")
.WithSchedule(csb)
.Build();
// Create listener
JobListener auctionListener = new JobListener();
auctionListener.JobExecutedHandler += new EventHandler(SendAuctionOrders);
IMatcher<JobKey> matcher = KeyMatcher<JobKey>.KeyEquals(jobkey);
_scheduler.ListenerManager.AddJobListener(auctionlistener, matcher);
// Add to scheduler
DateTimeOffset ft = _scheduler.ScheduleJob(job, trigger);
LogEvent(job.Key + " scheduled at: " + ft.ToLocalTime());
jobkey = new JobKey("Auction2", "G");
IJobDetail job = JobBuilder.Create<AuctionJob>()
.WithIdentity(jobkey)
.Build();
csb = CronScheduleBuilder.CronSchedule(new CronExpression(AuctionTime)).InTimeZone(TimeZoneInfo.Local);
ICronTrigger trigger = (ICronTrigger)TriggerBuilder.Create()
.WithIdentity("Auction-Trigger", "G")
.WithSchedule(csb)
.Build();
// Create listener
JobListener auction2Listener = new JobListener();
auction2Listener.JobExecutedHandler += new EventHandler(SendAuctionOrders);
IMatcher<JobKey> matcher = KeyMatcher<JobKey>.KeyEquals(jobkey);
_scheduler.ListenerManager.AddJobListener(auction2listener, matcher);
// Add to scheduler
ft = _scheduler.ScheduleJob(job, trigger);
LogEvent(job.Key + " scheduled at: " + ft.ToLocalTime());
_scheduler.Start();
}
void SendAuctionOrders()
{
// Do something
}
AuctionJob class looks like this:
public class AuctionJob : IJob
{
public void Execute(IJobExecutionContext context)
{
// Empty
Console.WriteLine("auction exec");
}
}
While the job listener looks like this:
public class JobListener : IJobListener
{
private static readonly ILog logger = LogManager.GetLogger(typeof(JobListener));
public event EventHandler JobExecutedHandler;
public void JobExecutionVetoed(IJobExecutionContext context)
{
logger.Info("JobExecutionVetoed");
}
public void JobToBeExecuted(IJobExecutionContext context)
{
logger.Info("JobToBeExecuted");
}
public void JobWasExecuted(IJobExecutionContext context, JobExecutionException jobException)
{
// Raise the event on executed
OnJobExecuted(EventArgs.Empty);
logger.Info("JobWasExecuted");
}
public string Name
{
get
{
return "JobListener";
}
}
// Event raiser
protected virtual void OnJobExecuted(EventArgs args)
{
// This code will prevent IllegalThreadContext exceptions
EventHandler jobExecHandler = JobExecutedHandler;
if (jobExecHandler != null)
{
ISynchronizeInvoke target = jobExecHandler.Target as ISynchronizeInvoke;
if (target != null && target.InvokeRequired)
{
target.Invoke(jobExecHandler, new object[] { this, args });
}
else
{
jobExecHandler(this, args);
}
}
}
}
When the job is triggered, the job was executed (console prints "auction exec") but job listener's JobWasExecuted does not pick it up. Could anyone help me with this?
Additional questions: is there another way to call the main form's method? Also, would the job listener class persist through the entire application's (which in theory will not be closed). Thanks.

Problem solved. When implementing IJobListener class, Name property is protected and when adding to ListenerManager, it overwrites the previous entry. Trick is to add a custom string when instantiating JobListener class and adding that to:
public string Name
{
get
{
return instanceName + "JobListener";
}
}

I ran into something similar today, where I had (log4net) debugging lines that were not outputting anything in my JobWasExecuted method, even though similar lines in JobToBeExecuted were working. After beating on it for about two hours, I hacked my way out of it by raising a custom Exception after my job completed successfully. (The "hack" part is having to raise an exception to get JobWasExecuted to react when I really just wanted to grab the final job status.)
I guess if I was doing this for a non-trivial project I'd just have my jobs drop messages onto a queue (and watch that), and it's possible I'm missing something I should be setting on my Jobs on the way out to raise an event, but I still found this part of the Quartz.NET world pretty messy.

My situation was similar, but had a different cause - thought I'd share it here in case anyone else has the same problem. Events and listeners didn't seem to be firing. I was creating jobs/triggers dynamically, pulling the trigger start time from the database. The problem is that although the datetime I pulled from the database was UTC, it wasn't marked as UTC (but rather local) when used in C#. The jobs were getting scheduled, but for 9 hours in the future when I expected them to fire within a minute. Marking the datetimes as UTC fixed this:
DateTime NextExecutionStartDate = DateTime.SpecifyKind(myDBObject.NextExecutionStartDate, DateTimeKind.Utc);
...Trigger.StartAt(NextExecutionStartDate)
instead of
...Trigger.StartAt(myDBObject.NextExecutionStartDate)

I need ideas to unit test this Tasked method

This is the method in question:
public void StartBatchProcessing(IFileBatch fileBatch)
{
var dataWarehouseFactsMerger = m_dataWarehouseFactsMergerFactory.Create(fileBatch);
dataWarehouseFactsMerger.Merge();
if(!m_isTaskStarted)
{
m_isTaskStarted = true;
m_lastQueuedBatchProcessingTask = new TaskFactory().StartNew(() => ProcessBatch(dataWarehouseFactsMerger));
}
else
{
m_lastQueuedBatchProcessingTask = m_lastQueuedBatchProcessingTask.ContinueWith(previous => ProcessBatch(dataWarehouseFactsMerger));
}
}
As you can see I'm using TPL to queue tasks one after the other and I would like to test that the tasks will execute in the order they arrive as soon as the previous one finishes.
The ProcessBatch method is protected so I think it could be overwritten in a derived class and be used to set some flag or something and assert that.
All ideas are welcome and appreciated.

You could create an implementation of DataWarehouseFactsMergerFactory that creates implementations of DataWarehouseFactsMerger that are capable of logging which fileBatch was entered and the start time of each task, but for the rest don't really do anything.