I have a situation where I need a recurring job registered with hangfire to run on every server in the cluster.
(The job is to copy some files locally so needs to run on every server regularly)
So far I have tried registering the same job with an id of the server name resulting in n job for n servers:
RecurringJob.AddOrUpdate(Environment.MachineName, () => CopyFiles(Environment.MachineName), Cron.MinuteInterval(_delay));
and the job itself checks if it is the correct server and only does something if it is:
public static void CopyFiles(string taskId)
{
if (string.IsNullOrWhiteSpace(taskId) || !taskId.Equals(Environment.MachineName))
{
return;
}
// do stuff here if it matches our taskname
}
The problem with this is that all jobs executes on the first server to come along, is marked as complete and as a result is not executed by the other servers.
Is there any way to ensure that the job runs on all servers?
or is there a way to ensure that only one server can process a given job? i.e. target the job at the server that created it
Found an answer using this link.
Simply assign the job to a queue that is specific to the server you want it processing on.
So I changed my enqueue to:
RecurringJob.AddOrUpdate(Environment.MachineName,
() => CopyFiles(Environment.MachineName),
Cron.MinuteInterval(_delay),
queue: Environment.MachineName.ToLower(CultureInfo.CurrentCulture));
And when I start my server I do this:
_backgroundJobServer = new BackgroundJobServer(new BackgroundJobServerOptions
{
Queues = new[] { Environment.MachineName.ToLower() }
});
Related
I have two Azure Functions. One is HTTP triggered, let's call it the API and the other one ServiceBusQueue triggered, and let's call this one the Listener.
The first one (the API) puts an HTTP request into a queue and the second one (the Listener) picks that up and processes that. The functions SDK version is: 3.0.7.
I have two projects in my solution for this. One which contains the Azure Functions and the other one which has the services. The API once triggered, calls a service from the other project that puts the message into the queue. And the Listener once received a message, calls a service from the service project to process the message.
Any long-running process?
The Listener actually performs a lightweight workflow and it all happens very quickly considering the amount of work it executes. The average time of execution is 90 seconds.
What's the queue specs?
The queue that the Listener listens to and is hosted in an Azure ServiceBus namespace has the following properties set:
Max Delivery Count: 1
Message time to live: 1 day
Auto-delete: Never
Duplicate detection window: 10 min
Message lock duration: 5 min
And here a screenshot for it:
The API puts the HTTP request into the queue using the following method:
public async Task ProduceAsync(string queueName, string jsonMessage)
{
jsonMessage.NotNull();
queueName.NotNull();
IQueueClient client = new QueueClient(Environment.GetEnvironmentVariable("ServiceBusConnectionString"), queueName, ReceiveMode.PeekLock)
{
OperationTimeout = TimeSpan.FromMinutes(5)
};
await client.SendAsync(new Message(Encoding.UTF8.GetBytes(jsonMessage)));
if (!client.IsClosedOrClosing)
{
await client.CloseAsync();
}
}
And the Listener (the service bus queue triggered azure function), has the following code to process the message:
[FunctionName(nameof(UpdateBookingCalendarListenerFunction))]
public async Task Run([ServiceBusTrigger(ServiceBusConstants.UpdateBookingQueue, Connection = ServiceBusConstants.ConnectionStringKey)] string message)
{
var data = JsonConvert.DeserializeObject<UpdateBookingCalendarRequest>(message);
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarListenerFunction)} picked up a message at {DateTime.Now}. Data: {data}");
await _workflowHandler.HandleAsync(data);
}
The Problem
The Listener function processes the same message 3 times! And I have no idea why! I've Googled and read through a few of StackOverFlow threads such as this one. And it looks like that everybody advising to ensure lock duration is long enough for the process to get executed completely. Although, I've put in 5 minutes for the lock, yet, the problem keeps coming. I'd really appreciate any help on this.
Just adding this in here so might be helpful for some others.
After some more investigations I've realized that in my particular case, the issue was regardless of the Azure Functions and Service Bus. In my workflow handler that the UpdateBookingCalendarListenerFunction sends messages to, I was trying to call some external APIs in a parallel approach, but, for some unknown reasons (to me) the handler code was calling off the external APIs one additional time, regardless of how many records it iterates over. The below code shows how I had implemented the parallel API calls and the other code shows how I've done it one by one that eventually led to a resolution for the issue I had.
My original code - calling APIs in parallel
public async Task<IEnumerable<StaffMemberGraphApiResponse>> AddAdminsAsync(IEnumerable<UpdateStaffMember> admins, string bookingId)
{
var apiResults = new List<StaffMemberGraphApiResponse>();
var adminsToAdd = admins.Where(ad => ad.Action == "add");
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarWorkflowDetailHandler)} Recognized {adminsToAdd.Count()} admins to add to booking with id: {bookingId}");
var addAdminsTasks = adminsToAdd.Select(admin => _addStaffGraphApiHandler.HandleAsync(new AddStaffToBookingGraphApiRequest
{
BookingId = bookingId,
DisplayName = admin.DisplayName,
EmailAddress = admin.EmailAddress,
Role = StaffMemberAllowedRoles.Admin
}));
if (addAdminsTasks.Any())
{
var addAdminsTasksResults = await Task.WhenAll(addAdminsTasks);
apiResults = _populateUpdateStaffMemberResponse.Populate(addAdminsTasksResults, StaffMemberAllowedRoles.Admin).ToList();
}
return apiResults;
}
And my new code without aggregating the API calls into the addAdminsTasks object and hence with no await Task.WhenAll(addAdminsTasks):
public async Task<IEnumerable<StaffMemberGraphApiResponse>> AddStaffMembersAsync(IEnumerable<UpdateStaffMember> members, string bookingId, string targetRole)
{
var apiResults = new List<StaffMemberGraphApiResponse>();
foreach (var item in members.Where(v => v.Action == "add"))
{
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarWorkflowDetailHandler)} Adding {targetRole} to booking: {bookingId}. data: {JsonConvert.SerializeObject(item)}");
apiResults.Add(_populateUpdateStaffMemberResponse.PopulateAsSingleItem(await _addStaffGraphApiHandler.HandleAsync(new AddStaffToBookingGraphApiRequest
{
BookingId = bookingId,
DisplayName = item.DisplayName,
EmailAddress = item.EmailAddress,
Role = targetRole
}), targetRole));
}
return apiResults;
}
I've investigated the first approach and the numbers of tasks were exact match of the number of the IEnumerable input, yet, the API was called one additional time. And within the _addStaffGraphApiHandler.HandleAsync, there is literally nothing than an HttpClient object that raises a POSTrequest. Anyway, using the second code has resolved the issue.
I have a .Net application that utilizes multiple Hangfire servers.
I want to be able to have one Hangfire RecurringJob trigger multiple BackgroundJobs that can be picked up by any available server. Currently whenever I schedule Background Jobs from a Hangfire Job only the server that scheduled them will process them.
For example, I have 5 Hangfire Servers and 10 tasks.
I would want there to be 2 tasks on each Hangfire server, instead I am seeing 1 server with 10 tasks and 4 with 0.
So again I have 5 Hangfire servers, all using the same database, and 1 RecurringJob, this RecurringJob just reads some files and enqueues several background jobs.
foreach (var file in reportSourceSetFileList)
{
_logger.LogInformation($"Queuing Background job for: {file}");
var backgroundJobId = BackgroundJob.Enqueue<IJobHandler>(job => job.ProcessFile(file, files, null));
}
However, only the Hangfire Server that ran the RecurringJob will process the Enqueued job.
How can I have those Enqueued jobs be processed by any of my 5 Hangfire Servers and not just the one that queued them?
There is no built in functionality in Hangfire to use a round robin type load balancer between multiple hangfire servers.
My solution was to use the Queuing system. When each Hangfire server starts they are given a task identifier, which is a GUID, I also add a unique queue to that server which uses the same GUID as its name.
So each server will look at 2 queues, Default and GUID.
Then I use the following code to find which server has the least jobs currently processing.
private string GetNextAvailableServer()
{
var serverJobCounts = new Dictionary<string, int>();
//get active servers
var serverList = JobStorage.Current.GetMonitoringApi().Servers();
foreach (var server in serverList)
{
if (server.Heartbeat.Value > DateTime.Now.AddMinutes(-1))
{
serverJobCounts.Add(server.Name, 0);
foreach (var queue in server.Queues)
{
var currentQueues = JobStorage.Current.GetMonitoringApi().Queues();
serverJobCounts[server.Name] += (int?)currentQueues.FirstOrDefault(e => e.Name == queue)?.Length ?? 0;
}
}
}
var jobs = JobStorage.Current.GetMonitoringApi().ProcessingJobs(0, int.MaxValue);
foreach (var job in jobs)
{
if (serverJobCounts.ContainsKey(job.Value.ServerId))
{
serverJobCounts[job.Value.ServerId] += 1;
}
}
var nextServer = serverJobCounts.OrderBy(e => e.Value).FirstOrDefault().Key;
return nextServer.Split(':')[0].Replace("-", string.Empty, StringComparison.InvariantCulture);
}
This returns the GUID of the server that has the least jobs, which is also the name of the Queue. Therefore you can schedule the next job to the specific queue with the least jobs currently processing.
var nextServer = GetNextAvailableServer();
var client = new BackgroundJobClient();
var state = new EnqueuedState(nextServer);
var enqueueJob = client.Create<IJobHandler>(job => job.ProcessFile(file, files, null), state);
Additionally when I wrote this Hangfire didn't allow for hyphens in a queue name, hence my string manipulation to make the GUIDs work. I think the newest version of hangfire lets you use hyphens in the name.
One thing to look out for, this solution breaks when one of you server dies. Since a job is given a unique Queue if the server watching that queue dies before processing the job it will never be picked up.
I have a Hangfire server set up with several recurring tasks. For local development I don't want these tasks to go through but I need to be able to manually trigger them manually through the Hangfire UI.
I am able to pull the Job Data for the currently running job but I don't see anything within it that tells me if it was manually triggered or not.
Here is an excerpt from my code where RunProcessReportsJob is my RecurringJob in Hangfire
public ExitCodeType RunProcessReportsJob(PerformContext context)
{
var jobId = context.BackgroundJob.Id;
var connection = JobStorage.Current.GetConnection();
var jobData = connection.GetJobData(jobId);
_logger.LogInformation("Reoccurring job disabled.");
return ExitCodeType.NoError;
}
The jobData has a ton of information about the job and context but again I don't see anything within this that tells me if it is a manually triggered job or a scheduled job.
Hope this helps
private bool JobWasManuallyExecuted(string jobId)
{
//'Triggered using recurring job manager' -- Manually triggerd via UI
//'Triggered by recurring job scheduler' -- using scheduller
var jobDetails = JobStorage.Current.GetMonitoringApi().JobDetails(jobId);
if (jobDetails == null)
return false;
return jobDetails.History.ToList().Any(x => x.Reason == "Triggered using recurring job manager");
}
This message appears on the UI as well.
Executed using the scheduler:
Manually executed
Good evening,
In my SignalR application I have a javascript timer that is ran for all users "simultaneously". At the end of this timer, a server function is called, and this is where this problem starts.
As the function is called at the end of the timer, every connected user calls it at the same time, which is unnecessary because it will return the same output for all connected users. Being a logically complex function, having the server run it unnecessarily for all users adds up to be a great resource waste.
How can I make it so that it is ran only once (maybe the first time it is called (until the next timer stops))?
Thank you in advance
You could make use of GlobalHost.ConnectionManager.GetHubContext. This will allow you to get any hub context and then trigger Clients.All.YourFunction on that context. That will send send a message to all connected clients subscribed to that hub.
You will need to have a background process that runs every at the time your JavaScript function fires (by the way, relying on all your clients to call a JavaScript function simultaneously is really not a good idea; different client locations and different machine performance will mean they're not likely to be simultaneous).
The following is assuming that you're just running this on a single server. If you're going to be deploying this to a web farm, then you'll need to use a Database value to ensure you don't repeat the same work, or set up a particular server instance to be responsible for doing the calls (otherwise you'll end up with one call per server).
Create a process that runs in the Background (I'm sticking with a simple thread here, I actually use HangFire for this, but this will suffice for the example), e.g. On App_Start
Thread thread = new Thread(new ThreadStart(YourFunction));
thread.Start();
Then create YourFunction which will be responsible for your client calls:
private bool Cancel = false;
private void YourFunction()
{
do
{
string foo = "Foo";
IHubContext context = GlobalHost.ConnectionManager.GetHubContext<YourHub>();
context.Clients.All.SendYourMessage(foo);
Thread.Sleep(10000);
}
while(!Cancel)
}
And then on the client, just handle the message from the hub:
youyHub.client.sendYourMessage = function(message)
{
// message == "Foo"
};
What I have is an AJAX form on a View that makes a call to the server. This call perform n number of tasks where n is a number decided by records in a database (typically no more than 10 records). Each record corresponds to a Build Definition in TFS so what I am trying to do is get all of these Build Definitions, queue them in TFS, and as each build completes update the UI so that user knows which builds have completed.
Unfortunately I am not sure about how best to do this. I was thinking something along these lines:
foreach (var dep in builds)
{
TFS tfsServer = new TFS(TFS_SERVER_ADDRESS);
IBuildServer buildServer;
int id = tfsServer.QueuBuild(dep.TeamProject, dep.BuildDefinition);
string teamProject = dep.TeamProject;
Task.Factory.StartNew(() => GetBuildStatus(teamProject, id, tfsServer));
}
The task that is called is:
private void GetBuildStatus(string TeamProject, int BuildID, TFS Server)
{
Server.GetBuildStatus(TeamProject, BuildID);
AsyncManager.OutstandingOperations.Decrement();
}
The problem here is that my Completed method isn't going to get called until all of the builds have completed. How would I go about feeding data back up to the UI a piece at a time?
It is also worth mentioning that the GetBuildStatus method looks like this:
do
{
var build = buildsView.QueuedBuilds.FirstOrDefault(x => x.Id == BuildID);
if(build != null)
{
status = build.Status;
detail = build.Build;
}
} while (status != QueueStatus.Completed);
return detail.Status.ToString();
Given that the duration of a build will be longer than the timeout for an HTTP request you cannot leave the browser waiting while this happens. You need to return a page and then poll for updates from that page using AJAX. Typically you'd have a timer in javascript that triggers a regular call back to the server to get the updated status information.
But, since you are using .NET you could also consider trying SignalR which lets you use long polling, server sent events or web sockets to wait for updates from the server and it wraps it all up in some easy to implement .NET classes and Javascript.