Is there a way to set hangfire recurring jobs every few seconds?
I do not seek a solution where fire and forget task creates another fire and forget task, and if not, what are suggested alternatives?
Not sure when this became supported but tried this in ASP.NET Core 2.0 with Hangfire 1.7.0. The following code schedules a job every 20 seconds:
RecurringJob.AddOrUpdate<SomeJob>(
x => x.DoWork(),
"*/20 * * * * *");
If I am not mistaken 6 tokens (as opposed to standard 5 tokens) is supported due to Hangfire use of NCrontab which allows cron expressions with 6 tokens (second granularity instead of minute granularity).
Hangfire dashboard also nicely shows the small time interval between runs:
I think anyone who is against allowing a recurring trigger of less than 1 min is short sighted. After all, is 55 secs any less efficient than 1 min ? It seems so arbitrary! As much as I love Hangfire, I've encountered situations where I've had to steer a client to Quartz.net simply because there was a business requirement for a job to run every 55 secs or alike.
Anyone who makes the counter argument that if it was configured to run every 1sec it would have a serious impact on performance is again taking a closed view on things. Of course a trigger with a 1 sec interval is probably not a good idea, but do we dissalow 55 sec or 45 sec for the unlikely situation where someone will choose 1 sec ?
In any case, performance is both subjective and dependent on the host platform and hardware. It really isn't up to the API to enforce opinion when it comes to performance. Just make the polling interval and trigger recurrence configurable. That way the user can determine the best result for themselves.
Although a background process which is orchestrating a job to run every 55 sec may be an option, it isn't very satisfactory. In this case, the process isn't visible via the Hangfire UI so it's hidden from the administrator. I feel this approach is circumventing one of the major benefits of Hangfire.
If Hangfire was a serious competitor to the likes of Quartz.net it would at least match their basic functionality. If Quartz can support triggers with an interval below 1 min than why can't Hangfire!
Although Hangfire doesn't allow you to schedule tasks for less than a minute, you can actually achieve this by having the function recursively scheduling itself; i.e. let's say you want some method to be hit every 2s you can schedule a background job that calls the method on Startup;
BackgroundJob.Schedule(() => PublishMessage(), TimeSpan.FromMilliseconds(2000));
And then in your PublishMessage method do your stuff and then schedule a job to call the same method;
public void PublishMessage()
{
/* do your stuff */
//then schedule a job to exec the same method
BackgroundJob.Schedule(() => PublishMessage(), TimeSpan.FromMilliseconds(2000));
}
The other thing you need to override is the default SchedulePollingInterval of 15s, otherwise your method will only be hit after every 15s. To do so just pass in an instance of BackgroundJobServerOptions to UseHangfireServer in your startup, like so;
var options = new BackgroundJobServerOptions
{
SchedulePollingInterval = TimeSpan.FromMilliseconds(2000)
};
app.UseHangfireServer(options);
I don't know how "foolproof" my solution is, but I managed to achieve my goal with it and everything is "happy" in production.
I had to do the same but with 5 seconds. The default schedule polling interval is set to 15s. So it requires 2 steps to achieve a 5s interval job.
in Startup.cs
var options = new BackgroundJobServerOptions
{
SchedulePollingInterval = TimeSpan.FromMilliseconds(5000)
};
app.UseHangfireDashboard();
app.UseHangfireServer(options);
Your job
RecurringJob.AddOrUpdate(() => YourJob(), "*/5 * * * * *");
Hangfire doesn't support intervals of less than a minute for recurring jobs.
Why? Imagine if they allowed less than a minute: let say 1 sec. How frequently would hangfire check recurring jobs in the database? This would cause a lot of database IO.
See this discussion on Hangfire for more information.
I faced with the same problem, and here it is my solution:
private void TimingExecuteWrapper(Action action, int sleepSeconds, int intervalSeconds)
{
DateTime beginTime = DateTime.UtcNow, endTime;
var interval = TimeSpan.FromSeconds(intervalSeconds);
while (true)
{
action();
Thread.Sleep(TimeSpan.FromSeconds(sleepSeconds));
endTime = DateTime.UtcNow;
if (endTime - beginTime >= interval)
break;
}
}
intervalSeconds is minimal NCRON interval. It is 1 minute.
action is our job code.
Also I suggest to use DisableConcurrentExecution to avoid some collisions of concurrency.
I had a similar requirement, in that I had a recurring job that needs running every 15 seconds.
What I did to try to get around this limitation was to just delay the creation of the scheduled jobs (set to 1 minute intervals), which seemed to do the trick.
However what I found was that, taking into account the polling intervals (set the schedulepolling interval to my frequency) and delays in picking up the new jobs this isn't always as accurate as it should be, but is doing the trick for the moment. However a better/proper solution would be good.
feel a bit dirty having to resolve to the below approach, but it helped me out...
so in essence I created 4 jobs doing the same thing, created 15 seconds apart.
along the lines of:
...
new Thread(() =>
{
//loop from {id=1 through 4}
// create job here with {id} in the name at interval of 1 minute
// sleep 15 seconds
//end loop
}).Start();
...
Related
I have an application that uses hangfire to do long-running jobs for me (I know the time the job takes and it is always roughly the same), and in my UI I want to give an estimate for when a certain job is done. For that I need to query hangfire for the position of the job in the queue and the number of servers working on it.
I know I can get the number of enqueued jobs (in the "DEFAULT" queue) by
public long JobsInQueue() {
var monitor = JobStorage.Current.GetMonitoringApi();
return monitor.EnqueuedCount("DEFAULT");
}
and the number of servers by
public int HealthyServers() {
var monitor = JobStorage.Current.GetMonitoringApi();
return monitor.Servers().Count(n => (n.Heartbeat != null) && (DateTime.Now - n.Heartbeat.Value).TotalMinutes < 5);
}
(BTW: I exclude older heartbeats, because if I turn off servers they sometimes linger in the hangfire database. Is there a better way?), but to give a proper estimate I need to know the position of the job in the queue. How do I get that?
The problem you have is that hangfire is asynchronous, queued, parallel, exhibits an at-least-once durability semantic, and basically non-deterministic.
To know with certainty the order in which an item will finish being processed in such a system is impossible. In fact, if the requirement was to enforce strict ordering, then many of the benefits of hangfire would go away.
There is a very good blog post by #odinserj (the author of hangfire) where he outlines this point: http://odinserj.net/2014/05/10/are-your-methods-ready-to-run-in-background/
However, that said, it's not impossible to come up with a sensible estimation algorithm, but it would have to be one where the order of execution is approximated in some way. As to how you can arrive at such an algorithm I don't know but something like this might work (but probably won't):
Approximate seconds remaining until completion =
(
(average duration of job in seconds * queue depth)
/ (the lower of: number of hangfire threads OR queue depth)
)
- number of seconds already spent in queue
+ average duration of job in seconds
I have a number of tasks that i need run every 20 minutes. I can achieve this in Hangfire using the cron expression RecurringJob.AddOrUpdate(() => /* my method */, "*/20 * * * *")
The problem with this is, all my tasks will end up running at exactly the same time e.g. 00:00, 00:20, 00:40, 01:00
Does anyone know of a way to stagger one's 20 minute interval tasks without implementing some sort of manual intervention?
I'd just like Hangfire to run the jobs every 20 minutes from the time I create them, nothing too special :)
I'd try this: First, modify your add or update to specify a job id:
RecurringJob.AddOrUpdate("some-id", () => /* my method */, "*/20 * * * *");
followed by
RecurringJob.Trigger("some-id");
To run a recurring job now, call the Trigger method.
UPDATE: How about a delayed job then instead of a recurring job? Your "my method" creates a new delayed job at the end (Recursion), set to run after 20 mins. You will still use the Trigger method for the first run.
One 'hacky' way of achieving this is to set a small number of workers.
app.UseHangfire(config =>
{
config.UseServer(2);
});
// or
var server = new BackgroundJobServer(2);
Whether you look at this and say 'oh great!' or 'that is a terrible idea' will depend completely on your own usage of Hangfire. If you only run a few things an hour, or every 15 minutes and you find they're 'treading' on each other then this could be a simple solution.
A second way: You could make an async Task and run await Task.Delay(TimeSpan.FromSeconds(10)) to make a short delay. This quite likely could cause your app to shutdown more slowly because it would think you're still running tasks. So I only only using this in an emergency, or if you're ok with the consequences.
Hopefully Hangfire will support this in a more organized way at some point.
I've been given a task to write a program to count how many page views are requested from our site. My current approach is to get data from google analytics Real Time API, which works to my suprise.
My problem is that to get pageviews every minute I need to poll data from google API twice (cause it returns sum of last 29 minutes + a value from a timer that resets every minute). After I set up 'the point of reset', lets just say, on a 55th second every minute, I poll data on 56th and later on at 53th second, which gives me relatively good estimation of new users / page views requested.
So this is my current approach:
static System.Timers.Timer myTimer = new System.Timers.Timer();
myTimer.AutoReset = false;
myTimer.Interval = interval();
myTimer.Elapsed += myTimer_Elapsed2;
myTimer.Start();
static double interval()
{
return 1000 - DateTime.Now.Millisecond;
}
static void myTimer_Elapsed2(object sender, System.Timers.ElapsedEventArgs e)
{
if (DateTime.Now.Second == (resetPoint.Second - 1) % 60 && warden)
{
DoStuff(); //mostly inserting google API data to database
}
else if (DateTime.Now.Second == (resetPoint.Second + 1) % 60) //so we dont get riddiculous 60 and above
{
//I get some data here, to later use it in DoStuff - mostly to calculate the gap between later
}
myTimer.Interval = interval(); //Because DoStuff() takes about 0.5 sec to execute, i need to recalibrate
myTimer.Start();
}
And it works really well, until it stops after about 2 hours, for now I have no idea why (program runs, just timer doesn't do its work anymore).
How do I make it stable for long periods of time? Best case scenario would be to run it for months without intervention.
# I edited to give a better sense what I'm actually doing
#END CREDITS
I ended up using two timers, each running in a one minute circle. And a database writing sometimes crashed and I didn't handle the corresponding exception properly. Log told me that google API functions from time to time tend to retrieve data a bit longer, which led to multiple Threading.Event calls and made my database data handling throw an exception hence stopping the timer.
I tried to use Quartz approach but its lack of human-friendly howto made me abandon this library.
You should really look into using Quartz.net for scheduling events on a reliable basis. Using a timer for scheduling is asking for stuff like race conditions, event skips and database deadlocks.
http://www.quartz-scheduler.net/ allows you to schedule events at precise intervals, independant of when your code starts or stops.
An example on how you use it: This will build a trigger that will fire at the top of the next hour, then repeat every 2 hours, forever:
trigger = TriggerBuilder.Create()
.WithIdentity("trigger8") // because group is not specified, "trigger8" will be in the default group
.StartAt(DateBuilder.EvenHourDate(null)) // get the next even-hour (minutes and seconds zero ("00:00"))
.WithSimpleSchedule(x => x
.WithIntervalInHours(2)
.RepeatForever())
// note that in this example, 'forJob(..)' is not called
// - which is valid if the trigger is passed to the scheduler along with the job
.Build();
scheduler.scheduleJob(trigger, job);
http://www.quartz-scheduler.net/documentation/quartz-2.x/tutorial/simpletriggers.html has a few examples. I really URGE you to use it, since it will severely simplify development.
The .NET timer is reliable. That is, it won't just stop working randomly for no apparent reason.
Most likely, something in your timer event handler is throwing an exception, which is not surfaced because System.Timers.Timer squashes exceptions. As the documentation states:
The Timer component catches and suppresses all exceptions thrown by event handlers for the Elapsed event. This behavior is subject to change in future releases of the .NET Framework.
That bit about the behavior being "subject to change" has been there since at least .NET 2.0.
What I think is happening is that the timer calls your event handler. The event handler or one of the methods it calls throws an exception, and the timer just drops it on the floor because you don't handle it.
You need to put an exception handler in your myTimer_Elapsed2 method so that you can at least log any exceptions that crop up. With the information provided from the exception log, you can probably identify what the problem is.
Better yet, stop using System.Timers.Timer. Use System.Threading.Timer instead.
Finally, there's no way that your code as written will reliably give you a timer tick at exactly 55 seconds past the minute, every minute. The timer isn't exact. It will be off by a few milliseconds each minute. Over time, it's going to start ticking at 54 seconds (or maybe 56), and then 53 (or 57), etc. If you really need this to tick reliably at 55 seconds past the minute, then you'll need to reset the timer after every minute, taking into account the current time.
I suspect that your need to check every minute at exactly the 55 second mark is overkill. Just set your timer to tick every minute, and then determine the exact elapsed time since the last tick. So one "minute" might be 61 or 62 seconds, and another might be 58 or 59 seconds. If you store the number of requests and the elapsed time, subsequent processing can smooth the bumps and give you a reliable requests-per-minute number. Trying to gather the data on exact one-minute boundaries is going to be exceedingly difficult, if even possible with a non-real-time operating system like Windows.
I am working on an ASP.Net MVC 3 C# project.
I have read about ThreadPool.QueueUserWorkItem and Tasks, I think what i need should use one of them.
public static void ProcessDay(DateTime day) { ... }
Let's say that I want to process days from 2012-01-01 to 2012-01-31, So I have to call ProcessDay function 31 times.
How to to process 5 days at once, and keep running more threads as any of the 5 threads is done, till it finishes the 31 days?
Samples are highly appreciated, and considering showing progress bar is a good idea :)
Have a look at TPL library - for example,
DateTime startDate = new DateTime(2012, 01, 01);
var options = new ParallelOptions() { MaxDegreeOfParallelism = 5 };
Parallel.For(0, 30, options, day => { ProcessDay(startDay.AddDays(day)); });
A simple parallelization of this could be:-
var dates = Enumerable.Range(1, DateTime.DaysInMonth(year, month)).Select(day => new DateTime(year, month, day));
Parallel.ForEach(dates, date => { ... });
(Note that, if you have shared state that each thread needs to modify, then you will have to use thread synchronization techniques appropriately.)
As for the number of threads, it's usually best to leave Parallel.ForEach() to work this out for itself. It will use the thread pool internally and will use all available processors efficiently. You can limit the number of threads it will use by using an overload of ForEach which takes a ParallelOptions object with MaxDegreeOfParallelism set to the required amount, but you should run some benchmarks to see whether you can really achieve better results by doing this.
Moreover, if this is a 'long running task' for which you wish to show progress, then you won't be able to achieve this with a single web request. The notion of a 'progress bar' in a website is rather different to that in a desktop application. In a website this typically has to be done by polling. I.e. send a request which starts the process (response returned immediately), poll periodically for progress (which can be displayed in a progress bar or similar) and then display the result when finished.
There are also frameworks and techniques being developed for 'push' type notification e.g. SignalR, so this may also be an option if you want to investigate it.
So a simple enough question really.
How exactly does the interval for System.Timers work?
Does it fire 1 second, each second, regardless of how long the timeout event takes or does it require the routine to finish first and then restarts the interval?
So either:
1 sec....1 sec....1 sec and so on
1 sec + process time....1 sec + process time....1 sec + process time and so on
The reason I ask this is I know my "processing" takes much less than 1 second but I would like to fire it every one second on the dot (or as close as).
I had been using a Thread.Sleep method like so:
Thread.Sleep(1000 - ((int)(DateTime.Now.Subtract(start).TotalMilliseconds) >= 1000 ? 0 : (int)(DateTime.Now.Subtract(start).TotalMilliseconds)));
Where start time is registered at start of the routine. The problem here is that Thread.Sleep only works in milliseconds. So my routine could restart at 1000ms or a fraction over like 1000.0234ms, which can happen as one of my routines takes 0ms according to "TimeSpan" but obviously it has used ticks/nanoseconds - which would then mean the timing is off and is no longer every second. If I could sleep by ticks or nanoseconds it would be bang on.
If number 1 applies to System.Timers then I guess I'm sorted. If not I need some way to "sleep" the thread to a higher resolution of time i.e ticks/nanoseconds.
You might ask why I do an inline IF statement, well sometimes the processing can go above 1000ms so we need to make sure we don't create a minus figure. Also, by the time we determine this, the ending time has changed slightly - not by much, but, it could make the thread delay slightly longer causing the entire subsequent sleeping off.
I know, I know, the time would be negligible... but what happens if the system suddenly stalled for a few ms... it would protect against that in this case.
Update 1
Ok. So I didn't realise you can put a TimeSpan in as the timing value. So I used the below code:
Thread.Sleep(TimeSpan.FromMilliseconds(1000) - ((DateTime.Now.Subtract(start).TotalMilliseconds >= 1000) ? TimeSpan.FromMilliseconds(0) : DateTime.Now.Subtract(start)));
If I am right, this should then allow me to repeat the thread at exactly 1 second - or as close as the system will allow.
IF you have set AutoReset = true; then your theory 1 is true, otherwise you would have to deal with it in code – see the docuementation for Timer on MSDN.