Develop Reference

Hangfire causing locks in SQL Server

We are using Hangfire 1.7.2 within our ASP.NET Web project with SQL Server 2016. We have around 150 sites on our server, with each site using Hangfire 1.7.2. We noticed that when we upgraded these sites to use Hangfire, the DB server collapsed. Checking the DB logs, we found out there were multiple locking queries. We have identified one RPC Event “sys.sp_getapplock;1” In the all blocking sessions. It seems like Hangfire is locking our DB rendering whole DB unusable. We noticed almost 670+ locking queries because of Hangfire.
This could possibly be due to these properties we setup:
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(30),
QueuePollInterval = TimeSpan.FromHours(5)
Each site has around 20 background jobs, a few of them run every minute, whereas others every hour, every 6 hours and some once a day.
I have searched the documentation but could not find anything which could explain these two properties or how to set them to avoid DB locks.
Looking for some help on this.
EDIT: The following queries are executed at every second:
exec sp_executesql N'select count(*) from [HangFire].[Set] with (readcommittedlock, forceseek) where [Key] = #key',N'#key nvarchar(4000)',#key=N'retries'
select distinct(Queue) from [HangFire].JobQueue with (nolock)
exec sp_executesql N'select count(*) from [HangFire].[Set] with (readcommittedlock, forceseek) where [Key] = #key',N'#key nvarchar(4000)',#key=N'retries'
irrespective of various combinations of timespan values we set. Here is the code of GetHangfirServers we are using:
public static IEnumerable<IDisposable> GetHangfireServers()
{
// Reference for GlobalConfiguration.Configuration: http://docs.hangfire.io/en/latest/getting-started/index.html
// Reference for UseSqlServerStorage: http://docs.hangfire.io/en/latest/configuration/using-sql-server.html#configuring-the-polling-interval
GlobalConfiguration.Configuration
.SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
.UseSimpleAssemblyNameTypeSerializer()
.UseRecommendedSerializerSettings()
.UseSqlServerStorage(ConfigurationManager.ConnectionStrings["abc"]
.ConnectionString, new SqlServerStorageOptions
{
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(30),
QueuePollInterval = TimeSpan.FromHours(5), // Hangfire will poll after 5 hrs to check failed jobs.
UseRecommendedIsolationLevel = true,
UsePageLocksOnDequeue = true,
DisableGlobalLocks = true
});
// Reference: https://docs.hangfire.io/en/latest/background-processing/configuring-degree-of-parallelism.html
var options = new BackgroundJobServerOptions
{
WorkerCount = 5
};
var server = new BackgroundJobServer(options);
yield return server;
}
The worker count is set just to 5.
There are just 4 jobs and even those are completed (SELECT * FROM [HangFire].[State]):
Do you have any idea why the Hangfire is hitting so many queries at each second?

We faced this issue in one of our projects. The hangfire dashboard is pretty read heavy and it polls the hangfire db very frequently to refresh job status.
Best solution that worked for us was to have a dedicated hangfire database.
That way you will isolate the application queries from hangfire queries and your application queries won't be affected by the hangfire server and dashboard queries.

There is a newer configuration option called SlidingInvisibilityTimeout when configuring SqlServerStorage that causes these database locks as part of newer fetching non-transactional message fetching algorithm. It is meant for long running jobs that may cause backups of transactional logs to error out (as there is a database transaction that is still active as part of the long running job).
.UseSqlServerStorage(
"connection_string",
new SqlServerStorageOptions { SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5) });
Our DBA did not like the database locks, so I just removed this SlidingInvisibilityTimeout option to use the old transactional based message fetching algorithm since I didn't have any long running jobs in my queue.
Whether you enable this option or not is dependent on your situation. You may want to consider moving your queue database outside of your application database if it isn't already and enable the SlidingInvisibilityTimeout option. If your DBA can't live with the locks even if the queue is a separate database, then maybe you could refactor your tasks into many more smaller tasks that are shorter lived. Just some ideas.
https://www.hangfire.io/blog/2017/06/16/hangfire-1.6.14.html

SqlServerStorage runs Install.sql which takes an exclusive schema lock on the Hangfire-schema.
DECLARE #SchemaLockResult INT;
EXEC #SchemaLockResult = sp_getapplock #Resource = '$(HangFireSchema):SchemaLock',
#LockMode = 'Exclusive'
From the Hangfire documentation:
"SQL Server objects are installed automatically from the SqlServerStorage constructor by executing statements
described in the Install.sql file (which is located under the tools folder in the NuGet package). Which contains
the migration script, so new versions of Hangfire with schema changes can be installed seamlessly, without your
intervention."
If you don't want to run this script everytime you could set SqlServerStorageOptions.PrepareSchemaIfNecessary to false.
var options = new SqlServerStorageOptions
{
PrepareSchemaIfNecessary = false
};
var sqlServerStorage = new SqlServerStorage(connectionstring, options);
Instead run the Install.sql manually by using this line:
SqlServerObjectsInstaller.Install(connection);

How can you tell if Hangfire task was manually triggered

I have a Hangfire server set up with several recurring tasks. For local development I don't want these tasks to go through but I need to be able to manually trigger them manually through the Hangfire UI.
I am able to pull the Job Data for the currently running job but I don't see anything within it that tells me if it was manually triggered or not.
Here is an excerpt from my code where RunProcessReportsJob is my RecurringJob in Hangfire
public ExitCodeType RunProcessReportsJob(PerformContext context)
{
var jobId = context.BackgroundJob.Id;
var connection = JobStorage.Current.GetConnection();
var jobData = connection.GetJobData(jobId);
_logger.LogInformation("Reoccurring job disabled.");
return ExitCodeType.NoError;
}
The jobData has a ton of information about the job and context but again I don't see anything within this that tells me if it is a manually triggered job or a scheduled job.

Hope this helps
private bool JobWasManuallyExecuted(string jobId)
{
//'Triggered using recurring job manager' -- Manually triggerd via UI
//'Triggered by recurring job scheduler' -- using scheduller
var jobDetails = JobStorage.Current.GetMonitoringApi().JobDetails(jobId);
if (jobDetails == null)
return false;
return jobDetails.History.ToList().Any(x => x.Reason == "Triggered using recurring job manager");
}
This message appears on the UI as well.
Executed using the scheduler:
Manually executed

Send notification on specific date and time

In our App, We are storing questions with Question's startdate, enddate and resultdate. We need to send notification to app (iPhone and Andorid) once startdate of question is arrives.
Can anybody let me know how can we achieve this?
We don't want to use pull method. like in particular time interval it will check for question startdate and send notification.
I have a URL to send Notification for question. I need to call this URL when question's startdate is arrived.
Thanks.

Take a look at Quartz :
Quartz.NET is a full-featured, open source job scheduling system that can be used from smallest apps to large scale enterprise systems
Quartz Enterprise Scheduler .NET
You can create a new Quarts Job, lets call it QuestionSenderJob. Then your application can schedule a task in Quartz scheduler, jobs can have many instances of same Job with custom data - in your case QuestionId.
Additionally it supports storing Job scheduling in your SQL database (there are DDL Scripts included) so you can create some relations if you need for UI for example.
You can find table-creation SQL scripts in the "database/dbtables" directory of the Quartz.NET distribution
Lesson 9: JobStores
This way you leave firing in right moment to Quartz engine.
When you will go through Quartz .NET basics, see this code snippet I made a for your case to schedule job. Perhaps some modifications will be necessary thought.
IDictionary<string, object> jobData = new Dictionary<string, object> { { "QuestionId", questionId } };
var questionDate = new DateTime(2016, 09, 01);
var questionTriggerName = string.Format("Question{0}_Trigger", questionId);
var questionTrigger = TriggerBuilder.Create()
.WithIdentity(questionTriggerName, "QuestionSendGroup")
.StartAt(questionDate)
.UsingJobData(new Quartz.JobDataMap(jobData))
.Build();
scheduler
.ScheduleJob(questionSenderJob, questionTrigger);
Then in Job you will get your questionId through JobExecutionContext.
public class QuestionSenderJob: IJob
{
public void Execute(JobExecutionContext context)
{
JobDataMap dataMap = context.JobDetail.JobDataMap;
// Extract question Id and send message
}
}

What about using the Task Scheduler Managed Wrapper?
You do not want to use pooling, but if you write your own class that will encapsulate Timer (e.g. System.Thread.Timer) and check for the time each second, that will not take much resources. Depending on how exact you need it, you could check also less often, e.g. each minute. Maybe you should reconsider it.

If you use any third party service to manage your push notification such as Azure Notification Hub, Parse.com, ... they offer an integrated way to schedule push notifications. Either by passing in a send date or let them run a job periodically. I'm a user of the Azure service and it works very well.

The best implementation i can advice right now is for you to send the notification from a server.
All you just need is a good scheduler that can dispatch operation.
For me, my server is powered by Javascript (NodeJS) so i use "node-schedule". All i just do is
var schedule = require('node-schedule');
//Reporting rule at minute 1 every hour
var rule = new schedule.RecurrenceRule();
rule.minute = 1;
schedule.scheduleJob(rule, function () {
console.log(new Date().toTimeString() + ' Testing Scheduler! Executing Every other minute');
//sendPush()
});

Tricky Quartz.NET Scenario

Ok, little bit of background here. I have a large scale web application (MVC3) which does all kinds of unimportant stuff. I need this web application to have the ability to schedule ad-hoc Quartz.NET jobs in an Oracle database. Then, I want the jobs to be executed later on via a windows service. Ideally, I'd like to schedule them to run in even intervals, but with the option to add jobs via the web app.
Basically, the desired architecture is some variation of this:
Web app <--> Quartz.NET <--> Database <--> Quartz.NET <--> Windows Service
What I have coded up so far:
A windows service which (for now) schedules AND runs the Jobs. This obviously isn't going to be the case in the long run, but I'm wondering if I can keep just this and modify it to have it basically represent both "Quartz.NET's" in the diagram above.
The web app (details I guess aren't very important here)
The jobs (which are actually just another windows service)
And a couple important notes:
It HAS to be run from a windows service, and it HAS to be scheduled through the web app (to reduce load on IIS)
The architecture above can be rearranged a little bit, assuming the above bullet still applies.
Now, a few questions:
Is this even possible?
Assuming (1) passes, what do you guys think is the best architecture for this? See first bullet on what I've coded up.
Can somebody maybe give me a few Quartz methods that will help me out with querying the DB for jobs to execute once they're already scheduled?
There will be a bounty on this question in as soon as it is eligible. If the question is answered in a satisfactory way before then, I will still award the bounty to the poster of the answer. So, in any case, if you give a good answer here, you'll get a bounty.

I'll try answering your questions in the order you have them.
Yes, it's possible to do this. It's actually a common way of working with Quartz.Net. In fact, you can also write an ASP.Net MVC application that manages Quartz.Net schedulers.
Architecture. Ideally and at a high level, your MVC application will use the Quartz.Net API to talk to a Quartz.Net server that is installed as a windows service somewhere. Quartz.Net uses remoting to communicate remotely, so any limitations of using remoting apply (like it's not supported in Silverlight, etc). Quartz.Net provides a way to install it as a windows service out of the box, so there really isn't much work to be done here, other than configuring the service itself to use (in your case) an AdoJobStore, and also enabling remoting. There is some care to be taken around how to install the service properly, so if you haven't done that yet, take a look at this post.
Internally, in your MVC application you'll want to get a reference to the scheduler and store it as a singleton. Then in your code you'll schedule jobs and get information about the scheduler through this unique instance. You could use something like this:
public class QuartzScheduler
{
public QuartzScheduler(string server, int port, string scheduler)
{
Address = string.Format("tcp://{0}:{1}/{2}", server, port, scheduler);
_schedulerFactory = new StdSchedulerFactory(getProperties(Address));
try
{
_scheduler = _schedulerFactory.GetScheduler();
}
catch (SchedulerException)
{
MessageBox.Show("Unable to connect to the specified server", "Connection Error", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
}
}
public string Address { get; private set; }
private NameValueCollection getProperties(string address)
{
NameValueCollection properties = new NameValueCollection();
properties["quartz.scheduler.instanceName"] = "RemoteClient";
properties["quartz.scheduler.proxy"] = "true";
properties["quartz.threadPool.threadCount"] = "0";
properties["quartz.scheduler.proxy.address"] = address;
return properties;
}
public IScheduler GetScheduler()
{
return _scheduler;
}
}
This code sets up your Quart.Net client. Then to access the remote scheduler, just call
GetScheduler()
Querying
Here is some sample code to get all the jobs from the scheduler:
public DataTable GetJobs()
{
DataTable table = new DataTable();
table.Columns.Add("GroupName");
table.Columns.Add("JobName");
table.Columns.Add("JobDescription");
table.Columns.Add("TriggerName");
table.Columns.Add("TriggerGroupName");
table.Columns.Add("TriggerType");
table.Columns.Add("TriggerState");
table.Columns.Add("NextFireTime");
table.Columns.Add("PreviousFireTime");
var jobGroups = GetScheduler().GetJobGroupNames();
foreach (string group in jobGroups)
{
var groupMatcher = GroupMatcher<JobKey>.GroupContains(group);
var jobKeys = GetScheduler().GetJobKeys(groupMatcher);
foreach (var jobKey in jobKeys)
{
var detail = GetScheduler().GetJobDetail(jobKey);
var triggers = GetScheduler().GetTriggersOfJob(jobKey);
foreach (ITrigger trigger in triggers)
{
DataRow row = table.NewRow();
row["GroupName"] = group;
row["JobName"] = jobKey.Name;
row["JobDescription"] = detail.Description;
row["TriggerName"] = trigger.Key.Name;
row["TriggerGroupName"] = trigger.Key.Group;
row["TriggerType"] = trigger.GetType().Name;
row["TriggerState"] = GetScheduler().GetTriggerState(trigger.Key);
DateTimeOffset? nextFireTime = trigger.GetNextFireTimeUtc();
if (nextFireTime.HasValue)
{
row["NextFireTime"] = TimeZone.CurrentTimeZone.ToLocalTime(nextFireTime.Value.DateTime);
}
DateTimeOffset? previousFireTime = trigger.GetPreviousFireTimeUtc();
if (previousFireTime.HasValue)
{
row["PreviousFireTime"] = TimeZone.CurrentTimeZone.ToLocalTime(previousFireTime.Value.DateTime);
}
table.Rows.Add(row);
}
}
}
return table;
}
You can view this code on Github

How to get details of all scheduled jobs and triggers in Quartz.NET c#

I have to create administration page of all scheduled jobs and triggers. How can i get details of running jobs and triggers in Quartz.NET? Can I pause/stop or update jobs? Is there any sample code?

Here is how you would go about it using the StdSchedulerFactory
ISchedulerFactory schedFact = new StdSchedulerFactory();
foreach (IScheduler scheduler in schedFact.AllSchedulers)
{
var scheduler1 = scheduler;
foreach (var jobDetail in from jobGroupName in scheduler1.JobGroupNames
from jobName in scheduler1.GetJobNames(jobGroupName)
select scheduler1.GetJobDetail(jobName, jobGroupName))
{
//Get props about job from jobDetail
}
foreach (var triggerDetail in from triggerGroupName in scheduler1.TriggerGroupNames
from triggerName in scheduler1.GetTriggerNames(triggerGroupName)
select scheduler1.GetTrigger(triggerName, triggerGroupName))
{
//Get props about trigger from triggerDetail
}
}

Here an open project that does just this.
The project should have all the code you need to create you own, or you can just use the open source project.
Web Based admin page for Quartz.net
Allow registering of existing Quartz.net installations
Allow viewing of Jobs and Triggers
Allow scheduling of Jobs including editing JobDataMaps
Allow viewing of calendars
Allow viewing of trigger fire times
Silverlight based timeline showing upcoming schedules