Jobs fail and stop executing after undetermined amount of time

Version: 3.0.0

Expected behavior

Quartz runs the configured jobs on the configured cron schedules forever

Actual behavior

After an undetermined amount of time, between several days and several weeks, a load of jobs crash all at the same time with this error message being logged:

2018-02-07T04:15:00.0294285+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-23': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (8373252b)
2018-02-07T04:15:00.0450537+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-25': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (99aac8e1)
2018-02-07T04:15:00.0450537+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-27': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (4f42eca1)
2018-02-07T04:15:00.0450537+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-29': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (052177ee)
2018-02-07T04:15:00.0450537+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-30': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (169580e4)
2018-02-07T04:15:00.0450537+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-31': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (c34383b0)
2018-02-07T04:15:00.0450537+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-3': Problem instantiating type 'SystemChecker.Model.Jobs.WebRequestChecker' (4e03618f)
2018-02-07T04:15:00.0450537+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-33': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (15ff2bec)
2018-02-07T04:15:00.0450537+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-35': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (9b00c9b7)
2018-02-07T04:15:00.0450537+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-39': Problem instantiating type 'SystemChecker.Model.Jobs.WebRequestChecker' (9193c6e7)
2018-02-07T04:15:00.0606788+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-4': Problem instantiating type 'SystemChecker.Model.Jobs.WebRequestChecker' (cdd6f8ae)
2018-02-07T04:15:00.0606788+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-7': Problem instantiating type 'SystemChecker.Model.Jobs.WebRequestChecker' (ae06acc7)
2018-02-07T04:15:00.0606788+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-9': Problem instantiating type 'SystemChecker.Model.Jobs.WebRequestChecker' (d8270fbd)
2018-02-07T04:15:00.0606788+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-10': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (55cc06cb)
2018-02-07T04:20:00.0002806+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-24': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (1ed26de5)
2018-02-07T04:20:00.0002806+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-26': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (970db8c2)
2018-02-07T04:20:00.0002806+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-28': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (60a62a66)
2018-02-07T04:20:00.0002806+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-32': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (151f3418)
2018-02-07T04:20:00.0002806+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-36': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (2703dffc)
2018-02-07T04:20:00.0002806+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-40': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (ce2cc71e)
2018-02-07T05:00:00.0094520+00:00  [ERR] Scheduler error: An error occurred instantiating job to be executed. job= 'DEFAULT.check-6': Problem instantiating type 'SystemChecker.Model.Jobs.DatabaseChecker' (833baa0e)

After this point, the jobs do not run at all on their schedules anymore, until the schedules are reconfigured at which point it continues to work again.

Steps to reproduce

Not quite reproducible code but shows how I am creating the jobs

public async Task UpdateSchedule(Check check)
{
    await RemoveSchedule(check);
    if (!check.Active) { return; }
    var type = GetJobForCheck(check);
    IDictionary<string, object> data = new Dictionary<string, object>
    {
        ["CheckID"] = check.ID,
        ["Logger"] = _checkLogger
    };
    var job = JobBuilder.Create(type)
        .WithIdentity(GetJobKeyForCheck(check))
        .SetJobData(new JobDataMap(data))
        .Build();

    await _scheduler.AddJob(job, false, true);

    foreach (var schedule in check.Schedules.Where(x => x.Active))
    {
        if (CronExpression.IsValidExpression(schedule.Expression))
        {
            var trigger = TriggerBuilder.Create()
                .WithIdentity(GetTriggerKeyForSchedule(schedule))
                .WithCronSchedule(schedule.Expression)
                .ForJob(job)
                .Build();

            await _scheduler.ScheduleJob(trigger);
        }
        else
        {
            _logger.LogError($"Failed to create schedule for ID {schedule.ID} on check {check.Name} ({check.ID}): {schedule.Expression}");
        }
    }
}

I’m also using a custom job factory for dependency injection:

public class JobFactory : IJobFactory
{
    private readonly IServiceProvider _container;
    private Dictionary<IJob, IServiceScope> _scopes = new Dictionary<IJob, IServiceScope>();
    public JobFactory(IServiceProvider container)
    {
        _container = container;
    }
    public IJob NewJob(TriggerFiredBundle bundle, IScheduler scheduler)
    {
        var scope = _container.CreateScope();
        var job = scope.ServiceProvider.GetRequiredService(bundle.JobDetail.JobType) as IJob;
        _scopes[job] = scope;
        return job;
    }

    public void ReturnJob(IJob job)
    {
        (job as IDisposable)?.Dispose();
        _scopes[job]?.Dispose();
        _scopes.Remove(job);
    }
}

Unfortunately the logs do not contain any stack traces so I’m not sure if the error originates from the JobFactory. I’m going to add some catching/logging there to see if it is the case but in any case the logged errors should be more informative (as per #561)

[njsent]

Thank you @MattJeanes for sharing that. Very helpful!

[MattJeanes]

Hey @njsent - I injected the IServiceProvider into the Job Factory and essentially kept a concurrent dictionary containing the job as the key and a new service scope (serviceProvider.CreateScope()) and disposed it in the job factory return job method.

You can see my implementation here: https://github.com/MattJeanes/SystemChecker/blob/master/SystemChecker.Model/Helpers/JobFactory.cs

I’ve been running that in production for the last 10 months or so and no problems at all!

Hope this helps.