Triggers transform to Error state - Cannot access a disposed object. Object name: 'IServiceProvider'

Describe the bug

In some rare occasion we are getting the following exception: Quartz.SchedulerException: Problem instantiating type '<Job full name>': Cannot access a disposed object. Object name: 'IServiceProvider' This will transform the trigger into an Error state which will cause it cease firing… We are using the MicrosoftDependencyInjectionJobFactory and noticed it is happening when the service provider tried to create a scope (Even before it started to initialize the Job itself). So it tries to use a disposed IServiceProvider which shouldn’t occur in the first place, I dont have clear idea why it is getting disposed, but I guess it could happen in some rare cases, like right before restarting a pod. In any case such scenarios shouldn’t transfer a trigger to an Error state since the initialization of the job was not even started.

Version used

3.4

To Reproduce

not easy one since it is rare, the obvious option is to dispose the service provider while Quartz is running.

Expected behavior

Not transforming the trigger state to Error

Additional context Here is the stuck trace:

Quartz.SchedulerException: Problem instantiating type 'SchedulerService.Scheduler.QuartzNet.Jobs.AutoSyncImport.AutoSyncImportJob: Cannot access a disposed object.
Object name: 'IServiceProvider'.'
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.CreateScope(IServiceProvider provider)
   at Quartz.Simpl.MicrosoftDependencyInjectionJobFactory.InstantiateJob(TriggerFiredBundle bundle, IScheduler scheduler)
   at Quartz.Simpl.PropertySettingJobFactory.NewJob(TriggerFiredBundle bundle, IScheduler scheduler)
   at Quartz.Core.JobRunShell.Initialize(QuartzScheduler sched, CancellationToken cancellationToken)
   --- End of inner exception stack trace --- [See nested exception: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.CreateScope(IServiceProvider provider)
   at Quartz.Simpl.MicrosoftDependencyInjectionJobFactory.InstantiateJob(TriggerFiredBundle bundle, IScheduler scheduler)
   at Quartz.Simpl.PropertySettingJobFactory.NewJob(TriggerFiredBundle bundle, IScheduler scheduler)
   at Quartz.Core.JobRunShell.Initialize(QuartzScheduler sched, CancellationToken cancellationToken)]

Our setup:

            services.AddQuartz(q =>
            {
                q.UseSimpleTypeLoader();
                q.UseDefaultThreadPool(tp =>
                {
                    tp.MaxConcurrency = 10;
                });

                q.UseMicrosoftDependencyInjectionScopedJobFactory(options =>
                {
                    options.AllowDefaultConstructor = true;
                });
            });
            
            services.AddQuartzServer(options =>
            {
                options.WaitForJobsToComplete = true;
            });

[DorSedoff]

Hopefully I’ll have some time handling this and reproducing the next week, thanks!

[lahma]

Looking at the code again, I think changing exception type won’t help here. Would really require a proper detection that host is going down and should not even try to do anything.