QuartzHostedService: Wait for ApplicationStarted

Hosted services and (by extension) Quartz jobs are started before the application has successfully started. For example, if the application serves HTTP requests, the jobs are started before it is ready to respond to those requests - even before the app can respond to health checks.

This is particularly annoying for smart deployments. For example, one might use Kubernetes to spin up new application instances and consider them successfully started once they answer a health check positively. Only after this happens do we include the new instances in the load balancer and do we spin down the old application instances. However, if the new instances fail to become healthy in a timely manner, they are spun down again. The issue is that such never-officially-included instances may have already started running jobs.

Solution

The problem would be mostly mitigated if the Quartz hosted service would wait for application startup to complete:

await Task.Delay(Timeout.InfiniteTimeSpan, // Wait "indefinitely", until ApplicationStarted is triggered
    this.HostApplicationLifetime.ApplicationStarted) // IHostApplicationLifetime injected in ctor
    .ContinueWith( _ => {}, TaskContinuationOptions.OnlyOnCanceled) // Without an OperationCanceledException
    .ConfigureAwait(false));

This way, at least we know that the application has managed to start successfully. For example, any unrelated hosted services that cause startup failure will now automatically precede our jobs, which is good.

Optionally, this behavior could be configured by a boolean, QuartsHostedServiceOptions.AwaitApplicationStarted. However, I see no issue with the delay, and I consider the timing to be more correct, so I believe the adjusted behavior should be the default.

Could such a default be considered a breaking change? Probably not: Currently, there are no guarantees that a failure in a Quartz job will occur before startup completes (and thus would have the ability to prevent a successful startup). In actuality, they might happen early and prevent startup, or they might happen later and not prevent it. Going from an unpredictable scenario to a predictable seems non-breaking to me, but please correct me if I am overlooking something.

Alternatives

There is QuartzHostedServiceOptions.StartDelay. However, arbitrary delays are bad; startup times may be unpredictable.

Additional Context

We might consider the need to solve this problem as a design flaw in BackgroundService.

Services that perform startup and/or shutdown logic should implement IHostedService. Services that implement background logic (like Quartz jobs, arguably) should inherit from BackgroundService. This changes what methods the service needs to implement, in a way that makes the most sense for the situation.

For IHostedService, it makes sense that StartAsync is invoked before a host starts listening and application startup is considered completed, as an IHostedService is permitted to cause application startup to fail (since .NET Core 3). This enables startup dependencies.

However, BackgroundService also runs its Execute method before startup, as a result of IHostedService.StartAsync being invoked. I would argue that the abstract BackgroundService class should be responsible for awaiting the ApplicationStarted token. (In fact, I might open an issue with this very suggestion.)

Regardless, BackgroundService.Execute is currently being invoked early, so we should handle the situation as well as possible.

[lahma]

Closing this as fixed, thank you for the PR!

[lahma]

Thank you @andrewlock for the insights, I think we have an agreement that the change is welcome. @Timovzl I would appreciate a PR to tweak the behavior and maybe we can try to ping Andrew again if he happened to have the time to review it based on discussion.