Exceptions in BackgroundService ExecuteAsync are (sometimes) hidden

Describe the bug

Using the dotnet new worker template, I have reproduced a problem that’s been causing us problems in our testing. If an exception occurs inside the overriden ExecuteAsync() method of our service that inherits from BackgroundService, sometimes it is hidden/lost and the program hangs very ungracefully…

But it’s only sometimes hidden/lost… and here’s my best triage: if the exception occurs anywhere before the first “await”, then it’s NOT hidden/lost.

To Reproduce

Steps to reproduce the behavior:

1. Using version 3.0 preview 9 of the .NET Core SDK
2. Use the dotnet new worker command to generate a new project based on the Worker template
3. Modify the Main() method in Program.cs as follows:

        public static void Main(string[] args)
        {
            try
            {
                CreateHostBuilder(args).Build().Run();

                Console.WriteLine("No exception!");
            }
            catch (Exception ex)
            {
                Console.WriteLine("Main method caught exception:  " + ex);
                Environment.Exit(-1);
            }
        }

1. Modify Worker.cs as follows (an exception is thrown AFTER the await):

    public class Worker : BackgroundService
    {
        private readonly ILogger<Worker> _logger;

        public Worker(ILogger<Worker> logger)
        {
            _logger = logger;
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);

                await Task.Delay(1000, stoppingToken);

                throw new ApplicationException("If an application occurs here, it will NOT bubble up to Main()");
            }
        }
    }

1. Run the program, and when the exception occurs it never bubbles up to Main(). The program hangs, and you have to press Ctrl+C to end the application - but you have no indication an exception has occurred:

info: WorkerExceptionTest.Worker[0]
      Worker running at: 09/17/2019 16:24:01 -05:00
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\spikes\WorkerExceptionTest
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
No exception!

C:\spikes\WorkerExceptionTest\bin\Debug\netcoreapp3.0\WorkerExceptionTest.exe (process 30284) exited with code 0.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .

1. Modify Worker.cs as follows (an exception is thrown BEFORE the await):

    public class Worker : BackgroundService
    {
        private readonly ILogger<Worker> _logger;

        public Worker(ILogger<Worker> logger)
        {
            _logger = logger;
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);

                throw new ApplicationException("If an application occurs here, it will bubble up to Main()");

                await Task.Delay(1000, stoppingToken);
            }
        }
    }

1. Run the program, and when the exception occurs it does bubble up to Main():

info: WorkerExceptionTest.Worker[0]
      Worker running at: 09/17/2019 16:28:13 -05:00
Main method caught exception:  System.ApplicationException: If an application occurs here, it will bubble up to Main()
   at WorkerExceptionTest.Worker.ExecuteAsync(CancellationToken stoppingToken) in C:\spikes\WorkerExceptionTest\Worker.cs:line 26
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at WorkerExceptionTest.Program.Main(String[] args) in C:\spikes\WorkerExceptionTest\Program.cs:line 16

C:\spikes\WorkerExceptionTest\bin\Debug\netcoreapp3.0\WorkerExceptionTest.exe (process 20380) exited with code -1.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .

Expected behavior

I would expect that no matter WHEN the exception occurs (before or after await) it would bubble up to Main().

Additional context

This may (or may not) be related to #1836 ?

[gfoidl]

Don’t use negative ExitCodes, just use values in the range [0, 255]. So you’re code can run anywhere 😉

BTW: for logging use scopes, so you don’t have to repeat in the message where you are.

[shadow-cs]

Not sure if this is relevant or even whether we want to propagate exceptions to Main but it boils down to:

https://github.com/aspnet/Extensions/blob/aa7fa91cfc8f6ff078b020a428bcad71ae7a32ab/src/Hosting/Abstractions/src/BackgroundService.cs#L65

Handling cancellation of the StopAsync has been widely discussed in #2017, it was suggested that Microsoft.VisualStudio.Threading be used. Namely the WithCancellation, the method implementation (easier to understand than Microsoft.VisualStudio.Threading, but generally the same) is also suggested in the issue. It is important to note the two await calls that ensure the result of WhenAny is also awaited (which makes the exception pop-up).

IMHO the exception which happened during execution should eventually bubble up, the downside is that it may take a long time to do so since this happens during shutdown.