Unhandled exception in IHostedService does not stop the host

Describe the bug

If an unhandled exception occurs inside an implementation of a Microsoft.Extensions.Hosting.BackgroundService I would expect it to exit the application.

Using the generic host builder in a .net core 2.1 console application, I register hosted services that override BackgroundService. Each service returns a task that performs a long running operation. If an unhandled exception is thrown inside the task the host is not exited but continues to run until a Ctrl^C is triggered.

To Reproduce

Steps to reproduce the behavior:

1. Using version ‘2.2.0’ of package ‘Microsoft.Extensions.Hosting’
2. Run this code

internal class Program
    {
        public static async Task Main(string[] args)
        {
            var host = new HostBuilder()
                .ConfigureServices(services => services.AddHostedService<TestService>())
                .Build();

            await host.RunAsync();
        }
    }

    public class TestService : BackgroundService
    {
        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            try
            {
                await Task.Run(() =>
                    {
                        try
                        {
                            Thread.Sleep(5000);                        
                            throw new SystemException("Something went wrong!");
                        }
                        catch (Exception e)
                        {
                            Console.WriteLine(e);
                            throw;
                        }
                    }, stoppingToken);
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
                throw;
            }
        }
    }

1. See error The exception is correctly handled and rethrown but is consumed by the host.

Application started. Press Ctrl+C to shut down.
System.SystemException: Something went wrong!
   at TestHostBuilder.TestService.<>c.<ExecuteAsync>b__0_0() in Program.cs:line 31
Hosting environment: Production
Content root path: TestHostBuilder\bin\Debug\netcoreapp2.1\
System.SystemException: Something went wrong!
   at TestHostBuilder.TestService.<>c.<ExecuteAsync>b__0_0() in Program.cs:line 31
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at TestHostBuilder.TestService.ExecuteAsync(CancellationToken stoppingToken) in Program.cs:line 27

Expected behavior

I expected the host to exit on receiving the unhandled exception. See example below

Additional context

What I don’t understand is why when I run the following hosted service using “System.Reactive” Version=“4.1.3” the exception does bubble up to the host and the application does exit.

public class TestService2 : BackgroundService
        {
            protected override async Task ExecuteAsync(CancellationToken stoppingToken)
            {
                try
                {
                    await Task.Run(() =>
                    {
                        try
                        {
                            Observable.Interval(TimeSpan.FromSeconds(1))
                                .Subscribe(x =>
                                {
                                    if (x > 4)
                                    {
                                        throw new SystemException("Something went wrong!");
                                    }
                                });
                        }
                        catch (Exception e)
                        {
                            Console.WriteLine(e);
                            throw;
                        }
                    }, stoppingToken);
                }
                catch (Exception e)
                {
                    Console.WriteLine(e);
                    throw;
                }
            }
        }

Output

Application started. Press Ctrl+C to shut down.
Hosting environment: Production
Content root path: TestHostBuilder\bin\Debug\netcoreapp2.1\

Unhandled Exception: System.SystemException: Something went wrong!
   at TestHostBuilder.TestService.TestService2.<>c.<ExecuteAsync>b__0_1(Int64 x) in Program.cs:line 66
   at System.Reactive.AnonymousSafeObserver`1.OnNext(T value) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\AnonymousSafeObserver.cs:line 44
   at System.Reactive.Concurrency.Scheduler.<>c__67`1.<SchedulePeriodic>b__67_0(ValueTuple`2 t) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Concurrency\Scheduler.Services.Emulation.cs:line 79
   at System.Reactive.Concurrency.DefaultScheduler.PeriodicallyScheduledWorkItem`1.<>c.<Tick>b__5_0(PeriodicallyScheduledWorkItem`1 closureWorkItem) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Concurrency\DefaultScheduler.cs:line 127
   at System.Reactive.Concurrency.AsyncLock.Wait(Object state, Delegate delegate, Action`2 action) in D:\a\1\s\Rx.NET\Source\src\System.Reactive\Concurrency\AsyncLock.cs:line 93
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()

C:\Program Files\dotnet\dotnet.exe (process 38600) exited with code 0.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .

I believe the issue may be related to the implementation of BackgroundService

        /// <summary>
        /// Triggered when the application host is ready to start the service.
        /// </summary>
        /// <param name="cancellationToken">Indicates that the start process has been aborted.</param>
        public virtual Task StartAsync(CancellationToken cancellationToken)
        {
            // Store the task we're executing
            _executingTask = ExecuteAsync(_stoppingCts.Token);

            // If the task is completed then return it, this will bubble cancellation and failure to the caller
            if (_executingTask.IsCompleted)
            {
                return _executingTask;
            }

            // Otherwise it's running
            return Task.CompletedTask;
        }

If I update the code to always return the executing task and not Task.CompletedTaskthen it works as expected using both TestService and TestService2.

// <summary>
        /// Triggered when the application host is ready to start the service.
        /// </summary>
        /// <param name="cancellationToken">Indicates that the start process has been aborted.</param>
        public virtual Task StartAsync(CancellationToken cancellationToken)
        {
            // Store the task we're executing
            _executingTask = ExecuteAsync(_stoppingCts.Token);

            return _executingTask;
        }

[andriysavin]

I think the confusion comes from how BackgroundService API looks like. The protected override async Task ExecuteAsync(CancellationToken stoppingToken) looks similar to many other application models, so one implies there is some code which awaits it and thus does all what is expected from awaiting (crash the app with unhandled exception in particular). The method name only assures in this, while from the host perspective it essentially behaves as StartAsync. It becomes obvious how things really work when you look at BackgroundService source code, and the explanation provided in this thread sounds reasonable. But now you have to think how to implement all that error handling yourself (using IApplicationLifetime etc.), and do that every time you need to implement a new service (ok, another base class can be created for code sharing), or new app or project (ok, a shared project/package can be created with the base class).

Ideally, I would expect an out of the box implementation of that in the form of, again, another base class derived from BackgroundService (or any other approach, because adding IApplicationLifetime to the base class constructor will make the API a bit more complicated).

[jdwoolcock]

I’m running the application as a systemd service on Linux. There are many hosted services some of which run and then complete and some that run indefinitely or until the cancellation token is received.

If a fatal exception occurs that I can’t recover from I was hoping to catch it, log it and then re throw it causing the application to die and systemd to then restart the process.

On Thu, 13 Jun 2019, 21:11 David Fowler, [e-mail] wrote:

I think this is more about being able to detect those failures. We don’t log by default either. I’m not sure if a single bad service should kill the process?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aspnet/Extensions/issues/1836?email_source=notifications&email_token=AHGT2PBUHTCFP7CZJ7HQ33TP2KSXTA5CNFSM4HXXAEF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXU4TMI#issuecomment-501860785, or mute the thread https://github.com/notifications/unsubscribe-auth/AHGT2PCMJCQN7XDN7BPTCX3P2KSXTANCNFSM4HXXAEFQ .