Test Collection Cleanup Failure

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

[Collection("Tests Collection")]
public class AuthorizeEndpointIntegrationTests
{
    private readonly HttpClient _client;
    public AuthorizeEndpointIntegrationTests(CustomWebApplicationFactory<Program> factory) => _client = factory.CreateClient();
}

public class CustomWebApplicationFactory<TStartup> : WebApplicationFactory<TStartup>, IDisposable where TStartup : class
{
    public void Dispose()
    {
        using (var scope = Services.CreateScope())
        {
            var scopedServices = scope.ServiceProvider;
            var myDb = scopedServices.GetRequiredService<MyDbContext>();
            SeedData.CleanUpTestData(myDb);
        }
        base.Dispose(); // XXX Throws
        GC.SuppressFinalize(this);
    }
}

Sometime it passes, sometime it fails.

Expected Behavior

No exception

Steps To Reproduce

No response

Exceptions (if any)

      Message: 
        [Test Collection Cleanup Failure (Tests Collection)]: System.AggregateException : One or more hosted services failed to stop. (Object reference not set to an instance of an object.) (Not started. Call Start first.)
        ---- System.NullReferenceException : Object reference not set to an instance of an object.
        ---- System.InvalidOperationException : Not started. Call Start first.

      Stack Trace: 
        Host.StopAsync(CancellationToken cancellationToken)
        WebApplicationFactory`1.DisposeAsync()
        WebApplicationFactory`1.Dispose(Boolean disposing)
        WebApplicationFactory`1.Dispose()
        CustomWebApplicationFactory`1.Dispose() line 77
        ----- Inner Stack Trace #1 (System.NullReferenceException) -----
        ServerSideSessionCleanupHost.StopAsync(CancellationToken cancellationToken) line 70
        Host.StopAsync(CancellationToken cancellationToken)
        ----- Inner Stack Trace #2 (System.InvalidOperationException) -----
        TokenCleanupHost.StopAsync(CancellationToken cancellationToken) line 67
        Host.StopAsync(CancellationToken cancellationToken)

.NET Version

7.0.304

Anything else?

.NET SDK: Version: 7.0.304 Commit: 7e794e2806

Runtime Environment: OS Name: Windows OS Version: 10.0.22621 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\7.0.304\

Host: Version: 7.0.7 Architecture: x64 Commit: 5b20af47d9

.NET SDKs installed: 6.0.202 [C:\Program Files\dotnet\sdk] 6.0.401 [C:\Program Files\dotnet\sdk] 7.0.100 [C:\Program Files\dotnet\sdk] 7.0.304 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 6.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.18 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 7.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found: x86 [C:\Program Files (x86)\dotnet] registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables: Not set

global.json file: Not found

Learn more: https://aka.ms/dotnet/info

Download .NET: https://aka.ms/dotnet/download

[martincostello]

Using DisposeAsync() would just be an improvement if you wanted to use async code in your clean up method, but that’s a different topic. IAsyncLifetime was just a suggestion of a refactor to see if it made any difference. Apparently it doesn’t, so we’re back to the original issue of why are the exceptions being thrown. Let’s set aside those parts of my suggestion.

I only want to clean up and dispose after all the tests in the “Test Collection” completes.

I understand what you’re trying to do, but the call to base.Dispose() is throwing exceptions from IdentityServer related code. Without a repro, trying to find out why that is happening is going to be difficult. Finding the root cause will also clarify whether the issue is with your application code, ASP.NET Core or IdentityServer (or something different entirely).

[msftbot[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.