[EVE-33] Race condition in ESDB subscriptions with random NullReferenceException

Describe the bug Integration tests for an app using Eventuous 0.13.1 fail randomly when running in parallel.

Error is

System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
     at Eventuous.EventStore.Subscriptions.EventStoreCatchUpSubscriptionBase`1..ctor(EventStoreClient eventStoreClient, T options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory)
   at Eventuous.EventStore.Subscriptions.AllStreamSubscription..ctor(EventStoreClient eventStoreClient, AllStreamSubscriptionOptions options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory)
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Sometimes it occurs on some test, sometimes on another, sometimes it doesn’t.

All the tests (i.e: [Fact]) instantiate a TestServer because the test classes inherit from base class

public abstract class FunctionalTest
{
    protected HttpClient HttpClient { get; }
    
    protected FunctionalTest()
    {
        var webApplicationFactory =
            new WebApplicationFactory<Program>()
                .WithWebHostBuilder(webHostBuilder => webHostBuilder.UseEnvironment("Development"));
        
        HttpClient = webApplicationFactory.CreateClient();
    }
}

Could not yet reproduce in a GitLab runner.

To Reproduce

1. Clone this repository https://gitlab.com/sunnyatticsoftware/sample/eventuous-modular-race-condition-tests
2. Run ESDB and MongoDb in local

   docker run \
      --name esdb \
      -p 2113:2113 \
      -p 1113:1113 \
      eventstore/eventstore:22.10.0-buster-slim \
      --insecure \
      --enable-atom-pub-over-http \
      --mem-db=True
   

and

docker run \
   --name mongo \
   --mount type=tmpfs,destination=/data/db \
   -e MONGO_INITDB_ROOT_USERNAME=root \
   -e MONGO_INITDB_ROOT_PASSWORD=dummy \
   -p 27017:27017 \
   mongo:4.0

NOTE: This requires some entries in /etc/hosts.

# Local servers
127.0.0.1   eventstoredb
127.0.0.1   mongodb

If pointing to localhost, edit appsettings.json and replace eventstoredb and mongodb with localhost. 3. Run dotnet test

Expected behavior Tests to pass always

Actual behavior Tests randomly fail when running in parallel (by default in xUnit).

Sample:

dotnet test
  Determining projects to restore...
  All projects are up-to-date for restore.
  Modular.Clients -> /media/diegosasw/data/src/sandbox/eventuous-modular-race-condition-tests/src/client/Modular.Clients/bin/Debug/net7.0/Modular.Clients.dll
  Modular.ClientsInfo -> /media/diegosasw/data/src/sandbox/eventuous-modular-race-condition-tests/src/client-info/Modular.ClientsInfo/bin/Debug/net7.0/Modular.ClientsInfo.dll
  Modular.Api -> /media/diegosasw/data/src/sandbox/eventuous-modular-race-condition-tests/src/Modular.Api/bin/Debug/net7.0/Modular.Api.dll
  Modular.Api.FunctionalTests -> /media/diegosasw/data/src/sandbox/eventuous-modular-race-condition-tests/test/Modular.Api.FunctionalTests/bin/Debug/net7.0/Modular.Api.FunctionalTests.dll
Test run for /media/diegosasw/data/src/sandbox/eventuous-modular-race-condition-tests/test/Modular.Api.FunctionalTests/bin/Debug/net7.0/Modular.Api.FunctionalTests.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:01.41]     Modular.Api.FunctionalTests.AnotherTests.Three [FAIL]
[13:58:40 INF] Call failed with gRPC error status. Status code: 'Cancelled', Message: 'Call canceled by the client.'. <s:Grpc.Net.Client.Internal.GrpcCall>
[13:58:40 DBG] Starting subscription projections <s:Eventuous.Subscriptions.SubscriptionHostedService>
[13:58:40 DBG] Type ClientCreated registered as V1.ClientCreated <s:eventuous>
[13:58:40 DBG] Type ClientCreated registered as V1.ClientCreated <s:eventuous>
[13:58:40 DBG] Type ClientAdminCredentialsUpdated registered as V1.ClientAdminCredentialsUpdated <s:eventuous>
[13:58:40 DBG] Type ClientAdminCredentialsUpdated registered as V1.ClientAdminCredentialsUpdated <s:eventuous>
[13:58:40 DBG] Starting subscription projections <s:Eventuous.Subscriptions.SubscriptionHostedService>
[13:58:41 INF] [projections] Loaded checkpoint projections from MongoCheckpointStore: Checkpoint { Id = projections, Position = 733 } <s:Eventuous.Subscription>
[13:58:41 INF] [projections] Loaded checkpoint projections from MongoCheckpointStore: Checkpoint { Id = projections, Position = 733 } <s:Eventuous.Subscription>
[13:58:41 INF] [projections] Started <s:Eventuous.Subscription>
[13:58:41 INF] [projections] Started <s:Eventuous.Subscription>
[13:58:41 INF] Started subscription projections <s:Eventuous.Subscriptions.SubscriptionHostedService>
[13:58:41 INF] Started subscription projections <s:Eventuous.Subscriptions.SubscriptionHostedService>
[13:58:41 INF] [projections] Stored checkpoint projections in MongoCheckpointStore: Checkpoint { Id = projections, Position = 733 } <s:Eventuous.Subscription>
[13:58:41 INF] [projections] Stored checkpoint projections in MongoCheckpointStore: Checkpoint { Id = projections, Position = 733 } <s:Eventuous.Subscription>
[13:58:41 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature Microsoft.AspNetCore.Mvc.IActionResult Get() on controller Modular.Api.Controllers.HealthController (Modular.Api). <s:Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker>
[13:58:41 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature Microsoft.AspNetCore.Mvc.IActionResult Get() on controller Modular.Api.Controllers.HealthController (Modular.Api). <s:Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker>
[13:58:41 INF] Executing OkObjectResult, writing value of type 'System.String'. <s:Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor>
[13:58:41 INF] Executing OkObjectResult, writing value of type 'System.String'. <s:Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor>
[13:58:41 INF] Executed action Modular.Api.Controllers.HealthController.Get (Modular.Api) in 11.2942ms <s:Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker>
[13:58:41 INF] Executed action Modular.Api.Controllers.HealthController.Get (Modular.Api) in 11.2955ms <s:Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker>
[13:58:41 INF] HTTP GET /health responded 200 in 36.7654 ms <s:Serilog.AspNetCore.RequestLoggingMiddleware>
[13:58:41 INF] HTTP GET /health responded 200 in 36.8072 ms <s:Serilog.AspNetCore.RequestLoggingMiddleware>
[13:58:41 INF] HTTP GET /health responded 200 in 69.0459 ms <s:Serilog.AspNetCore.RequestLoggingMiddleware>
[13:58:41 INF] HTTP GET /health responded 200 in 69.0513 ms <s:Serilog.AspNetCore.RequestLoggingMiddleware>
[13:58:41 DBG] Type ClientCreated registered as V1.ClientCreated <s:eventuous>
[13:58:41 DBG] Type ClientCreated registered as V1.ClientCreated <s:eventuous>
[13:58:41 DBG] Type ClientAdminCredentialsUpdated registered as V1.ClientAdminCredentialsUpdated <s:eventuous>
[13:58:41 DBG] Type ClientAdminCredentialsUpdated registered as V1.ClientAdminCredentialsUpdated <s:eventuous>
[13:58:41 DBG] Type ClientCreated registered as V1.ClientCreated <s:eventuous>
[13:58:41 DBG] Type ClientCreated registered as V1.ClientCreated <s:eventuous>
[13:58:41 DBG] Type ClientAdminCredentialsUpdated registered as V1.ClientAdminCredentialsUpdated <s:eventuous>
[13:58:41 DBG] Type ClientAdminCredentialsUpdated registered as V1.ClientAdminCredentialsUpdated <s:eventuous>
[13:58:41 DBG] Starting subscription projections <s:Eventuous.Subscriptions.SubscriptionHostedService>
[13:58:41 DBG] Starting subscription projections <s:Eventuous.Subscriptions.SubscriptionHostedService>
[13:58:41 INF] [projections] Loaded checkpoint projections from MongoCheckpointStore: Checkpoint { Id = projections, Position = 733 } <s:Eventuous.Subscription>
[13:58:41 INF] [projections] Loaded checkpoint projections from MongoCheckpointStore: Checkpoint { Id = projections, Position = 733 } <s:Eventuous.Subscription>
[13:58:41 INF] [projections] Started <s:Eventuous.Subscription>
[13:58:41 INF] Started subscription projections <s:Eventuous.Subscriptions.SubscriptionHostedService>
[13:58:41 INF] [projections] Started <s:Eventuous.Subscription>
[13:58:41 INF] Started subscription projections <s:Eventuous.Subscriptions.SubscriptionHostedService>
[13:58:41 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature Microsoft.AspNetCore.Mvc.IActionResult Get() on controller Modular.Api.Controllers.HealthController (Modular.Api). <s:Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker>
[13:58:41 INF] Executing OkObjectResult, writing value of type 'System.String'. <s:Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor>
[13:58:41 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature Microsoft.AspNetCore.Mvc.IActionResult Get() on controller Modular.Api.Controllers.HealthController (Modular.Api). <s:Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker>
[13:58:41 INF] Executed action Modular.Api.Controllers.HealthController.Get (Modular.Api) in 1.0113ms <s:Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker>
[13:58:41 INF] HTTP GET /health responded 200 in 4.1970 ms <s:Serilog.AspNetCore.RequestLoggingMiddleware>
[13:58:41 INF] HTTP GET /health responded 200 in 6.7394 ms <s:Serilog.AspNetCore.RequestLoggingMiddleware>
[13:58:41 INF] Executing OkObjectResult, writing value of type 'System.String'. <s:Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor>
[13:58:41 INF] Executed action Modular.Api.Controllers.HealthController.Get (Modular.Api) in 0.2744ms <s:Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker>
[13:58:41 INF] HTTP GET /health responded 200 in 1.8965 ms <s:Serilog.AspNetCore.RequestLoggingMiddleware>
[13:58:41 INF] HTTP GET /health responded 200 in 3.5090 ms <s:Serilog.AspNetCore.RequestLoggingMiddleware>
[13:58:41 INF] [projections] Stored checkpoint projections in MongoCheckpointStore: Checkpoint { Id = projections, Position = 733 } <s:Eventuous.Subscription>
[13:58:41 INF] [projections] Stored checkpoint projections in MongoCheckpointStore: Checkpoint { Id = projections, Position = 733 } <s:Eventuous.Subscription>
  Failed Modular.Api.FunctionalTests.AnotherTests.Three [1 ms]
  Error Message:
   System.NullReferenceException : Object reference not set to an instance of an object.
  Stack Trace:
     at Eventuous.EventStore.Subscriptions.EventStoreCatchUpSubscriptionBase`1..ctor(EventStoreClient eventStoreClient, T options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory)
   at Eventuous.EventStore.Subscriptions.AllStreamSubscription..ctor(EventStoreClient eventStoreClient, AllStreamSubscriptionOptions options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory)
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Failed!  - Failed:     1, Passed:     4, Skipped:     0, Total:     5, Duration: 520 ms - Modular.Api.FunctionalTests.dll (net7.0)

_EVE-33

[alexeyzimarev]

The last alpha will double-check that the options argument is not null, just to confirm (or not) that it is an issue.

[diegosasw]

Tried with 0.14.1-alpha.0.7. The issue must come from somewhere else but it manifests there.

Running tests separately or with parallelization disabled with

[assembly: CollectionBehavior(CollectionBehavior.CollectionPerClass, DisableTestParallelization = true)]

avoids the problem, but unfortunately when running in parallel it still fails randomly with this exception.

System.ArgumentNullException: Value cannot be null. (Parameter 'options')

System.ArgumentNullException
Value cannot be null. (Parameter 'options')
   at Eventuous.EventStore.Subscriptions.EventStoreCatchUpSubscriptionBase`1..ctor(EventStoreClient eventStoreClient, T options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory)
   at Eventuous.EventStore.Subscriptions.AllStreamSubscription..ctor(EventStoreClient eventStoreClient, AllStreamSubscriptionOptions options, ICheckpointStore checkpointStore, ConsumePipe consumePipe, ILoggerFactory loggerFactory)
   at InvokeStub_AllStreamSubscription..ctor(Object, Object, IntPtr*)
   at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

EDIT: Alejandro-SB changes solves the mysterious issue!