Async losing context

I am losing context in serilog when doing the following

    public async Task<byte[]> TestAsync(TestQuery query)
    {
        using (LogContext.PushProperty("CorrelationId", query.CorrelationId))
        {
            var builder = new Foo(this.logger);
            return await builder.DoSomething(query);
        }
    }

When I say lose context I mean that if DoSomething continues to log then the early messages have the correlation id and the later ones don’t.

If instead I do the following all is good - i.e. the correlation id appears on all downstream messages.

public async Task<byte[]> TestAsync(TestQuery query)
{
    var lgr = this.logger.ForContext("CorrelationId", query.CorrelationId);
    var builder = new Foo(lgr);
    return await builder.DoSomething(query);
}

[ffMathy]

Couldn’t we use this class to get around this problem?

https://msdn.microsoft.com/en-us/library/dn906268(v=vs.110).aspx

[jakubkwa]

Hello all,

I think we’ve run into an issue related to this one lately and would like to share some thoughts.

The problem

In our work environment we have a bunch of Web API applications. In one of them, a very simple one, we decided to do request authentication straight in the controller itself (not handler or anything like that) - well in a collaborator, which is called from the controller to be precise. We also have decorated this collaborator call with another one, which is responsible for adding an access token, upon which we authenticate, to the Serilog LogContext using the PushProperty method. We thought that using LogContext there would be nice, because we can add the value where we extract it and it will just be there if we log something, right? Well, not entirely 😃 We discovered the problem, when an exception was thrown somewhere in further Controller code (so after adding to the LogContext) - this exception should be handled by our custom ExceptionHandler implementation and logged in our custom ExceptionLogger (these are both pretty standard Web API services). So in ExceptionLogger we called the Serilog ILogger, but the LogEvent did not contain the access token added to the LogContext in the controller.

The investigation

Now, we were not entirely sure if this was Serilog issue, our issue, Web API issue, so we started to google around and peek in the source code of Serilog and Web API a bit. The fact is that Serilog documentation provides only usages of LogContext.PushProperty in a using clause, in which this should work with no problems. So maybe the problem is that we wanted to use it in a more sophisticated way - calling the PushProperty method in the controller stack and disposing until the request is disposed. However logging an exception in Web API ExceptionLogger with all the information gathered previously in Handlers or Controllers doesn’t seem like a very odd scenario, does it? 😃

So what we know after some investigation:

• Serilog LogContext uses CallContext LogicalSetData and LogicalGetData methods under the hood.
• CallContext, according to the documentation, is a “specialized collection object similar to a Thread Local Storage for method calls and provides data slots that are unique to each logical thread of execution. The slots are not shared across call contexts on other logical threads. Objects can be added to the CallContext as it travels down and back up the execution code path, and examined by various objects along the path.” And this is true for synchronous calls - as can be seen below in the reproduction section.
• For asynchronous calls using CallContext will be problematic - it looks like the values are copied over when invoking a method, so you can get, read and use values added in the specific class in its collaborators, but not the other way around - as also can be seen in the reproduction section.
• In a Web API case we have a lot of async/await calls under the hood, even if we don’t use them that much ourselves (so even if we have synchronous methods in controllers). This means, that CallContext in a scenario like that is not a way to go at all.

The reproduction

I’ve written a couple of tests using Serilog LogContext to showcase various scenarios, but this can be done easily the same with CallContext directly. Anyway, I’ve created a simple Serilog sink to actually capture the log event that was passed.

  public class LogEventCache
  {
    public LogEvent LogEvent { get; set; }
  }

  public class LogEventCacheSink : ILogEventSink
  {
    private readonly LogEventCache _cache;
    
    public LogEventCacheSink(LogEventCache cache)
    {
      _cache = cache;
    }

    public void Emit(LogEvent logEvent)
    {
      _cache.LogEvent = logEvent;
    }
  }

I’ve also created two classes only for testing purposes - the names are a bit Web API inspired 😃 So I have a Controller class with two methods - one adding a property to LogContext only, and one adding a property and actually logging. I also add the property to a collection, to make sure it won’t be disposed nor garbage collected.

  public class Controller
  {
    private readonly ILogger _logger;
    private readonly List<IDisposable> _properties = new List<IDisposable>();

    public Controller(ILogger logger)
    {
      _logger = logger;
    }

    public void AddToContext()
    {
      _properties.Add(LogContext.PushProperty("Controller", "Value"));
    }

    public void AddToContextAndLog()
    {
      _properties.Add(LogContext.PushProperty("Controller", "Value"));

      _logger.Information("Message");
    }
  }

And I have a similar Handler class, that can either only add a property (and then asks the Controller to log) or add a property and log (in this case it asks Controller only to add its own property).

  public class Handler
  {
    private readonly ILogger _logger;
    private readonly Controller _controller;
    private readonly List<IDisposable> _properties = new List<IDisposable>();

    public Handler(ILogger logger, Controller controller)
    {
      _logger = logger;
      _controller = controller;
    }

    public void AddToContextAndCallControllerWithLog()
    {
      _properties.Add(LogContext.PushProperty("Handler", "Value"));
      
      _controller.AddToContextAndLog();
    }

    public void AddToContextAndCallControllerAndLog()
    {
      _properties.Add(LogContext.PushProperty("Handler", "Value"));

      _controller.AddToContext();

      _logger.Information("Message");
    }
  }

Now this is how this behaves in a synchronous world:

  [TestFixture]
  public class SerilogLogContextUnitTests
  {
    private ILogger _logger;
    private LogEventCache _cache;
    private Handler _handler;
    private Controller _controller;

    [SetUp]
    public void SetUp()
    {
      _cache = new LogEventCache();

      _logger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .WriteTo.Sink(new LogEventCacheSink(_cache))
        .CreateLogger();

      _controller = new Controller(_logger);
      _handler = new Handler(_logger, _controller);
    }

    [Test]
    public void WhenValueIsAddedToTheContextInTheLoggingMethod_ShouldValueBeAccessibleInLogEvent()
    {
      _controller.AddToContextAndLog();

      AssertPropertyValue("Controller", "Value");
    }

    [Test]
    public void WhenValueIsAddedToTheContextInTheMethodCallingTheLoggingMethod_ShouldValueBeAccessibleInLogEvent()
    {
      _handler.AddToContextAndCallControllerWithLog();

      AssertPropertyValue("Handler", "Value");
    }

    [Test]
    public void WhenValueIsAddedToTheContextInTheMethodCalledFromTheLoggingMethod_ShouldValueBeAccessibleInLogEvent()
    {
      _handler.AddToContextAndCallControllerAndLog();

      AssertPropertyValue("Controller", "Value");
    }

    private void AssertPropertyValue(string propertyName, string propertyValue)
    {
      Assert.AreEqual((_cache.LogEvent.Properties[propertyName] as ScalarValue).Value, propertyValue);
    }
  }

All this tests are passing. So:

• If I add a property to LogContext in the Controller and log there, it will be okay.
• If I add a property to LogContext in the Handler and call the Controller to log, it will be okay.
• If I add a property to LogContext in the Controller and log in the Handler after that, it will be okay.

Now this looks pretty straightforward, however it changes when we get to introduce asynchronous stuff. So my Controller and Handler now look like this:

  public class Handler
  {
    private readonly ILogger _logger;
    private readonly Controller _controller;
    private readonly List<IDisposable> _properties = new List<IDisposable>();

    public Handler(ILogger logger, Controller controller)
    {
      _logger = logger;
      _controller = controller;
    }

    public async Task AddToContextAndCallControllerWithLogAsync()
    {
      _properties.Add(LogContext.PushProperty("Handler", "Value"));

      await _controller.AddToContextAndLogAsync();
    }

    public async Task AddToContextAndCallControllerAndLogAsync()
    {
      _properties.Add(LogContext.PushProperty("Handler", "Value"));

      await _controller.AddToContextAsync();

      _logger.Information("Message");
    }
  }

  public class Controller
  {
    private readonly ILogger _logger;
    private readonly List<IDisposable> _properties = new List<IDisposable>();

    public Controller(ILogger logger)
    {
      _logger = logger;
    }

    public async Task AddToContextAsync()
    {
      _properties.Add(LogContext.PushProperty("Controller", "Value"));

      await Task.Delay(10);
    }

    public async Task AddToContextAndLogAsync()
    {
      _properties.Add(LogContext.PushProperty("Controller", "Value"));

      await Task.Delay(10);

      _logger.Information("Message");
    }
  }

And the tests:

  [TestFixture]
  public class SerilogLogContextUnitTests
  {
    private ILogger _logger;
    private LogEventCache _cache;
    private Handler _handler;
    private Controller _controller;

    [SetUp]
    public void SetUp()
    {
      _cache = new LogEventCache();

      _logger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .WriteTo.Sink(new LogEventCacheSink(_cache))
        .CreateLogger();

      _controller = new Controller(_logger);
      _handler = new Handler(_logger, _controller);
    }

    [Test]
    public void WhenValueIsAddedToTheContextInTheLoggingMethod_ShouldValueBeAccessibleInLogEvent()
    {
      _controller.AddToContextAndLogAsync().Wait();

      AssertPropertyValue("Controller", "Value");
    }

    [Test]
    public void WhenValueIsAddedToTheContextInTheMethodCallingTheLoggingMethod_ShouldValueBeAccessibleInLogEvent()
    {
      _handler.AddToContextAndCallControllerWithLogAsync().Wait();

      AssertPropertyValue("Handler", "Value");
    }

    [Test]
    public void WhenValueIsAddedToTheContextInTheMethodCalledFromTheLoggingMethod_ShouldValueBeAccessibleInLogEvent()
    {
      _handler.AddToContextAndCallControllerAndLogAsync().Wait();

      AssertPropertyValue("Controller", "Value");
    }

    private void AssertPropertyValue(string propertyName, string propertyValue)
    {
      Assert.AreEqual((_cache.LogEvent.Properties[propertyName] as ScalarValue).Value, propertyValue);
    }
  }

The output of this tests is that the third one is failing. So:

• If I add a property to LogContext in the Controller and log there, it will be okay. This shows, that the CallContext is preserved after the awaited call.
• If I add a property to LogContext in the Handler and call the Controller to log, it will be okay. This shows, that the CallContext is copied over to the next method call, just as in synchronous scenario.
• If I add a property to LogContext in the Controller and log in the Handler after that, it will fail. This shows, that the CallContext is not copied over on the way back - in the Handler you get just those values, which you had already before the await (something similar can be found in async/await best practices https://msdn.microsoft.com/en-us/magazine/jj991977.aspx - “by default, when an incomplete Task is awaited, the current ‘context’ is captured and used to resume the method when the Task completes.”).

The solution or workaround

Not sure if what we did for now is a solution or workaround, but we decided to basically use the LogContext.PushProperty method as it is in the documentation. So we limit ourselves not to add values to LogContext anywhere in the app, but just before the ILogger call. This guarantees, that the values will be there. Of course in our scenario this means, that we need to provide the access token somehow to the class responsible for making the logging call, but this is something we can do in various ways, all of them probably connected to using HttpContext.Current - we can either extract it directly from the HttpContext.Current.Request headers, or still have a decorator extracting the token and putting it in HttpContext.Current.Items in one place and then take it from there just before the logging call, to put it in LogContext.

This works for us right now, and I’m not sure if anything can be done on Serilog side to avoid problems like that - that’s probably a question for @nblumhardt 😃 In this very scenario using HttpContext instead of CallContext would work, but that’s probably not something you’d like to include in the codebase just like that. But maybe in a separate package for Web? Or maybe give a chance to choose somehow in the configuration the storage for LogContext properties? Thinking loudly here, but with Web API being still relatively popular, and used along with a custom global ExceptionHandler and/or ExceptionLogger it might be a cause of problems for at least some people, like it was for us 😃

PS We’ve also investigated how log4net handles stuff like that. There are 3 types of contexts you can add properties to in log4net - none of them would support a scenario like we had. There is a GlobalContext which is preserved all the time, ThreadLogicalContext which also relies on CallContext so will behave in a similar way like Serilog LogContext, and ThreadContext, which is connected to current managed thread, so will have it’s own problems in ASP .NET application.