Question: Scope in AMQP events

Please mark the type framework used:

• ASP.NET MVC
• ASP.NET Web API (OWIN)
• ASP.NET Core
• WPF
• WinForms
• Xamarin
• Other:

Please mark the type of the runtime used:

• .NET Framework
• Mono
• .NET Core

• Version: 5.0.6

Please mark the NuGet packages used:

• Sentry
• Sentry.Serilog
• Sentry.NLog
• Sentry.Log4Net
• Sentry.Extensions.Logging
• Sentry.AspNetCore

• Version: 3.4.0

We have a microservice architecture and are looking into using the new tracing feature of Sentry. In this context I was reworking the way we use Sentry scopes around AMQP events. Background: Consider the following architecture:

• an ASP:NET Core backend A that receives web requests and may publish an AMQP message on some of them
• an ASP.NET Core backend B that can receive web requests and also consumes AMQP messages
• a NodeJS Express backend C that can handle web requests

The flow would be, a web request comes to A which creates a AMQP message that is consumed by B. B needs some additional information from C to process the message which will be pulled via an internal HTTP request. I wanted to configure tracing for all involved backends so I read through the documentation and come up with the following.

In backend A I have something like (using RabbitMQ.Client):

IBasicProperties props = _amqpService.Channel.CreateBasicProperties();
props.CorrelationId = "<correlation-id>";
props.Type = "<message-type>";
ISpan span = SentrySdk.GetSpan();
if (span != null)
{
  var traceHeaderValue = span.GetTraceHeader().ToString();
  props.Headers.Add("sentry-trace", traceHeaderValue);
}

// ...
_amqpService.Channel.BasicPublish(settings.Exchange, settings.Route, false, props, body);

In backend B I have set up the listener like this:

private void AmqpMessageHandler(object sender, BasicDeliverEventArgs ea)
{
  SentryTraceHeader sentryTraceHeader = null;

  if (ea.BasicProperties.Headers != null && ea.BasicProperties.Headers.TryGetValue(
        "sentry-trace",
        out var sentryHeaderValue))
  {
    try
    {
      // header values are in byte[] form
      sentryTraceHeader = SentryTraceHeader.Parse(Encoding.UTF8.GetString((byte[])sentryHeaderValue));
    }
    catch (FormatException ex)
    {
      _logger.LogWarning(ex, "unable to parse Sentry trace header value: {Message}.", ex.Message);
      ex.Data["bad-sentry-header"] = sentryHeaderValue;
      SentrySdk.CaptureException(ex);
    }
  }

  ITransaction transaction = sentryTraceHeader == null
    ? SentrySdk.StartTransaction("<messagename>", ea.BasicProperties.Type)
    : SentrySdk.StartTransaction("<messagename>", ea.BasicProperties.Type, sentryTraceHeader);
  transaction.Description = "A description of what is done here.";

  using (LogContext.PushProperty("CorrelationId", ea.BasicProperties.CorrelationId, true))
  {
    SentrySdk.WithScope(async s =>
    {
      _correlationContextFactory.Create(ea.BasicProperties.CorrelationId, "X-Correlation-ID");

      s.Transaction = transaction; // needed for HTTPClient to retrieve Sentry trace header
      s.SetTag("task", ea.BasicProperties.Type);
      s.SetExtra("CorrelationId", ea.BasicProperties.CorrelationId);

      try
      {
        await HandleMessageTask(ea, transaction);

        _amqpService.Channel.BasicAck(ea.DeliveryTag, multiple: false);
        transaction.Finish(SpanStatus.Ok);
      }
      catch (Exception ex)
      {
        _logger.LogError("Failed to run worker {Ex}", ex);
        _amqpService.Channel.BasicNack(deliveryTag: ea.DeliveryTag, multiple: false, requeue: false);

        if (!transaction.IsFinished)
        {
          transaction.Finish(ex, SpanStatus.UnknownError);
        }
      }
    });
  }
}

During HandleMessageTask a DependencyInjection scope is created and from this a service is received that handles the HTTP request. The service gets an HTTPClientFactory injected and creates a client that is prepared for communicating with the other backend. The startup code for this looks like this:

services.AddHttpClient("<client-for-backend>", client =>
  {
    client.BaseAddress = new Uri("backend-url");
    client.DefaultRequestHeaders.Add("backend-origin", "<backend-b>");
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", "<some-token>");

    Sentry.ISpan span = SentrySdk.GetSpan();
    if (span != null)
    {
      client.DefaultRequestHeaders.Add("sentry-trace", span.GetTraceHeader().ToString());
    }
  })

Now this all seems to work fine, I can see the correct transactionId in backend C. But I still have some questions:

1. is using SentrySdk.WithScope correct or should I rather use SentrySdk.ConfigureScope(Async)? Or does Sentry already create a scope for me?
2. In the catch block of the AMQP message handler, should I call Sentry.CaptureException(ex); or is it enough that the exception is attached to the transaction.Finish call?
3. I have also configured a TracesSampler to filter out health check and metrics requests and set the general sampleRate to 0.5. I want all of the AMQP message transactions to be send, do I have to configure it in the TracesSampler or are manual transaction always send?

(Bonus question): I was trying to extract the part where the HTTP client is created in a company internal NuGet package since it could be of use in multiple projects. I am sturggling to write a unit test for it where SentrySdk.GetSpan() would return something. I tried to start a transaction in the unit test and create spans from it but to no avail. I also tried ConfigureScope() but it would work either. Am I missing something? Unit test looks like this:

ITransaction transaction = SentrySdk.StartTransaction("unittest", "ShouldSetSentryTraceHeader");
SentrySdk.ConfigureScope(s => s.Transaction = transaction);
ISpan span1 = transaction.StartChild("Configure HTTP Client DI");

var clientName = "test";
var services = new ServiceCollection();
services.MyAddHttpClientExtensionMethod(clientName, "http://localhost", "dev", "unittest");
ServiceProvider provider = services.BuildServiceProvider(new ServiceProviderOptions { ValidateOnBuild = true });
span1.Finish(SpanStatus.Ok);

IHttpClientFactory httpClientFactory = provider.GetRequiredService<IHttpClientFactory>();
ISpan span2 = transaction.StartChild("Create HTTP client with trace header");
HttpClient client = httpClientFactory.CreateClient(clientName);
span2.Finish(SpanStatus.Ok);

Assert.True(client.DefaultRequestHeaders.TryGetValues(
  "sentry-trace",
  out IEnumerable<string> testyHeaderValues));
Assert.Equal(transaction.GetTraceHeader().ToString(), testyHeaderValues.FirstOrDefault());

transaction.Finish(SpanStatus.Ok);

[Tyrrrz]

Could you point me to a documentation on how to achieve this? Would it be enough to filter for the transaction name in the sampler?

Yeah, filtering by transaction name is an option. Alternatively, you can use the custom sampling context to set arbitrary data on the transaction and then filter by that data in the sampler.

https://github.com/getsentry/sentry-dotnet/blob/a45e222d17b8897e64a0b4dc57c52f565f374030/src/Sentry/TransactionSamplingContext.cs#L15-L18

https://github.com/getsentry/sentry-dotnet/blob/a45e222d17b8897e64a0b4dc57c52f565f374030/src/Sentry/SentrySdk.cs#L324-L331

This uses a slightly different overload, so you’d need to adjust the code to something like this:

var transaction = SentrySdk.StartTransaction(
    new TransactionContext(name, operation, traceHeader),
    new Dictionary<string, object>>
    {
        ["custom-data-for-sampling"] = 1337
    }
);

I saw in another libary that adds Sentry information to something that I need to initialize the SDK for certain Sentry function to work. Right now I use a disabled key in one of the Sentry projects for a unit test (so at least Sentry will drop events coming form the unit test) but I would rather have not Sentry trying to send events during unit test without mocking the whole Sentry SDK. Maybe I should replace the HTTPClient Sentry uses?

Yeah, actually this could really well be the reason it doesn’t work. If no DSN is provided, Sentry will be initialized in disabled state.

Try setting the DSN and use SentryOptions.CreateHttpClientHandler to inject a custom (fake) handler:

https://github.com/getsentry/sentry-dotnet/blob/a45e222d17b8897e64a0b4dc57c52f565f374030/src/Sentry/SentryOptions.cs#L312-L315

I played a little bit around with the AMQP stuff today and come up with the question: How does Sentry know what scope to use? It seems I do not need to use WithScope but could also use ConfigureScope() but then I am not sure if Sentry would still use another scope for web requests in the same backend and also for other AMQP messages. Also would Sentry dispose the scope itself after the message handling is done? The whole action runs inside an event handler so I am not sure if I should dispose the scope myself inside the event handler - which is why I used WithScope in the first place.

So what I am trying to achieve is every web request and every AMQP message should have its own Sentry scope. If I understand the docs correctly web requests already run in their own Sentry scope - I am just unsure if using ConfigureScope() in an event handler could somehow affect a web request that happen to run at the same time.

If I understand correctly (I’m still not entirely sure how scopes work myself, @bruno-garcia is the expert there), a new scope is created on each incoming http request (in ASP.NET Core integration). The difference between WithScope(...) and ConfigureScope(...) is that the former creates a new scope while the latter mutates the existing one. In case for WithScope(...) the scope is dropped at the end of the block (of the inner lambda). There is also PushScope() that returns an IDisposable.

[Tyrrrz]

1. In the catch block of the AMQP message handler, should I call Sentry.CaptureException(ex); or is it enough that the exception is attached to the transaction.Finish call?

You’d still want to call Sentry.CaptureException(ex). Finishing a transaction with an exception makes it so that the transaction appears faulty and also adds a link to the event that contains the same exception (assuming it’s been captured).

1. I have also configured a TracesSampler to filter out health check and metrics requests and set the general sampleRate to 0.5. I want all of the AMQP message transactions to be send, do I have to configure it in the TracesSampler or are manual transaction always send?

Manual and autoinstrumented transactions go through the same sampling logic. That means if you start a transaction via SentrySdk.StartTransaction(...), it’s not guaranteed that it will actually be sent. So, in summary, you’d want to add the corresponding rule to your sampler.

By the way, Sentry.AspNetCore integration adds a filter for AddHttpClient(...) that inserts a custom handler which in turn injects the trace header from the current scope automatically. so you most likely don’t need to do it yourself.

(Bonus question): I was trying to extract the part where the HTTP client is created in a company internal NuGet package since it could be of use in multiple projects. I am sturggling to write a unit test for it where SentrySdk.GetSpan() would return something. I tried to start a transaction in the unit test and create spans from it but to no avail. I also tried ConfigureScope() but it would work either. Am I missing something? Unit test looks like this:

It should work. Which part of the test fails? The assert?