AddAspNetCoreInstrumentation does not output traces when behind Ocelot proxy

Bug Report

List of all OpenTelemetry NuGet packages:

• OpenTelemetry.Exporter.Console 1.4.0-rc.1
• OpenTelemetry.Exporter.OpenTelemetryProtocol 1.4.0-rc.1
• OpenTelemetry.Exporter.Prometheus.AspNetCore 1.4.0-rc.1
• OpenTelemetry.Extensions.Hosting 1.4.0-rc.1
• OpenTelemetry.Instrumentation.AspNetCore 1.0.0-rc9.10
• OpenTelemetry.Instrumentation.Http 1.0.0-rc9.10
• OpenTelemetry.Instrumentation.SqlClient 1.0.0-rc9.10

Runtime version:

• net6.0

Symptom

Using Ocelot as an API gateway to downstream services, and using AddAspNetCoreInstrumentation in the downstream services, the HTTP requests going to those downstream services do not produce trace output. If we call the downstream services directly, traces are logged. (Tested with both console exporter and OTLP exporter.)

What is the expected behavior?

Traces should be logged also when using an API gateway in front.

What is the actual behavior?

Traces are only logged for non-forwarded requests.

Reproduce

Using the following set-up code for OpenTelemetry in the downstream service:

services.AddOpenTelemetry()
                .WithMetrics(
                    (builder) =>
                    {
                        builder.AddAspNetCoreInstrumentation();
                        builder.AddHttpClientInstrumentation();
                        builder.AddPrometheusExporter();
                    }
                )
                .WithTracing(
                    (builder) =>
                    {
                        builder.AddHttpClientInstrumentation();
                        builder.AddAspNetCoreInstrumentation();
                        builder.AddSqlClientInstrumentation(
                            options => options.SetDbStatementForText = true);
                        builder.AddHotChocolateInstrumentation();
                        builder.AddOtlpExporter(options =>
                            configuration.GetSection("Tracing:OpenTelemetry").Bind(options)
                        );
                    }
                )
                .StartWithHost();

No OpenTelemetry is added to the API gateway service.

Additional Context

Using the following Ocelot configuration:

{
  "Routes": [
    {
      "DownstreamPathTemplate": "/api/{everything}",
      "DownstreamScheme": "http",
      "DownstreamHostAndPorts": [
        {
          "Host": "api",
          "Port": 80
        }
      ],
      "UpstreamPathTemplate": "/{everything}",
      "UpstreamHttpMethod": [ "Get", "Post", "Put", "Delete" ],
      "SwaggerKey": "MyService"
    }
  ]
}

Where api is the name of the downstream service running with docker-compose.

[vishweshbankwar]

Does it repro if you use AlwaysOnSampler ?

[aristotelos]

OK, I think I get it. So any ASP.NET Core-based API gateway will probably create parent spans for each request, and this will stop my downstream service spans from being sampled, because the default sampling strategy is to skip traces that have parents that were not sampled. In my case, I will anyways add OpenTelemetry instrumentation and an OpenTelemetry exporter to the API gateway, which solves the problem for me.