Instrumentation.AspNetCore for gRPC services omits ALL rpc.* attributes under certain conditions

Bug Report

NuGet package(s):

• OpenTelemetry.Instrumentation.AspNetCore 1.0.0-rc.1 NuGet package (and tip of tree)

Runtimes:

• netcoreapp3.1

Symptom

rpc.* attributes required for RPC Spans are not added to the created gRPC server Span under particular circumstances. All I see are http.* attributes. This happens when using a composite text map propagator made up of B3 and W3C in that order (B3 takes precedence if there). We have nginx and linkerd services sitting in front of these netcore gRPC services which understand and augment traces via the B3 headers before passing through to the gRPC service. Existing traceparent headers are treated as opaque data and forwarded unmodified as intended. The gRPC instrumentation correctly extracts the context from the B3 headers but still fails to output the rpc attributes.

This is happening because a new Activity is created in this block: https://github.com/open-telemetry/opentelemetry-dotnet/blob/ccc191f0c61a5a16c5a4cf577f82a7ac1c5dbd41/src/OpenTelemetry.Instrumentation.AspNetCore/Implementation/HttpInListener.cs#L86 with a name (ActivityCreatedByHttpInListener) that doesn’t satisfy a check in the dotnet-grpc code which is responsible for adding a tag on the Activity called grpc.method. It looks specifically for an Activity named Microsoft.AspNetCore.Hosting.HttpRequestIn. See https://github.com/grpc/grpc-dotnet/blob/76ae64d3b0e28772bc4d97c0d8a146b3609b465d/src/Grpc.AspNetCore.Server/Internal/HttpContextServerCallContext.cs#L410

The otel instrumentation looks for the presence of that grpc.method tag before adding the rpc.* tags.

What is the expected behavior?

rpc.* attributes are present according to https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/rpc.md

Repro

Add a composite propagator to the Examples.Grpc.Service. See the last commit on my test branch: https://github.com/pcwiese/opentelemetry-dotnet/commit/6cc33a137fad61ff852c9dabc47cf39a2722c410

Start the Examples.Grpc.Service and hit it with a grpcurl request similar to this one:

grpcurl.exe -proto "Z:\src\opentelemetry-dotnet\examples\GrpcService\Protos\greet.proto" -import-path "Z:\src\opentelemetry-dotnet\examples\GrpcService\Protos" -H "traceparent: 00-120dc44db5b736468afb112197b0dbd3-5dfbdf27ec544544-01" -H "x-b3-traceid: 120dc44db5b736468afb112197b0dbd3" -H "x-b3-spanid: b0966f651b9e0126" -H "x-b3-sampled: 1" localhost:44335 greet.Greeter/SayHello

The trace ids match between the traceparent header and B3 headers but the span ids do not. When you make the request you should see the rpc.* attributes outputted from the service but they aren’t there.

Possible fix

Given that the Grpc.AspNetCore.Server NuGet package requires the Activity to be a certain name, we could simply create the Activity as we do now in the instrumentation library, but use the incoming Activity’s OperationName instead of ActivityCreatedByHttpInListener. I tried this out locally and it does work but I’m sure what the implication of that is elsewhere.

[alanwest]

I think I like option 3 the best because it continues to rely on the same side-effect of the library adding tags to a specific Activity.

I’m intrigued by option 1, because it seems to move away from relying on the library’s side-effect. Though, I haven’t studied this enough to know if relying on the request’s ContentType and Path will be reliable. Also, as you’ve noted, still not sure how to handle status code.

Side note, I’m a bit embarrassed to say, but I’m still a little unclear on why we create sibling Activity in some circumstances…

[pcwiese]

I think I like option 3 the best because it continues to rely on the same side-effect of the library adding tags to a specific Activity.

I’m intrigued by option 1, because it seems to move away from relying on the library’s side-effect. Though, I haven’t studied this enough to know if relying on the request’s ContentType and Path will be reliable. Also, as you’ve noted, still not sure how to handle status code.

Side note, I’m a bit embarrassed to say, but I’m still a little unclear on why we create sibling Activity in some circumstances…

I would agree that Option 3 also sounds the most appealing and it is the path I took to patch a private version of this assembly that our team is using.