Http filter / enrich delegates are not invoked when service makes Grpc requests

Bug Report

OpenTelemetry 1.2.0-beta2.1 OpenTelemetry.Instrumentation.Http 1.2.0-beta2.1

Tested on .NET5.0, looks like it will affect all runtimes.

Symptom

If a service with http instrumentation makes a Grpc request before a plain old Http request, the Filter and Enrich delegates within the HttpClientInstrumentation will not get invoked for any subsequent http requests.

This is particularly painful when dealing with Azure Isolated functions, which always makes a Grpc request on start-up, which due to this issue breaks all Http instrumentation. (You still get spans exported, you just can’t filter or enrich them).

What is the expected behavior?

Filter & Enrich methods should be invoked for all Http requests.

What is the actual behavior?

Filter & Enrich methods are not exported for any Http requests.

This is because:

The first time startRequestFetcher.TryFetch is invoked, innerFetcher is null, and so it calls PropertyFetch.FetcherForProperty which creates a TypedPropertyFetch

When the first request the application makes is a Grpc request, obj is of type Grpc.Shared.TelemetryHeaderHandler+ActivityStartData. This means the innerFetcher now always looks for Grpc ActivityStartData instances, and when any subsequent http request is invoked, TypedPropertyFetch.TryFetch fails because obj is now of type System.Net.Http.DiagnosticsHandler+ActivityStartData, so OnStartActivity exits prematurely.

Reproduce

Sample Repo - OtelTestHarness is an Azure isolated function which executes a Http request via a timer trigger every 15 seconds. The Filter and Enrich methods within Program.cs will not be executed for any of the http requests, as the isolated function SDK makes a Grpc request during start-up.

[CodeBlanch]

Thanks for the sample repo @BlakeWills! I opened a PR that I think will take care of the problem without users needing to add in AddGrpcClientInstrumentation to their tracer config.

[BlakeWills]

FYI a workaround is to set SuppressDownstreamInstrumentation:

using (Sdk.CreateTracerProviderBuilder()
	.AddSource("test-source")
	.AddGrpcClientInstrumentation(o =>
	{
		o.SuppressDownstreamInstrumentation = false;
	})