TraceId is not propageted when using dapr grpc client's InvokeMethodAsync method

It seems that the W3C trace headers in dapr grpc calls are not propagated correctly.

I have a simple API Gateway where the W3C Activity format is specified like:

Activity.DefaultIdFormat = ActivityIdFormat.W3C;

The API Gateway is executing subsequent calls to several microservices, either via dapr http/grpc or vanilla grpc with generated proto files from .net (no dapr is involved).

Expected Behavior

If I execute a call via Rest to the ApiGateway–>Dapr–>REST-WeatherMicroservice as shown in the below log: The Trace Id is propagated correctly and the request correlation can be made.

If I execute a call via Rest to the ApiGateway–>Vanilla Grpc–>Grpc-WeatherMicroservice as shown in the below log: The Trace Id is propagated correctly and the request correlation can be made.

Actual Behavior

If I execute a call via Rest to the ApiGateway–>Dapr–>Grpc-WeatherMicroservice as shown in the below log: The Trace Id is broken and Dapr generates a new TraceId which breaks the request correlation.

Steps to Reproduce the Problem

See example project: https://gitlab.com/quickstar1/microservice-prototype

Release Note

RELEASE NOTE:

[rynowak]

thanks @quickstar - I was able to reproduce this issue using the Dapr v1.0.0-rc3 runtime and v1.0.0-rc2 SDK. There are some changes to how these APIs behave in RC3 but nothing that should impact tracing.

What’s happening here is that the grpc-trace-bin metadata is being passed (but not traceparent) when Dapr invokes a gRPC server. ASP.NET Core has no understanding of grpc-trace-bin, and doesn’t populate an activity based on it.

Here’s basically what happens:

browser ->
API Gateway (T1) // The api gateway doesn't see a value for traceparent and so it generates a new trace ID
API Gateway Dapr Sidecar (T1) // Sees the traceparent value and forwards it
Weather Dapr Sidecar (T1) // Sees the traceparent value and forwards it as grpc-trace-bin
Weather (T2) // Doesn't see a value for traceparent so it generates a new trace ID

[pkedy]

Hey @quickstar, the link to your example project is giving me a 404. Can you double check the permissions on it? I’d like to use it to validate the potential fix I have. Thanks!