Question: How to properly intercept response body on Enrich method

Question

I’ve been digging into a way to intercept the response body using the Enrich method available in the AddAspNetCoreInstrumentation hook.

The thing is that I haven’t found a way to properly get the actual response body into a string version or even an object, the HttpResponse object is available at the OnStopActivity event name but, turns out, the body stream is kind not ready to be read at that point. Would like to know if someone has already been able to accomplish that or if there’s even a way of accomplishing it.

Quick code example:

services.AddOpenTelemetryTracing((builder) =>
{
    builder
    .AddAspNetCoreInstrumentation((options) => options.Enrich = async (activity, eventName, rawObject) =>
    {
        if (eventName.Equals("OnStartActivity"))
        {
            // no issues here, was able to properly get the Request Body

            if (rawObject is HttpRequest httpRequest)
            {
                if (httpRequest?.Body != null && httpRequest.Body.CanRead)
                {
                    using (var sr = new StreamReader(httpRequest.Body))
                    {
                        var requestBody = await sr.ReadToEndAsync();
                        activity.SetTag("http.request.body", requestBody);
                    }
                }

                activity.SetTag("http.request.headers", httpRequest?.Headers.ToString());
                activity.SetTag("requestProtocol", httpRequest.Protocol);
            }
        }
        else if (eventName.Equals("OnStopActivity"))
        {
            if (rawObject is HttpResponse httpResponse)
            {
                // heres the part where I'm kinda struggling to get the Response body, Ive tried a few different attempts, like the request one, but not luck yet
            }
        }
    })
})

Describe your environment.

ASP.NET Core 3.1 Web API project (using the weather example that Visual Studio automatically creates).

What are you trying to achieve? Being able to properly intercept the response body so I can add that into the activity as a tag (I bet this is pretty simple to accomplish and it’s just me being a noob at it 😄 )

[CodeBlanch]

@DavoBR

You can try the below if you want, it seems to work in my limited testing. But I cannot stress enough this is a really really bad awful idea which will destroy the throughput of your service(s) 💣

builder.AddAspNetCoreInstrumentation(options => options.Enrich = EnrichAspNetCore);

static void EnrichAspNetCore(Activity activity, string eventName, object payload)
{
    switch (eventName)
    {
        case "OnStartActivity":
            if (payload is HttpRequest request)
            {
                var context = request.HttpContext;

                var bufferedResponseStream = new MemoryStream();

                var streamResponseBodyFeature = new StreamResponseBodyFeature(
                    bufferedResponseStream,
                    context.Features.Get<IHttpResponseBodyFeature>()!);

                context.Features.Set<IHttpResponseBodyFeature>(streamResponseBodyFeature);

                context.Response.OnStarting(async () =>
                {
                    try
                    {
                        if (bufferedResponseStream.TryGetBuffer(out var data))
                        {
                            string responseBody = Encoding.UTF8.GetString(data.Array!, 0, data.Count);

                            activity.SetTag("http.response_content", responseBody);
                        }
                    }
                    finally
                    {
                        var originalStream = streamResponseBodyFeature.PriorFeature?.Stream;
                        if (originalStream != null)
                        {
                            bufferedResponseStream.Position = 0;

                            await bufferedResponseStream.CopyToAsync(originalStream).ConfigureAwait(false);
                        }
                    }
                });
            }

            break;
    }
}

[makiyamad]

edit: Look at the comments below pointed by @CodeBlanch before trying this out. This doubles the memory hit for every request/response.

My original comment: One thing that worked well for me was enriching the HttpClient Instrumentation and using HttpRequestMessage and HttpResponseMessage. This way I could use ReadAsStringAsync() for the request & response. My TraceProviderBuilder ended up like the following:

builder
	.SetSampler(new AlwaysOnSampler())
	.AddAspNetCoreInstrumentation()
	.AddHttpClientInstrumentation(options =>
	{
		options.Enrich = (activity, eventName, rawObject) =>
		{
			if (eventName.Equals("OnStartActivity"))
			{
				if (rawObject is HttpRequestMessage httpRequest)
				{
					var request = "empty";
					if (httpRequest.Content != null)
						request = httpRequest.Content.ReadAsStringAsync().Result;
					activity.SetTag("http.request_content", request);
				}
			}

			if (eventName.Equals("OnStopActivity"))
			{

				if (rawObject is HttpResponseMessage httpResponse)
				{
					var response = "empty";
					if (httpResponse.Content != null)
						response = httpResponse.Content.ReadAsStringAsync().Result;
					activity.SetTag("http.response_content", response);
				}
			}

		};
	})

This way I was able to visualize the http.request_content & http.response_content in the UI of my chosen telemetry tool.