Memory Leak in ProcessingRequests in ASP.Net

Describe the bug Starting in 1.5, “ProcessingRequests” never has removal called for a majority of HTTP Requests.

Leads to an unbounded memory leak. Also means the spans are never ended, so the HTTP Calls aren’t included in APM (See #863 )

Observation (which I’m still trying to trace the exact split of working/non-working addresses):

• localhost/private requests affected
• public internet requests not affected

The “System.Net.Http.Desktop.HttpRequestOut.Stop” event stops being generated for requests that aren’t to the public internet.

~And a doubled up “System.Net.Http.Desktop.HttpRequestOut.Start” event starts being generated for requests that are going to public internet~ - caused by redirect

To Reproduce Steps to reproduce the behavior:

1. Create basic ASP.Net WebAPI project with APM 1.5.1
2. Use HttpClient to GET a URL repeatedly
3. Observe no HTTP spans logged
4. Observe memory leak

Expected behavior All HTTP Spans logged, no memory leak.

Example controller action that will generate the problem:

       public async Task<string> Get(int id)
        {
            var url = "http://www.elastic.co";
            switch (id)
            { 
                case 1:
                    url = $"http://{ControllerContext.Request.RequestUri.Host}:{ControllerContext.Request.RequestUri.Port}/api/testing";
                    break;
                case 2:
                    url = "http://localhost:49487/api/values";
                    break;
                case 3:
                    url = "http://127.0.0.1:49487/api/values";
                    break;
            }

            using (var httpClient = new HttpClient())
            {
                
                var response = await httpClient.SendAsync(new HttpRequestMessage(HttpMethod.Get, url));
                var content = await response.Content.ReadAsStringAsync();
                return $"Downloaded {content.Length} from {url}";
            }
        }

[rorueda]

I’ve done some investigation on this issue and here is what I have found out.

In #800, the agent started creating an Activity with the W3C format for each transaction. Because of this, the children activities, that are created for the http requests, are created with the same format. And then we have the HttpHandlerDiagnosticListener issues when using W3C: https://github.com/dotnet/corefx/pull/40777 and https://github.com/dotnet/runtime/issues/38152. This seems to explain why this issue is present only in 1.5.0 and 1.5.1.

It was particularly difficult to understand why the AspNetFullFrameworkSampleApp seemed to work (using home/Contact). It turned out there are two other modules being used there: TelemetryCorrelationHttpModule and ApplicationInsightsWebTracking. Didn’t have the time to investigate exactly what was going on, but, by removing both, the issue was reproduced.

A possible workaround is to force the Hierarchical format on children activities by setting Activity.ForceDefaultIdFormat to true (might want to make sure Hierarchical is the default - the modules above, for instance, were setting it to W3C) , but I’m not sure this is not going to break things.

As for the memory leak problem, it’s a bit odd that spans of and ended transaction are kept in memory. I don’t know the internals, but they probably should be dropped (maybe with some kind of warning). I think it could be a good safeguard for leaks like this.

Any thoughts?

[gregkalapos]

#896 merged, with that we have the same behavior for ASP.NET Classic as we had with pre 1.5.0 versions.