High memory usage observed when APM added

Describe the bug After enabling DataDog APM the memory uses by the container instance increases dramatically. Observed in two services during APM testing.

Both services have been stable in memory usage prior to adding and enabling APM.

To Reproduce Steps to reproduce the behavior: Update Dockerfile to include DataDog APM on ASP.NET Core 3.1 runtime image as base.

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1 AS base
WORKDIR /app
EXPOSE 80
ENV DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true DOTNET_CLI_TELEMETRY_OPTOUT=1

# Install DataDog APM
RUN curl -LO https://github.com/DataDog/dd-trace-dotnet/releases/download/v1.12.0/datadog-dotnet-apm_1.12.0_amd64.deb
RUN dpkg -i ./datadog-dotnet-apm_1.12.0_amd64.deb

# Setup DataDog APM
ENV CORECLR_ENABLE_PROFILING=1 
ENV DD_LOGS_INJECTION=true 
ENV CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8} 
ENV CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so 
ENV DD_INTEGRATIONS=/opt/datadog/integrations.json 
ENV DD_DOTNET_TRACER_HOME=/opt/datadog 
ENV DD_DIAGNOSTIC_SOURCE_ENABLED=true 

The Datadog.Trace library is included in our project:

<PackageReference Include="Datadog.Trace" Version="1.12.0" />

Configure APM on startup within Program.cs

var dataDogAgentServer = EcsHostInfo.HostIpV4;

var settings = TracerSettings.FromDefaultSources();

settings.ServiceName = configuration.GetValue<string>("DataDog:ApmServiceName");
settings.AgentUri = new Uri($"http://{dataDogAgentServer}:{configuration.GetValue<string>("DataDog:ApmPort")}", UriKind.Absolute);
settings.DebugEnabled = true;    

Tracer.Instance = new Tracer(settings);    

Expected behavior Consistent memory usage.

Screenshots

Runtime environment

• Instrumentation mode: datadog-dotnet-apm_1.12.0_amd64.deb installed into Docker image. Manual span in middleware.
• Tracer version: 1.12.0
• OS: Debian 10 - Linux
• CLR: .NET Core 3.1

[andrewsouthard1]

Hey all - wanted to give this issue and update per @lucaspimentel PR that was submitted yesterday.

We’re actively working on reducing the performance overhead of the .NET tracer, this is one of many improvements we will be making over the coming weeks. This change specifically though, https://github.com/DataDog/dd-trace-dotnet/pull/685 we plan on adding to an official release by end of this week, once we have an updated release we encourage affected teams to update.

[dan655t]

I would like to +1 this issue as we are unable to use the APM in production due to the performance impact. We initially noticed high CPU when our app scaled out to about two times as many containers after enabling APM tracing. After reading this post we’re hypothesizing that the high CPU may be due to high memory forcing more intensive GC collect events.

We started experimenting with the APM on 2/3:

We are running on the mcr.microsoft.com/dotnet/core/runtime-deps:3.1 image. The rest of our configuration is the same as the OPs.