InvalidProgramException in 1.9.0 in .NET Core 2.2 / Fargate

Today I spot again “System.InvalidProgramException: Common Language Runtime detected an invalid program” errors in our logs

To Reproduce I don’t know how to reproduce it in a deterministic way. It is the first time I spot the error since upgrading to 1.9.0 of Datadog.Trace.ClrProfiler.Managed

Expected behavior This exception should not happen.

Screenshots

System.InvalidProgramException: Common Language Runtime detected an invalid program.
   at Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.ResultNext<TFilter, TFilterAsync>(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
   at async Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultFilters()
   at async Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at void Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at void Rethrow(object)
   at void Datadog.Trace.ClrProfiler.Integrations.AspNetCoreMvc2Integration.Rethrow(object context, int opCode, int mdToken, long moduleVersionPtr)
   at Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
   at async Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at async Task Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at async Task Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
   at async Task Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
   at async Task Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
   at async Task <internal-middleware3>:line 39
   at async Task <internal-middleware2>:line 29
   at async Task <internal-middleware1>:line 33
   at async Task Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)

Runtime environment:

• Instrumentation mode: manual with NuGet package
• Tracer version: 1.9.0
• OS: mcr.microsoft.com/dotnet/core/aspnet:2.2.3-stretch-slim (docker image) @ Fargate v1.3 (LATEST)
• CLR: .NET Core 2.2.3

Additional context We had few serious problems with InvalidProgramException while running with 1.7.0. We upgraded to 1.9.0 on 14th November and today (Nov 20, 2019 @ 13:37:54.901 UTC) I saw this error for the first time since upgrade. Since first failure, it made that particular Fargate task dead, responding with such error to all incoming requests.

Please let me know if you need any more details from me.

[lpodolak]

thanks @lucaspimentel - I was away last week. Today I will upgrade our runtime to 2.2.8. You can close the issue - I will report the problem if I ever see the the bug again. I appreciate your help a lot!

[lpodolak]

Unfortunately, I have to report again that we had InvalidProgramExceptions still few times after upgrade to .net core 2.2.8 runtime, so this is not a workaround. Right now, instead of finding a fix for this (from dd or clr team?) I try to make sure we can recover from such failures.