IIS AspNetCore Module V2 throws failed to load coreclr

We upgraded our asp.net core site from .NET Core 2.2 --> 3.1

The site works fine for about 18 hours and then crashes. Restarting it gives us:

HTTP Error 500.37 - ANCM Failed to Start Within Startup Time Limit

Looking in the event log, we see this:

3/4/2020 2:31:54 AM
Source: WAS
EventId: 5009
A process serving application pool 'SSO' terminated unexpectedly. The process id was '11960'. The process exit code was '0xc0000409'.

3/4/2020 2:33:51 AM
Source: WAS
EventId: 5010
A process serving application pool 'SSO' failed to respond to a ping. The process id was '13988'.

3/4/2020 2:35:52 AM
EventId: 1007
Source: IIS AspNetCore Module V2
Application '/LM/W3SVC/28/ROOT' with physical root 'D:\Sites\SSO\Site-Prod\' failed to load coreclr. Exception message:
Managed server didn't initialize after 120000 ms.

No amount of recycling / rebooting or redeploying gets the site back up and running.

Deploying the old 2.2 code works immediately.

Any idea what might be causing this?

[davidfowl]

Putting this in the issue since I looked at the dump.

After looking at your code I think I’ve figured out the issue. You have calls to BuildServiceProvider throughout your startup code some in per request locations. That will create a new service provider universe and you’re also not disposing things.

Some examples on how to fix things:

services.AddSingleton<SsoClient>((provider) => {
                var svcProvider = services.BuildServiceProvider(); 
                var factory = svcProvider.GetService<ILoggerFactory>();
                var logger = factory.CreateLogger<SsoClient>(); 
                return new SsoClient(salesforceApiSettings.RestBaseUrl,
                    salesforceApiSettings.Username,
                    salesforceApiSettings.Password,
                    salesforceApiSettings.ApiKey,
                    salesforceApiSettings.ClientSecret,
                    salesforceApiSettings.ClientId,
                    logger);
            });

Should be:

services.AddSingleton<SsoClient>((provider) => {
                var factory = provider.GetService<ILoggerFactory>();
                var logger = factory.CreateLogger<SsoClient>(); 
                return new SsoClient(salesforceApiSettings.RestBaseUrl,
                    salesforceApiSettings.Username,
                    salesforceApiSettings.Password,
                    salesforceApiSettings.ApiKey,
                    salesforceApiSettings.ClientSecret,
                    salesforceApiSettings.ClientId,
                    logger);
            });

The real issue is in OnMessageReceived in the call to AddOpenIdConnect, you build the service provider potentially twice per request.

var sp = services.BuildServiceProvider();
var interactSvc = sp.GetService<IIdentityServerInteractionService>();

Should be:

var interactSvc = m.HttpContext.RequestServices.GetService<IIdentityServerInteractionService>();

[akiander]

I’ll work on getting a runnable sample I can upload - it might be a few days given my current workload. Just have to carve out the portions of the site that are confidential.