OutOfMemoryException in EventBasedLRUCache after upgrading to Microsoft.IdentityModel.Tokens 6.10.1

We are experiencing increased Thread counts and OutOfMemoryExceptions after upgrading to System.IdentityModel.Tokens.Jwt version 6.10.1 in our ASP.NET Core application running in Azure.

After rolling back the package to 6.9.0 the Thread counts are normalized:

The relevant part of the stack trace of the OutOfMemoryExceptions reported in AppInsigts:

{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Threading.Tasks.Task.ScheduleAndStart","level":0,"line":0}
{"assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Threading.Tasks.Task.Start","level":1,"line":0}
{"assembly":"Microsoft.IdentityModel.Tokens, Version=6.10.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.IdentityModel.Tokens.EventBasedLRUCache`2..ctor","level":2,"line":0}
{"assembly":"Microsoft.IdentityModel.Tokens, Version=6.10.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.IdentityModel.Tokens.InMemoryCryptoProviderCache..ctor","level":3,"line":0}
{"assembly":"Microsoft.IdentityModel.Tokens, Version=6.10.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.IdentityModel.Tokens.CryptoProviderFactory..ctor","level":4,"line":0}
{"assembly":"redacted, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null","method":"redacted.AzureKeyVaultTokenSignatureService.CreateSignedToken"}

We are using the following code to sign JWT tokens using AzureKeyVault:

public string CreateSignedToken(JwtPayload jwtPayload)
{
    var azureServiceTokenProvider = new AzureServiceTokenProvider();

    var signingCredentials =
        new SigningCredentials(
            new KeyVaultSecurityKey($"https://{configuration["KeyVaultName"]}.vault.azure.net/keys/{configuration["redacted"]}",
                new KeyVaultSecurityKey.AuthenticationCallback(azureServiceTokenProvider
                    .KeyVaultTokenCallback)), SecurityAlgorithms.RsaSha512)
        {
            CryptoProviderFactory = new CryptoProviderFactory()
                { CustomCryptoProvider = new KeyVaultCryptoProvider() }
        };

    var jwtHeader = new JwtHeader(signingCredentials);

    var token = new JwtSecurityToken(jwtHeader, jwtPayload);

    var handler = new JwtSecurityTokenHandler();
    return handler.WriteToken(token);
}

Here is the diff our our change which caused the errors start to happen:

- <PackageReference Include="Microsoft.IdentityModel.KeyVaultExtensions" Version="6.9.0" />
- <PackageReference Include="System.IdentityModel.Tokens.Jwt" Version="6.9.0" />
+ <PackageReference Include="Microsoft.IdentityModel.KeyVaultExtensions" Version="6.10.1" />
+ <PackageReference Include="System.IdentityModel.Tokens.Jwt" Version="6.10.1" />

We are on .NET Core 3.1.407.

[brentschmaltz]

@dmetzgar @mackie1001 @nemesv thanks for the detail. Our perf and stress tests didn’t pick this up because they don’t create a new CryptoProviderFactory, ill add that. We had a bug in our code where a task was left running, we assumed incorrectly, a single CryptoProviderFactory in play.

It is probably more efficient when you need to support a new algorithm to make use of CryptoProviderFactory.CustomCryptoProvider as we will call into that object first and if you respond true to IsSupportedAlgorithm, we will pass you control.

I will close this out for now, please ping us if you have issues.

[dmetzgar]

@mackie1001 To get the stacks, we did a core dump of the container and opened it in Visual Studio. VS is limited on Linux dumps but at least gives a thread view where we can see the stacks on each thread. I later confirmed with LLDB the number of threads.