[Bug] obo_cache_key_not_in_cache_error with SQL DistributedTokenCache

Logs and network traces Without logs or traces, it is unlikely that the team can investigate your issue. Capturing logs and network traces is described in Logging wiki.

2022-11-15 15:16:01,284 [46] DEBUG notifications.EMailBaseQueue [(null)] - (Log) - MSAL Logging: False MSAL 4.47.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Standard [2022-11-15 14:16:01Z] [Internal cache] Clearing user token cache accessor.
2022-11-15 15:16:01,284 [46] DEBUG notifications.EMailBaseQueue [(null)] - (Log) - MSAL Logging: False MSAL 4.47.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Standard [2022-11-15 14:16:01Z - 3a69b238-071c-4d69-9f1f-9c85d0203447] [Internal cache] Total number of cache partitions found while getting access tokens: 0
2022-11-15 15:16:01,284 [46] DEBUG notifications.EMailBaseQueue [(null)] - (Log) - MSAL Logging: False MSAL 4.47.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Standard [2022-11-15 14:16:01Z - 3a69b238-071c-4d69-9f1f-9c85d0203447] [FindAccessTokenAsync] Discovered 0 access tokens in cache using partition key: ************
2022-11-15 15:16:01,284 [46] DEBUG notifications.EMailBaseQueue [(null)] - (Log) - MSAL Logging: False MSAL 4.47.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Standard [2022-11-15 14:16:01Z] [Internal cache] Total number of cache partitions found while getting refresh tokens: 0
2022-11-15 15:16:01,300 [46] DEBUG notifications.EMailBaseQueue [(null)] - (Log) - MSAL Logging: False MSAL 4.47.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Standard [2022-11-15 14:16:01Z - 3a69b238-071c-4d69-9f1f-9c85d0203447] [FindRefreshTokenAsync] Discovered 0 refresh tokens in cache using key: ************
2022-11-15 15:16:01,300 [46] DEBUG notifications.EMailBaseQueue [(null)] - (Log) - MSAL Logging: False MSAL 4.47.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Standard [2022-11-15 14:16:01Z - 3a69b238-071c-4d69-9f1f-9c85d0203447] [OBO request] AcquireTokenInLongRunningProcess was called and no access or refresh tokens were found in the cache.
2022-11-15 15:16:01,300 [46] DEBUG notifications.EMailBaseQueue [(null)] - (Log) - MSAL Logging: False MSAL 4.47.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Standard [2022-11-15 14:16:01Z - 3a69b238-071c-4d69-9f1f-9c85d0203447] Exception type: Microsoft.Identity.Client.MsalClientException
, ErrorCode: obo_cache_key_not_in_cache_error

   at Microsoft.Identity.Client.Internal.Requests.OnBehalfOfRequest.<RefreshRtOrFetchNewAccessTokenAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Identity.Client.Internal.Requests.OnBehalfOfRequest.<ExecuteAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.<RunAsync>d__12.MoveNext()

Which version of MSAL.NET are you using? MSAL.NET 4.47.2.0

Platform .NET 4.6.1

What authentication flow has the issue?

• Desktop / Mobile
  • Interactive
  • Integrated Windows Authentication
  • Username Password
  • Device code flow (browserless)
• Web app
  • Authorization code
  • [*] On-Behalf-Of
• Daemon app
  • Service to Service calls

Other?

Is this a new or existing app?

b. The app is in production, I haven’t upgraded MSAL, but started seeing this issue.

Repro

var msalConfig = DbContext.msalConfiguration.FirstOrDefault(mc => mc.idMsalConfiguration == connectionInfo.SmtpAuthMode);
smtpClient = new MailKit.Net.Smtp.SmtpClient();
smtpClient.Connect(connectionInfo.SystemMessageServerIp, connectionInfo.SystemMessageServerPort, connectionInfo.SystemMessageUseSsl);

var scopes = new[] 
{
	"email",
	"offline_access",
	"https://outlook.office.com/IMAP.AccessAsUser.All", // Only needed for IMAP
	//"https://outlook.office.com/POP.AccessAsUser.All",  // Only needed for POP
	"https://outlook.office.com/SMTP.Send", // Only needed for SMTP
};

var app = ConfidentialClientApplicationBuilder.CreateWithApplicationOptions(new ConfidentialClientApplicationOptions
{
	ClientId = msalConfig.backendClientId,
	ClientSecret = msalConfig.backendClientSecret,
	RedirectUri = msalConfig.backendRedirectUri,
	AadAuthorityAudience = AadAuthorityAudience.AzureAdAndPersonalMicrosoftAccount
}).WithAuthority("https://login.microsoftonline.com/common")
	.WithLogging((level, message, containsPii) =>
	{
		Log.Debug($"MSAL Logging: {message}");
	}).Build();

app.AddDistributedTokenCache(services =>
{
	services.AddDistributedSqlServerCache(options =>
	{
		// Requires to reference Microsoft.Extensions.Caching.SqlServer
		options.ConnectionString = msalConfig.backendCacheConnectionString;
		options.SchemaName = msalConfig.backendCacheSchemaName;
		options.TableName = msalConfig.backendCacheTableName;

		// You don't want the SQL token cache to be purged before the access token has expired. Usually
		// access tokens expire after 1 hour (but this can be changed by token lifetime policies), whereas
		// the default sliding expiration for the distributed SQL database is 20 mins. 
		// Use a value above 60 mins (or the lifetime of a token in case of longer-lived tokens)
		options.DefaultSlidingExpiration = TimeSpan.FromHours(24);
	});
});

Log.Info("Current SmtpMsalSessionKey: " + connectionInfo.SmtpMsalSessionKey);

AuthenticationResult result = null;
try
{
	result = ((ILongRunningWebApi)app).AcquireTokenInLongRunningProcess(scopes, connectionInfo.SmtpMsalSessionKey).ExecuteAsync().Result;
}
catch (AggregateException e)
{
	if (e.InnerException is MsalClientException ie)
	{
		Log.Warn("Error while acquireing token in long running process", ie);

		string sessionkey = connectionInfo.SmtpMsalSessionKey;
		Log.Info("session key before: " + sessionkey);
		result = ((ILongRunningWebApi)app).InitiateLongRunningProcessInWebApi(scopes, connectionInfo.SmtpMsalSessionUserToken, ref sessionkey).ExecuteAsync().Result;
		Log.Info("session key after: " + sessionkey);
	}
}

if (result == null) throw new Exception("Result not fetched, login failed");

enforceSendMailAddress = result.Account.Username;

var oauth2 = new SaslMechanismOAuth2 (result.Account.Username, result.AccessToken);

smtpClient.Authenticate(oauth2);

Expected behavior Using the OBO key acquired by my web application in another console application should work with the same database cache.

Actual behavior I got the session key with AcquireTokenInLongRunningProcess in my web application and tried to use it in a background worker service (Console Application) to authenticate and send mails on behalf of the user. The log file reports, that the key cannot be found in the database cache, but the key is there and the connection string is the same as in the web application.

It works for a few days, if i refresh the key in the AggregateException handler. This code was added during troubleshooting of the problem.

Possible solution

Additional context / logs / screenshots / links to code

[zmarkusdev]

Documentation of my troubleshooting steps as per your recommendations:

1. Changed the connection string with TrustServerCertificate=true - same exception
2. Updated Microsoft.Extensions.Caching.SqlServer to 6.0.13 - same exception
3. Add explicit reference to Microsoft.Data.SQLClient 2.0.1 and Microsoft.Data.SQLClient.SNI 2.0.1 - works

Thanks for your help, we were able to fix this problem!

[pmaytak]

This error doesn’t come from MSAL. One of the users came across a very similar error and they fixed it by explicitly referencing Microsoft.Data.SQLClient 2.0.1 and Microsoft.Data.SQLClient.SNI 2.0.1. You can also try updating to the latest version possible. They also made sure the connection string contains TrustServerCertificate=true. Another thing to try is to update Microsoft.Extensions.Caching.SqlServer package to the latest.