Redis timeout exceptions

• Your Abp package version:

<PackageReference Include="Abp" Version="7.0.1" />
<PackageReference Include="Abp.Zero.Common" Version="7.0.1" />

• Your base framework: .Net Framework or .Net Core. <TargetFramework>net6.0</TargetFramework>

• Exception message and stack trace if available. Message:

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 33417ms elapsed, timeout is 5000ms), command=GET, next: GET n:AspNet.Identity.SecurityStamp,c:.1, inst: 0, qu: 0, qs: 0, aw: True, rs: ReadAsync, ws: Writing, in: 17564, serverEndpoint: some.server.com:1234, mc: 1/1/0, mgr: 10 of 10 available, clientName: 9ab3ec7efa14, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=49,Free=32718,Min=1,Max=32767), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) 

Stack trace:

StackExchange.Redis.RedisTimeoutException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Abp.Runtime.Caching.Redis.AbpRedisCache+<TryGetValueAsync>d__5.MoveNext (Abp.RedisCache, Version=7.0.1.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Abp.Runtime.Caching.AbpCacheBase`2+<GetOrDefaultAsync>d__21.MoveNext (Abp, Version=7.0.1.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at ProjectName.Web.Authentication.JwtBearer.JwtSecurityStampHandler+<ValidateSecurityStampFromCache>d__9.MoveNext (ProjectName.Web.Core, Version=10.3.0.0, Culture=neutral, PublicKeyToken=null: C:\agent\_work\3\s\aspnet-core\src\ProjectName.Web.Core\Authentication\JwtBearer\JwtSecurityStampHandler.cs:85)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at ProjectName.Web.Authentication.JwtBearer.JwtSecurityStampHandler+<Validate>d__5.MoveNext (ProjectName.Web.Core, Version=10.3.0.0, Culture=neutral, PublicKeyToken=null: C:\agent\_work\3\s\aspnet-core\src\ProjectName.Web.Core\Authentication\JwtBearer\JwtSecurityStampHandler.cs:60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at ProjectName.Web.Authentication.JwtBearer.ProjectNameJwtSecurityTokenHandler+<ValidateSecurityStampAsync>d__12.MoveNext (ProjectName.Web.Core, Version=10.3.0.0, Culture=neutral, PublicKeyToken=null: C:\agent\_work\3\s\aspnet-core\src\ProjectName.Web.Core\Authentication\JwtBearer\ProjectNameJwtSecurityTokenHandler.cs:130)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Nito.AsyncEx.Synchronous.TaskExtensions.WaitAndUnwrapException (Nito.AsyncEx.Tasks, Version=5.1.2.0, Culture=neutral, PublicKeyToken=null)
   at System.Threading.ExecutionContext.RunInternal (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Nito.AsyncEx.Synchronous.TaskExtensions.WaitAndUnwrapException (Nito.AsyncEx.Tasks, Version=5.1.2.0, Culture=neutral, PublicKeyToken=null)
   at Nito.AsyncEx.AsyncContext.Run (Nito.AsyncEx.Context, Version=5.1.2.0, Culture=neutral, PublicKeyToken=null)
   at ProjectName.Web.Authentication.JwtBearer.ProjectNameJwtSecurityTokenHandler.ValidateToken (ProjectName.Web.Core, Version=10.3.0.0, Culture=neutral, PublicKeyToken=null: C:\agent\_work\3\s\aspnet-core\src\ProjectName.Web.Core\Authentication\JwtBearer\ProjectNameJwtSecurityTokenHandler.cs:70)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler+<HandleAuthenticateAsync>d__6.MoveNext (Microsoft.AspNetCore.Authentication.JwtBearer, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

• Steps needed to reproduce the problem.

1. Deploy an application to azure B1 app service instance
2. Put application under high load (I put 500 simple GET requests at once)
3. Exception will be thrown

Extra: Connection string template:

some.server.com:1234,password=password,ssl=True,abortConnect=False,channelPrefix=Development

Configuration:

ConnectionMultiplexer.SetFeatureFlag("preventthreadtheft", true);

Configuration.Caching.UseRedis(options =>
{
	options.ConnectionString = _appConfiguration["ConnectionStrings:Redis"];
	options.DatabaseId = _appConfiguration["Abp:RedisCache:DatabaseId"].To<int>();
});

I had reviewed database & connection provider here: https://github.com/aspnetboilerplate/aspnetboilerplate/blob/dev/src/Abp.RedisCache/Runtime/Caching/Redis/AbpRedisCacheDatabaseProvider.cs

As I can see one connection across one instance is being used. I saw a few topics which recommended using a connection pool instead of one single connection. How do you think will it help? Maybe Redis configuration is wrong / etc? Maybe fallback options exist to ignore dependency failure?

Could you please advise/resolve the timeout problem?

I am going to fix it pretty straightforward: just replace RedisCacheManager and ignore all timeout exceptions (fallback to database/default value instead of 500 error and request failure).

I never had problems locally + app services with higher size produces fewer exceptions. B1 produces ~400 exceptions per ~26.5K requests. It is 24 hours statistics. B2 produces ~400 exceptions per ~379K requests. It is monthly statistics.

Thank you in advance.

[imgd]

StackExchange.Redis 组件就是有这个问题，主要表现场景为存取大容量数据容易出现。 用csredis替换则正常。https://github.com/2881099/csredis

[DrAnaconda]

Hi @ismcagdas , sorry, drowned in daily routine

Yes. I incorporated following approach (same for TryGetValues & others):

	public override async Task<ConditionalValue<object>> TryGetValueAsync(string key)
	{
		try
		{
			var result = await base.TryGetValueAsync(key);

			return result;
		}
		catch (RedisTimeoutException ex)
		{
			_logger.Error("Redis dependency failed", ex);
			return CreateEmptyConditionalValue(new[] { key }).First();
		}
	}

	protected virtual ConditionalValue<object>[] CreateEmptyConditionalValue(IEnumerable<string> keys)
	{
		return keys.Select(x => new ConditionalValue<object>(false, null)).ToArray();
	}

It is working now, no issues anymore. Maybe we will use another redis client but this solution is appreciable for now. It handles common anomalies. While testing I used a randomizer to throw exceptions inside this method. If exceptions will be continuous, not rare application/request may load/perform forever.

Regarding PerRequest I had the same issues, so I was looking for guaranteed solution with no exceptions