TLS failures (BoringSSL with delegating executor) after updating Netty
See original GitHub issueI’ve been trying to reproduce this for some time without success… unfortunately we’re not able to reproduce it in staging… thought it was worth sharing all my debugging so far to get some fresh eyes.
Expected behavior
Actual behavior
Here are some samples I pulled from the server logs, quick note is that it is a public facing server… so there are also a lot of bots connecting and probing, but it was evident from client crash logs and metrics that there were problems.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:100003fc:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_RECORD_MAC
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at com.sulake.h4k.common.util.io.NonBlockingIOThreadFactory.lambda$newThread$0(NonBlockingIOThreadFactory.java:18) ~[h4k-common-7.0.532.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at com.sulake.h4k.common.util.io.NonBlockingIOThreadFactory.lambda$newThread$1(NonBlockingIOThreadFactory.java:26) ~[h4k-common-7.0.532.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.70.Final.jar:4.1.70.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
Caused by: javax.net.ssl.SSLException: error:100003fc:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_RECORD_MAC
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1071) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1365) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1305) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1392) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1435) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:222) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
... 24 more
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:1000042e:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at com.sulake.h4k.common.util.io.NonBlockingIOThreadFactory.lambda$newThread$0(NonBlockingIOThreadFactory.java:18) ~[h4k-common-7.0.532.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at com.sulake.h4k.common.util.io.NonBlockingIOThreadFactory.lambda$newThread$1(NonBlockingIOThreadFactory.java:26) ~[h4k-common-7.0.532.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.70.Final.jar:4.1.70.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
Caused by: javax.net.ssl.SSLException: error:1000042e:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1071) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1365) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1305) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1392) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1435) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:222) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
... 24 more
client (Unity with C#) crash logs
TlsException: Failed to read data to TLS context - error code: UNITYTLS_INTERNAL_ERROR
Mono.Unity.Debug.CheckAndThrow (Mono.Unity.UnityTls+unitytls_errorstate errorState, System.String context, Mono.Security.Interface.AlertDescription defaultAlert) (at <00000000000000000000000000000000>:0)
Mono.Unity.Debug.CheckAndThrow (Mono.Unity.UnityTls+unitytls_errorstate errorState, System.String context, Mono.Security.Interface.AlertDescription defaultAlert) (at <00000000000000000000000000000000>:0)
Mono.Unity.UnityTlsContext.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.MobileAuthenticatedStream.ProcessRead (Mono.Net.Security.BufferOffsetSize userBuffer) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncReadRequest.Run (Mono.Net.Security.AsyncOperationStatus status) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<ProcessOperation>d__24.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<InnerRead>d__25.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.MobileAuthenticatedStream+<InnerRead>d__66.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1[TResult,TInstance].Complete (TInstance thisRef, System.Func`3[T1,T2,TResult] endMethod, System.IAsyncResult asyncResult, System.Boolean requiresSynchronization) (at <00000000000000000000000000000000>:0)
System.AsyncCallback.Invoke (System.IAsyncResult ar) (at <00000000000000000000000000000000>:0)
System.Threading.WaitCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ThreadPoolWorkQueue.Dispatch () (at <00000000000000000000000000000000>:0)
--- End of stack trace from previous location where exception was thrown ---
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<StartOperation>d__23.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.Finish (System.Boolean bUserDelegateExecuted) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetException (System.Object exceptionObject) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetException (System.Exception exception) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<ProcessOperation>d__24.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<InnerRead>d__25.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.MobileAuthenticatedStream+<InnerRead>d__66.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1[TResult,TInstance].Complete (TInstance thisRef, System.Func`3[T1,T2,TResult] endMethod, System.IAsyncResult asyncResult, System.Boolean requiresSynchronization) (at <00000000000000000000000000000000>:0)
System.AsyncCallback.Invoke (System.IAsyncResult ar) (at <00000000000000000000000000000000>:0)
System.Threading.WaitCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ThreadPoolWorkQueue.Dispatch () (at <00000000000000000000000000000000>:0)
Rethrow as AuthenticationException: A call to SSPI failed, see inner exception.
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <00000000000000000000000000000000>:0)
Mono.Net.Security.MobileAuthenticatedStream+<StartOperation>d__58.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<StartOperation>d__23.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.Finish (System.Boolean bUserDelegateExecuted) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetException (System.Object exceptionObject) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetException (System.Exception exception) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<ProcessOperation>d__24.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<InnerRead>d__25.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.MobileAuthenticatedStream+<InnerRead>d__66.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1[TResult,TInstance].Complete (TInstance thisRef, System.Func`3[T1,T2,TResult] endMethod, System.IAsyncResult asyncResult, System.Boolean requiresSynchronization) (at <00000000000000000000000000000000>:0)
System.AsyncCallback.Invoke (System.IAsyncResult ar) (at <00000000000000000000000000000000>:0)
System.Threading.WaitCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ThreadPoolWorkQueue.Dispatch () (at <00000000000000000000000000000000>:0)
Steps to reproduce
Minimal yet complete reproducer code (or URL to code)
@Override
public void initChannel(SocketChannel channel) throws Exception {
final ChannelPipeline pipeline = channel.pipeline();
pipeline.addLast(new IdleStateHandler(MAX_READ_IDLE_SECONDS, 0, 0));
pipeline.addLast(new ProxyMessageDecoder(trustedIpMatcher)); // HAProxy proxy protocol message decoder
final SSLEngine sslEngine = sslContext.newEngine(channel.alloc());
sslEngine.setEnabledProtocols(ENABLED_PROTOCOLS); // "TLSv1.2", client is also configured to use TLS12
pipeline.addLast(null, INNER_SSL_HANDLER, new SslHandler(sslEngine, false, sslForkJoinPool)); // StartTLS false, delegatedTaskExecutor supplied
pipeline.addLast(EvaServerDecoder.class.getSimpleName(), new EvaServerDecoder()); // LengthFieldBasedFrameDecoder
pipeline.addLast(lengthFieldPrepender);
pipeline.addLast(tcpStatisticsHandler); // mbean counters
pipeline.addLast(proxyChannelHandler); // handler to broadcast / disconnect all clients
pipeline.addLast(new ClientSession(beanFactory, channel));
}
Netty version
4.1.70.Final + tcnative 2.0.45.Final fail 4.1.67.Final + tcnative 2.0.40.Final fail 4.1.51.Final + tcnative 2.0.34.Final ok
JVM version (e.g. java -version
)
openjdk version “1.8.0_302” OpenJDK Runtime Environment (build 1.8.0_302-b08) OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)
OS version (e.g. uname -a
)
NAME=“Amazon Linux” VERSION=“2” ID=“amzn” ID_LIKE=“centos rhel fedora” VERSION_ID=“2” PRETTY_NAME=“Amazon Linux 2”
Issue Analytics
- State:
- Created 2 years ago
- Comments:60 (60 by maintainers)
Top GitHub Comments
@normanmaurer thanks 👌 hit me up if you’re in Italy.
@johnou thanks a lot for all your help to track this down.