DecoderException/SSLException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT

Hi netty-team, we run into an (probably) netty related exception while reading from a Microsoft Azure blob storage using Java API azure-storage-blob and azure-identy to identify against blob storage, which internally depends on azure-core-http-netty including netty-handler and other netty dependencies.

The error is not systematic (i.e. it occurred from time to time, sometimes after several minutes, sometime after several hours), hence it is not really reproducible. After exception entire data read process gets stuck.

Stacktrace

2021-11-03 10:51:53,909 DEBUG SslHandler           : [id: 0xde011f0a, L:/xx.xx.xxx.xxx:43400 - R:blobcontainer.blob.core.windows.net/xx.xx.xxx.xxx:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
2021-11-03 10:51:53,932 DEBUG SslHandler           : [id: 0x68ee1974, L:/xx.xx.xxx.xxx:43394 - R:blobcontainer.blob.core.windows.net/xx.xx.xxx.xxx:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
2021-11-03 10:51:53,942 DEBUG SslHandler           : [id: 0x38b405f3, L:/xx.xx.xxx.xxx:43398 - R:blobcontainer.blob.core.windows.net/xx.xx.xxx.xxx:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
...
2021-11-03 11:00:23,255 DEBUG CountedOpenSslEngine : SSL_read failed with 1: OpenSSL error: 503316581 error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
2021-11-03 11:00:23,263 ERROR BlobContainer        : Error dropped: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
reactor.core.Exceptions$ErrorCallbackNotImplemented: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
Caused by: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1071)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1365)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1305)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1392)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1435)
        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:222)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342)
        at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1246)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
        ... 17 common frames omitted

Netty version

4.1.69.FINAL also tried with 4.1.54.Final

JVM version (e.g. java -version)

openjdk 11.0.10 2021-01-19
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.10+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.10+9, mixed mode)

OS version (e.g. uname -a)

Linux 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Thank you in advance for your help.

[vpanta]

Unfortunately I have neither experience with Netty nor with Bazel’s source code (beyond simple “update these deps” PRs), so I can’t really help you there. The best I can do is point you to https://github.com/bazelbuild/bazel/blob/6872fd230b7fe4a15fa900d16f6f9ddd5726cdc3/src/main/java/com/google/devtools/build/lib/remote/http/HttpCacheClient.java#L273 which seems to be the only place referencing Netty’s SslHandler in Bazel.

[ronisec]

Unfortunately, as @vpanta stated above this is a transient issue for us as well, we aren’t encountering it consistently.