SSL exceptions under a high request load

Expected behavior

No SSL related exceptions under a high load

Actual behavior

This is a repost from an issue I raised with the Vert.x team for visibility: https://github.com/vert-x3/vertx-web/issues/843 But, it seemed more appropriate to ask here, since I am observing Netty related SSL exceptions:

Minimal yet complete reproducer code (or URL to code)

NA

Steps to reproduce

See below.

Netty version

• netty: 4.1.19.Final
• netty-tcnative-boringssl-static: 2.0.7.Final

JVM version (e.g. java -version)

openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

OS version (e.g. uname -a)

Linux xxx 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Context

I have a service based on Vert.x which uses Netty version listed above. I am using a performance test client which POSTs a 150kb JSON as a raw body data to the service, using an increasing stepped load. I start with 500 RPS and increase RPS by 500 every 20 minutes.

After sometime as the test load has increased to ~2000 RPS, I started observing the following exceptions related to SSL (there were many of those, so I am just providing a few samples):

2018-02-18 15:56:36,571 DEBUG [vert.x-eventloop-thread-4] SslHandler: [id: 0x44d6799c, L:/X.X.X.X:xxxx - R:/X.X.X.X:xxxx] HANDSHAKEN: TLS_ECDHE_XXXX_WITH_XXX_XXX_XXXXXX
2018-02-18 15:56:36,579 DEBUG [vert.x-eventloop-thread-0] ReferenceCountedOpenSslEngine: SSL_read failed: OpenSSL error: error:100000d7:SSL routines:OPENSSL
_internal:SSL_HANDSHAKE_FAILURE
2018-02-18 15:56:36,579 WARN  [vert.x-eventloop-thread-0] DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pi
peline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000d7:SSL routines:OPENSSL_internal:SSL_HANDSHAKE_FAILURE
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: error:100000d7:SSL routines:OPENSSL_internal:SSL_HANDSHAKE_FAILURE
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:876)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1124)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1080)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1146)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1189)
        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1248)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1159)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1194)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)

2018-02-18 15:56:36,841 DEBUG [vert.x-eventloop-thread-2] SslHandler: [id: 0xf26e1ca6, L:/X.X.X.X:xxxx - R:/X.X.X.X:xxxx] Swallowing a harmless '
connection reset by peer / broken pipe' error that occurred while writing close_notify in response to the peer's close_notify
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1108)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:345)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:126)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

Any thoughts/ideas?

[ruslan-belinskyy]

Have the same issue for VertX 3.5.4 running on top of Netty 4.1.31.Final when loading application via Apache JMeter. It’s second project where i have this issue, luckily this time in test and not in PROD.

15:56:09.048 [vert.x-eventloop-thread-0] DEBUG io.netty.handler.ssl.SslHandler - [id: 0xa0974742, L:/127.0.0.1:9443 - R:/127.0.0.1:61219] Swallowing a harmless 'connection reset by peer / broken pipe' error that occurred while writing close_notify in response to the peer's close_notify java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_181] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_181] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_181] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_181] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_181] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) ~[netty-all-4.1.31.Final.jar:4.1.31.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-all-4.1.31.Final.jar:4.1.31.Final] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) ~[netty-all-4.1.31.Final.jar:4.1.31.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) ~[netty-all-4.1.31.Final.jar:4.1.31.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:648) [netty-all-4.1.31.Final.jar:4.1.31.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:583) [netty-all-4.1.31.Final.jar:4.1.31.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500) [netty-all-4.1.31.Final.jar:4.1.31.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-all-4.1.31.Final.jar:4.1.31.Final]

[bmwcmw]

I have the same issue with exactly same dependencies. In my case we have to keep using OpenSSL provider since some ciphers and features such as cache are not supported by the JDK provider.