Failed to decode Websocket Frame due to size being too big on Java 12+

Expected behavior

Hi, I’m using a Netty-based Websocket Client Catnip, which relies on Vert.x to connect to a WebSocket-based Service Discord. On Java 11 and before, it connected just fine to the websocket.

Actual behavior

On Java 12 (and also latest 13-ea builds), it errors with the following:

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Fail to unwrap network record
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: javax.net.ssl.SSLException: Fail to unwrap network record
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:470)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
	at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634)
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
	... 17 common frames omitted
Caused by: java.lang.RuntimeException: javax.crypto.ShortBufferException: Output buffer too small
	at java.base/com.sun.crypto.provider.ChaCha20Cipher.engineDoFinal(ChaCha20Cipher.java:703)
	at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:826)
	at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
	at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2503)
	at java.base/sun.security.ssl.SSLCipher$T12CC20P1305ReadCipherGenerator$CC20P1305ReadCipher.decrypt(SSLCipher.java:2188)
	at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
	at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
	at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
	at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
	at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
	... 25 common frames omitted
Caused by: javax.crypto.ShortBufferException: Output buffer too small
	at java.base/com.sun.crypto.provider.ChaCha20Cipher$EngineAEADDec.doFinal(ChaCha20Cipher.java:1360)
	at java.base/com.sun.crypto.provider.ChaCha20Cipher.engineDoFinal(ChaCha20Cipher.java:701)
	... 36 common frames omitted

The full log is: https://hasteb.in/alubotop.lua

Steps to reproduce

I made a Gradle-based mini test with just the bare minimum required.

This bug was found while trying to connect to a Discord Bot, which requires you to create a account, which is free, and also create an application at https://discordapp.com/developers, which is also free.

The test code requires a bot token as the only parameter of the program.

Discord proxies itself through Cloudflare so there are probably better ways to set-up a way to reproduce, I just doesn’t have the tools to do it.

Minimal yet complete reproducer code (or URL to code)

https://github.com/adriantodt/catnip-netty-failtest

Netty version

Tested on Netty 4.1.30.Final and 4.1.36.Final

JVM version (e.g. java -version)

On Linux:

openjdk version "12.0.1" 2019-04-16
OpenJDK Runtime Environment (build 12.0.1+12)
OpenJDK 64-Bit Server VM (build 12.0.1+12, mixed mode)

On Windows:

openjdk version "12.0.1" 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 12.0.1+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 12.0.1+12, mixed mode, sharing)

OS version (e.g. uname -a)

Linux:

Linux 03f498af05e1 4.15.0-22-generic #24-Ubuntu SMP Wed May 16 12:15:17 UTC 2018 x86_64 Linux
and
Linux aru.pw 4.15.0-22-generic #24-Ubuntu SMP Wed May 16 12:15:17 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Windows:

Microsoft Windows 10 Pro Version 1803 (OS Build 17134.706)

[normanmaurer]

@adriantodt thanks a lot!

[normanmaurer]

A patch was proposed for OpenJDK that should fix it:

https://mail.openjdk.java.net/pipermail/security-dev/2019-August/020535.html

Will close the issue as it is not a Netty issue but a openJDK one.