Netty sends TCP-RST instead of TCP-FIN

Expected behavior

We are using netty for our Https server. We have a case where a client sends a Https Request and uses an Invalid Certificate for SSL Handshake. Server should send a Handshake Failure TLS Alert with 3-state TCP-FIN packet

Actual behavior

Server sends a Handshake Failure TLS Alert but instead of TCP-FIN server sends a TCP-RST. The Server TCP-RST acts destructively because it potentially destroys the unprocessed TLS message in the receive buffer of its TCP peer. See wireshark trace snapshot

Steps to reproduce

Setup a simple Https Server with keystore and truststore. Configure a Https Client with keystore(with a certificate that is not trust-able by server) and truststore

Netty version

4.1.6.Final

JVM version (e.g. java -version)

1.8.0-u121

OS version (e.g. uname -a)

CentOS Linux version 7

[Scottmitch]

I don`t think this is a TCP issue

Netty doesn’t implement TCP and therefore doesn’t explicitly send a TCP RST nor a TCP FIN. Netty calls close and the underlying TCP stack determines what is “appropriate”. What is appropriate may vary depending on the underlying implementation of TCP (the RFC allows some wiggle room with the SHOULD language https://github.com/netty/netty/issues/6818#issuecomment-314493013).

Netty may attempt to read data before the close and drain any pending data. However this approach is racy (what if data still arrives after we read but before we call close()) and introduces heuristics (how long should we read and wait to call close). I don’t think we want to want to go down this road to “fix” something which I’m not convinced is an issue in the first place (this is allowed by the TCP RFCs).

Please follow the the code snipped of @ramahmoo and his instructions, they make perfectly sense.

As described in https://github.com/netty/netty/issues/6818#issuecomment-313143309 I did follow these steps and my client (curl) was able to correctly process the alert despite a RST being sent (see screen shots and description in my comment). The client should be able to read the alert regardless of FIN or RST being the next packet.

[johnou]

While it may not be related to the original report pay attention if you are running your application behind an AWS NLB https://medium.com/tenable-techblog/lessons-from-aws-nlb-timeouts-5028a8f65dda