Can't remove SSLHandler after failed handshake

I’m trying to implement something like OptionalSslHandler but for a client side. i.e if server supports SSL, connect with SSL and use plaintext otherwise.

To do that I added an SslHandler to pipeline, added listener to delete handler in case of handshake failure or do nothing and keep sslhandler in case of success. This approach works when server has SSL=on (SSlHandler on server side), optional (OptionalSsLHandler on server side) but doesn’t work with SSL=off (no ssl related handlers at all on server side) and failure in case of SSL packets.

I added logging of handshakeFuture cause and got

java.nio.channels.ClosedChannelException: null
 at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1063)

I suspect handshake bytes are written into pendingUnencryptedWrites if no-ssl side doesn’t process them and closes the channel - but shouldn’t handler be “deletable” despite on closure on another end? We know that handshake failed and there were no other data exchange - one would expect that handler is clean after that.

Expected behavior

When handshakeFuture is resolved and result is failed it’s possible to remove SSLHandler from the pipeline.

Actual behavior

io.netty.channel.ChannelException: Pending write on removal of SslHandler
 at io.netty.handler.ssl.SslHandler.handlerRemoved0(SslHandler.java:677)

client side setup

@Override
protected void initChannel(Channel ch) throws Exception {
    SslContext sslContext = sslContextProvider.clientContext();
    SslHandler sslHandler = sslContext.newHandler(ch.alloc());
    sslHandler.setHandshakeTimeoutMillis(10 * 1000L);
    sslHandler.engine().setUseClientMode(true);

    sslHandler.handshakeFuture().addListener(future -> {
        // If handshake was successful keep sslHandler in pipeline
        if (!future.isSuccess()) {
            ch.pipeline().remove(sslHandler);
        }
    });

    ch.pipeline().addLast(sslHandler);
    sslHandler.engine().beginHandshake();
}

server side - no special handlers for SSL (neither SslHandler nor OptionalSslHandler)

Netty version

4.1.65.Final

JVM version (e.g. java -version)

openjdk version “15.0.2” 2021-01-19 OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.2+7) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.2+7, mixed mode, sharing)

OS version (e.g. uname -a)

sw_vers ProductName: macOS ProductVersion: 11.2.3

[normanmaurer]

Let me think about this for a bit and see if we can fix this or not

[BaurzhanSakhariev]

@chrisvest, thanks for bringing it up!