"bad_certificate" on TLS connection to remote dockerd

Summary:

I am unable to connect to a remote dockerd instance due to TLS issues.

Versions & Error Messages:

• docker-plugin version: 1.2.2
• jenkins version: 2.263.3
• docker engine version: 20.10.2
• java version: OpenJDK 1.11.0 (Java 11)
• OS: ubuntu-sever-20.04
• issue occurs during cloud setup, before any containers are in use
• stack trace:

javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:337)
	at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:186)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
	at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
	at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
	at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
Caused: io.netty.handler.codec.DecoderException
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:648)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:583)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

• TLS signature algorithms: All certificates use ecdsa-with-SHA256

Details

The remote host (remote.host) with an exposed docker socket is reachable from the Jenkins master (local.host) using:

$ docker -H tcp://remote.host:2376 --tlsverify --tlscacert=root_ca.crt --tlscert=fullchain.pem --tlskey=privkey.pem info
# ... info printed successfully ...

The root_ca.crt has also been imported into local.host’s system Java truststore for good measure.

When I attempt to create a new Docker cloud in https://local.host/configureClouds/, I specify the following settings:

name: remote-1 docker host uri: tcp://remote.host:2376 server credentials: X.509 Client Certificate with the privkey.pem, fullchain.pem and root_ca.crt copied into the respective fields. These are all standard X.509 base64 encoded certificates (e.g. not der).

No other fields are changed from their defaults.

I then run ‘Test Connection’, which results in the stacktrace above.

Notes

1. Thank you for putting in the effort to develop this plugin. It’s much appreciated.
2. I also tried Yet Another Docker, which also didn’t work (same error using the default client)

[pjdarton]

1. If you’ve got a .hpi file, you can install it into your Jenkins instance via Manage Plugins -> Advanced and upload it there. There’s no need to rebuild the docker-plugin - it uses the docker-java library that’s provided by the docker-java-api-plugin.

2. Don’t panic The Jenkins OSS project has a process to allow someone who wants changes made to obtain access to a plugin that no longer has an active maintainer. That’s how Eric got access to the docker-java-api-plugin so that #777 could be merged last year; if we’re lucky, he’ll be willing to help bump the version again. …and FYI while I’m officially the maintainer of this plugin, I’m only here because “I touched it last” and that situation is not uncommon - folks get involved, drive things forwards for a while, achieve what they wanted, and then reduce their involvement; eventually someone else picks things up again.

3. The docker-java-api-plugin is a shared resource; it’s used by a few plugins, not just this one; that’s why it lives independently and why it isn’t “under the wing” of this one. …and it’s why anyone wanting to bump the version of docker-java will need to coordinate with those other plugins in order to ensure that they can test things before it’s released. Last time we bumped the docker-java version, it broke the world and required some rapid fix-ups (fortunately simple ones, but urgent) which wasn’t fun - there’s no guarantee of binary compatibility between versions.

[dnwe]

@Osipion you mentioned fullchain.pem which suggests your client certificate has more than one cert contained within it? I think I’m hitting the same issue as you and I think it might be caused by https://github.com/docker-java/docker-java/issues/1171 — which wasn’t fixed until docker-java 3.2.2 under https://github.com/docker-java/docker-java/pull/1371 — and Jenkins is only exposing docker-java 3.1.5 (via the org.jenkins-ci.plugins:docker-java-api wrapper plugin)