Private DockerHub Repository cannot pull agent for labeled agent build

I’ve provisioned a cloud agent here - https://hub.docker.com/repository/docker/chrisbbehrens/agent-dnc. When this repo is public, the agent provisioning works fine, and a build assigned to its label builds correctly.

When I make the repo private, and assign the DockerHub creds to the agent in the Agent template, the agent cannot be provisioned, and the build sits at ‘Jenkins’ doesn’t have label ‘LABEL’ forever. Here are the contents of the log around this agent:

Cached connection io.jenkins.docker.client.DockerAPI$SharableDockerClient@231ac40a to DockerClientParameters{dockerUri=tcp://host.docker.internal:2375, credentialsId=null, readTimeoutInMsOrNull=60000, connectTimeoutInMsOrNull=60000}
Apr 02, 2020 11:18:28 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud canAddProvisionedSlave
Provisioning 'chrisbbehrens/agent-dnc:v4' number 1 (of 10) on 'docker'; Total containers: 0 (of 100)
Apr 02, 2020 11:18:28 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision
Will provision 'chrisbbehrens/agent-dnc:v4', for label: 'DOTNETCORE', in cloud: 'docker'
Apr 02, 2020 11:18:28 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Started provisioning Image of chrisbbehrens/agent-dnc:v4 from docker with 1 executors. Remaining excess workload: 0
Apr 02, 2020 11:18:28 PM INFO com.nirima.jenkins.plugins.docker.DockerTemplate pullImage
Pulling image 'chrisbbehrens/agent-dnc:v4'. This may take awhile...
Apr 02, 2020 11:18:28 PM INFO io.jenkins.docker.client.DockerAPI getOrMakeClient
Cached connection io.jenkins.docker.client.DockerAPI$SharableDockerClient@38419f0 to DockerClientParameters{dockerUri=tcp://host.docker.internal:2375, credentialsId=null, readTimeoutInMsOrNull=300000, connectTimeoutInMsOrNull=60000}

A successful provisioning in another build for another public agent looks like this in the logs:

Asked to provision 1 slave(s) for: docker
Apr 02, 2020 11:35:12 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud canAddProvisionedSlave
Provisioning 'chrisbbehrens/jenkinsdocker:v1' on 'docker'; Total containers: 0 (of 100)
Apr 02, 2020 11:35:12 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision
Will provision 'chrisbbehrens/jenkinsdocker:v1', for label: 'docker', in cloud: 'docker'
Apr 02, 2020 11:35:12 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Started provisioning Image of chrisbbehrens/jenkinsdocker:v1 from docker with 1 executors. Remaining excess workload: 0
Apr 02, 2020 11:35:12 PM INFO com.nirima.jenkins.plugins.docker.DockerTemplate pullImage
Pulling image 'chrisbbehrens/jenkinsdocker:v1'. This may take awhile...
Apr 02, 2020 11:35:12 PM INFO io.jenkins.docker.client.DockerAPI getOrMakeClient
Cached connection io.jenkins.docker.client.DockerAPI$SharableDockerClient@98e64c2 to DockerClientParameters{dockerUri=tcp://host.docker.internal:2375, credentialsId=null, readTimeoutInMsOrNull=300000, connectTimeoutInMsOrNull=60000}
Apr 02, 2020 11:35:13 PM INFO com.nirima.jenkins.plugins.docker.DockerTemplate pullImage
Finished pulling image 'chrisbbehrens/jenkinsdocker:v1', took 927 ms
Apr 02, 2020 11:35:13 PM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Trying to run container for chrisbbehrens/jenkinsdocker:v1
Apr 02, 2020 11:35:13 PM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Trying to run container for node Docker builder image-0000i1zamejp6 from image: chrisbbehrens/jenkinsdocker:v1
Apr 02, 2020 11:35:14 PM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Started container ID 298fa6f10321824a1d4125a755569f078aa2f3070b61f07e57e17d96fc381bf6 for node Docker builder image-0000i1zamejp6 from image: chrisbbehrens/jenkinsdocker:v1
Apr 02, 2020 11:35:16 PM WARNING io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
Unexpected data on container stderr: Apr 02, 2020 11:35:16 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Using /agent.log as an agent error log destination; output log will not be generated

Apr 02, 2020 11:35:17 PM WARNING io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
Unexpected data on container stderr: channel started

Apr 02, 2020 11:35:20 PM INFO hudson.slaves.NodeProvisioner lambda$update$6
Image of chrisbbehrens/jenkinsdocker:v1 provisioning successfully completed. We have now 2 computer(s)
Apr 02, 2020 11:36:23 PM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel Docker builder image-0000i1zamejp6
java.net.SocketException: Socket closed
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at io.jenkins.docker.client.DockerMultiplexedInputStream.readInternal(DockerMultiplexedInputStream.java:48)
	at io.jenkins.docker.client.DockerMultiplexedInputStream.read(DockerMultiplexedInputStream.java:30)
	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:91)
	at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)

Apr 02, 2020 11:36:23 PM WARNING hudson.remoting.Request$2 run
Failed to send back a reply to the request hudson.remoting.Request$2@f4f1a3e
java.net.SocketException: Socket closed
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at io.jenkins.docker.client.DockerMultiplexedInputStream.readInternal(DockerMultiplexedInputStream.java:48)
	at io.jenkins.docker.client.DockerMultiplexedInputStream.read(DockerMultiplexedInputStream.java:30)
	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:91)
	at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@32b58ba7:Docker builder image-0000i1zamejp6": channel is already closed
	at hudson.remoting.Channel.send(Channel.java:760)
	at hudson.remoting.Request$2.run(Request.java:382)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
	at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Apr 02, 2020 11:36:23 PM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'Docker builder image-0000i1zamejp6'.
Apr 02, 2020 11:36:23 PM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'Docker builder image-0000i1zamejp6'.
Apr 02, 2020 11:36:23 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
Build Agent-DNC #5 completed: SUCCESS
Apr 02, 2020 11:36:33 PM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container '298fa6f10321824a1d4125a755569f078aa2f3070b61f07e57e17d96fc381bf6' for node 'Docker builder image-0000i1zamejp6'.
Apr 02, 2020 11:36:33 PM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container '298fa6f10321824a1d4125a755569f078aa2f3070b61f07e57e17d96fc381bf6' for node 'Docker builder image-0000i1zamejp6'.

This all worked prior to Wednesday. Could this be related to the Java Docker API update?

[FeynmanFan]

I can confirm that the 3.1.5.2 update fixes the problem with no obvious regression elsewhere.

[pjdarton]

I’ll second that: @ericcitaire Thank you for fixing this.