The read/connect timeouts default to zero - which is potentially bad

I’m using the latest version of the plugin, version v0.16.2 .

I was running against a Docker Swarm endpoint when I observed this particular problem

I observed that Jenkins stopped processing work, and a container that was being terminated did not appear to be progressing with that task. Taking a stack trace using jstack I saw the following reference to _terminate()

"Computer.threadPoolForRemoting [#216]" #4966 daemon prio=5 os_prio=0 tid=0x00007fd2d0e25000 nid=0x2d17 runnable [0x00007fd2c8e8c000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
    at org.apache.http.impl.io.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:129)
    at org.apache.http.impl.io.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:53)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
    at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252)
    at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684)
    at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:681)
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:437)
    at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:343)
    at com.github.dockerjava.jaxrs.StopContainerCmdExec.execute(StopContainerCmdExec.java:31)
    at com.github.dockerjava.jaxrs.StopContainerCmdExec.execute(StopContainerCmdExec.java:12)
    at com.github.dockerjava.jaxrs.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:23)
    at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
    at com.github.dockerjava.core.command.StopContainerCmdImpl.exec(StopContainerCmdImpl.java:63)
    at com.nirima.jenkins.plugins.docker.DockerSlave._terminate(DockerSlave.java:168)
    at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67)
    at com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy$1$1.run(DockerOnceRetentionStrategy.java:112)
    at hudson.model.Queue._withLock(Queue.java:1315)
    at hudson.model.Queue.withLock(Queue.java:1192)
    at com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy$1.run(DockerOnceRetentionStrategy.java:106)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

This was present in two invocations of jstack 7 minutes apart

Result: 2016-09-28 07:47:54
Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode): 

&

Result: 2016-09-28 07:55:50
Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):

In particular it seems that having called the QueueLock() method, we are now blocking Jenkins from doing normal work. It could be the case that this is the underlying docker-java library not setting / providing a way to set a timeout - or it is just too large, but either way, this is detrimental to the operation of the server as a whole.

I don’t know whether or not this affects all docker-java method calls that call out via HTTP, or because this particular one happens to take a lock that it then doesn’t release it

I have only seen this once so far, but it seemed to have a bad enough effect to bring it up

[PierreBtz]

I also came into this issue. Maybe a simple warning in the tooltip:

‘Timeout for opening connection to Docker API. 0 is infinity.’ -> ‘Timeout for opening connection to Docker API. 0 is infinity. Be careful as infinity can block the whole Jenkins Queue if the connection hangs.’

Also, maybe, 0 should not be the default value, as it seems more like something a user would want to do on purpose.

[ndeloof]

0.17 does not invoke docker API while owning Queue Lock.