Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Hanging on stop, does not detect when a container exited...

See original GitHub issue

Description

On our jenkins server/slave about half the time the dmp fails to stop the containers it started… on closer inspection it appears that the container did exit, but that dmp failed to detect this…

[INFO] --- docker-maven-plugin:0.25.2:stop (stop-containers) @ functional-tests ---
consul>     2018/05/18 18:34:41 [DEBUG] http: Request GET /v1/kv/config/application/?recurse&wait=55s&index=48 (55.579435756s) from=172.17.0.8:56272
homeaway> May 18, 2018 6:34:41 PM org.apache.coyote.AbstractProtocol pause
homeaway> INFO: Pausing ProtocolHandler ["http-bio-8080"]
kafka> [2018-05-18 18:34:41,749] INFO [GroupCoordinator 1]: Preparing to rebalance group HOMEAWAY with old generation 1 (__consumer_offsets-5) (kafka.coordinator.group.GroupCoordinator)
kafka> [2018-05-18 18:34:41,757] INFO [GroupCoordinator 1]: Group HOMEAWAY with generation 2 is now empty (__consumer_offsets-5) (kafka.coordinator.group.GroupCoordinator)
homeaway> May 18, 2018 6:34:41 PM org.apache.coyote.AbstractProtocol pause
homeaway> INFO: Pausing ProtocolHandler ["ajp-bio-8009"]
homeaway> May 18, 2018 6:34:41 PM org.apache.catalina.core.StandardService stopInternal
homeaway> INFO: Stopping service Catalina
homeaway> May 18, 2018 6:34:41 PM com.rentpayment.listener.AppServletContextListener contextDestroyed
homeaway> INFO: AppServletContextListener destroyed.
homeaway> May 18, 2018 6:34:41 PM com.rentpayment.listener.AppServletContextListener contextDestroyed
homeaway> INFO: AppServletContextListener shutdown.
homeaway> May 18, 2018 6:34:41 PM com.sun.xml.ws.transport.http.servlet.WSServletDelegate destroy
homeaway> INFO: WSSERVLET15: JAX-WS servlet destroyed
homeaway> May 18, 2018 6:34:41 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextDestroyed
homeaway> INFO: WSSERVLET13: JAX-WS context listener destroyed
homeaway> May 18, 2018 6:34:41 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesJdbc
homeaway> SEVERE: The web application [] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
homeaway> May 18, 2018 6:34:41 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
homeaway> SEVERE: The web application [] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
homeaway> May 18, 2018 6:34:41 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
homeaway> SEVERE: The web application [] appears to have started a thread named [ReportServiceExecutor1] but has failed to stop it. This is very likely to create a memory leak.
homeaway> May 18, 2018 6:34:41 PM org.apache.coyote.AbstractProtocol stop
homeaway> INFO: Stopping ProtocolHandler ["http-bio-8080"]
homeaway> May 18, 2018 6:34:41 PM org.apache.coyote.AbstractProtocol stop
homeaway> INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
homeaway> May 18, 2018 6:34:41 PM org.apache.coyote.AbstractProtocol destroy
homeaway> INFO: Destroying ProtocolHandler ["http-bio-8080"]
homeaway> May 18, 2018 6:34:41 PM org.apache.coyote.AbstractProtocol destroy
homeaway> INFO: Destroying ProtocolHandler ["ajp-bio-8009"]

CONTAINER ID        IMAGE                                                   COMMAND                  CREATED             STATUS                       PORTS                                                        NAMES
22b84da977fe        yapstone/domestic:latest                                "catalina.sh run"        7 minutes ago       Exited (143) 6 minutes ago

docker inspect 22b84da977fe
[
    {
        "Id": "22b84da977fe5d1f9aff090c231cd723f93d8c6734f3b162a5fe367988cc3e48",
        "Created": "2018-05-18T18:33:45.528591424Z",
        "Path": "catalina.sh",
        "Args": [
            "run"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 143,
            "Error": "",
            "StartedAt": "2018-05-18T18:33:45.868135081Z",
            "FinishedAt": "2018-05-18T18:34:42.04234177Z"
        },

jstack on the maven process

"main" #1 prio=5 os_prio=0 tid=0x00007fce1c00a800 nid=0x5b36 runnable [0x00007fce2488d000]
   java.lang.Thread.State: RUNNABLE
	at com.kenai.jffi.Foreign.invokeN3O1(Native Method)
	at com.kenai.jffi.Invoker.invokeN3(Invoker.java:1061)
	at jnr.enxio.channels.Native$LibC$jnr$ffi$1.read(Unknown Source)
	at jnr.enxio.channels.Native.read(Native.java:95)
	at jnr.enxio.channels.NativeSocketChannel.read(NativeSocketChannel.java:68)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
	- locked <0x00000007835b5078> (a java.lang.Object)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
	- locked <0x00000007835b54a8> (a sun.nio.ch.ChannelInputStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140)
	at io.fabric8.maven.docker.access.hc.ApacheHttpClientDelegate.post(ApacheHttpClientDelegate.java:102)
	at io.fabric8.maven.docker.access.hc.ApacheHttpClientDelegate.post(ApacheHttpClientDelegate.java:108)
	at io.fabric8.maven.docker.access.hc.DockerAccessWithHcClient.stopContainer(DockerAccessWithHcClient.java:208)
	at io.fabric8.maven.docker.service.RunService.shutdown(RunService.java:446)
	at io.fabric8.maven.docker.service.RunService.stopStartedContainers(RunService.java:179)
	at io.fabric8.maven.docker.StopMojo.executeInternal(StopMojo.java:68)
	at io.fabric8.maven.docker.AbstractDockerMojo.execute(AbstractDockerMojo.java:222)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:956)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:290)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:194)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

Info

d-m-p version : 0.25.2
Maven version (mvn -v) :

Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z)
Maven home: /mnt/ebs1/jenkins/tools/hudson.tasks.Maven_MavenInstallation/apache-maven-3.5.3-with-takari/apache-maven-3.5.3
Java version: 1.8.0_172, vendor: Oracle Corporation
Java home: /mnt/ebs1/jenkins/tools/hudson.model.JDK/8u172-unlimited/jdk1.8.0_172/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.9.91-40.57.amzn1.x86_64", arch: "amd64", family: "unix"

Docker version :

Client:
 Version:	17.12.1-ce
 API version:	1.35
 Go version:	go1.9.4
 Git commit:	3dfb8343b139d6342acfd9975d7f1068b5b1c3d3
 Built:	Tue Apr  3 23:37:44 2018
 OS/Arch:	linux/amd64

Server:
 Engine:
  Version:	17.12.1-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.4
  Git commit:	7390fc6/17.12.1-ce
  Built:	Tue Apr  3 23:38:52 2018
  OS/Arch:	linux/amd64
  Experimental:	false

Issue Analytics

State:
Created 5 years ago
Reactions:2
Comments:9 (1 by maintainers)

Top GitHub Comments

2reactions

aaronjwhitesidecommented, May 22, 2018

Setting

<wait>
  <kill>2000</kill>
</wait>

made no difference…

Shouldn’t DMP have a read timeout? Jenkins killed the job after 2 hours (the integration test timeout value), but I feel like the DMP’s read timeout should have kicked in long before that?

0reactions

chontoncommented, Sep 10, 2019

Same problem as #552?

Top Results From Across the Web

Why Does My Docker Container Stop? - Tutorial Works

After a container finishes executing its default command, it will stop. ... A Docker container exiting doesn't usually indicate that something is wrong....

Cannot stop or restart a docker container - Stack Overflow

When I run docker ps -a I see the container as up and running. However the application inside it is malfunctioning so I...

Find out Why Your Docker Container Keeps Crashing

Your Docker container is exiting without a useful error message? Here are a few ways to find out what's happening.

Docker unresponsive after few minutes. Stopping Planning ...

When we try to stop containers through Powershell, the whole shutdown process hangs eventually. Even basic Docker commands are not responding.

6 ways to debug an exploding Docker container - Medium

Can't start your container at all? If you've got a initial command or entrypoint that immediately crashes, Docker will immediately shut it back...