Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cannot connect to containers when using docker-in-docker

See original GitHub issue

I’m using docker-in-docker inside of kubernetes in jenkins. My containers appear to start up fine, but the host/port never becomes accessible and so testcontainers cannot connect to them.

Everything works fine when I run the tests locally using docker desktop, it’s only on when running in jenkins using docker-in-docker that they fail.

The container ip address is 172.17.0.2 but it’s trying to connect to 172.17.0.1 which is the gateway. Is that normal?

I’ve tested this on both testcontainers 1.16.0 and 1.16.1 and they both fail. This could be related to #4594, except my problem happens on 1.16.0 as well. This also might be related to #4591 although I am not using TESTCONTAINERS_HOST_OVERRIDE

Here is my test case where I start up an elasticsearch container, log the ipAddress and hostAddress, and then run docker inspect and log the results:

class ElasticsearchContainerTest {

    companion object {
        private val logger = KontextLogger.logger {}
        private const val ELASTICSEARCH_VERSION = "6.7.0"
        private val container = ElasticsearchContainer(
            DockerImageName.parse("{REDACTED CONTAINER MIRROR URL}/elasticsearch")
                .withTag(ELASTICSEARCH_VERSION)
                .asCompatibleSubstituteFor("docker.elastic.co/elasticsearch/elasticsearch")
        ).withLogConsumer(Slf4jLogConsumer(KontextLogger.logger {}))

        @BeforeAll
        @JvmStatic
        fun beforeAll() {
            container.start()
        }

        @AfterAll
        @JvmStatic
        fun afterAll() {
            logger.error("Container IP address: ${container.containerIpAddress}")
            logger.error("Container  url: ${container.httpHostAddress}")

            val dockerInspect = container.dockerClient.inspectContainerCmd(container.containerId).exec()
            val mapper = ObjectMapper().disable(SerializationFeature.FAIL_ON_EMPTY_BEANS)
            val jsonOutput = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(dockerInspect)
            logger.error("docker inspect elasticsearch: $jsonOutput")

            container.stop()
        }
    }

    @Test
    fun `check that container is running`() {
        Assert.assertTrue("Container is not running", container.isRunning)
    }
}

The test fails due to Timed out waiting for URL to be accessible (http://172.17.0.1:49154/ should return HTTP 200:

:integrationTest > 0 tests completed> :integrationTest > Executing test com.example.test.Elasticsea
 ElasticsearchContainerTest > initializationError FAILED
     org.testcontainers.containers.ContainerLaunchException: Container startup failed
         at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:336)
         at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
         at com.example.test.ElasticsearchContainerTest$Companion.beforeAll(ElasticsearchContainerTest.kt:28)
         at com.example.test.ElasticsearchContainerTest.beforeAll(ElasticsearchContainerTest.kt)
 
         Caused by:
         org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
             at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
             at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
             ... 3 more
 
             Caused by:
             org.testcontainers.containers.ContainerLaunchException: Could not create/start container
                 at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:525)
                 at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
                 at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
                 ... 4 more
 
                 Caused by:
                 org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://172.17.0.1:49154/ should return HTTP 200)
                     at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:264)
                     at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
                     at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
                     at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
                     ... 6 more

Here is the output of docker inspect:

{
 "Args" : [ "eswrapper" ],
 "Config" : {
   "healthcheck" : null,
   "AttachStderr" : false,
   "AttachStdin" : false,
   "AttachStdout" : false,
   "Cmd" : [ "eswrapper" ],
   "Domainname" : "",
   "Entrypoint" : [ "/usr/local/bin/docker-entrypoint.sh" ],
   "Env" : [ "discovery.type=single-node", "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ELASTIC_CONTAINER=true", "JAVA_HOME=/opt/jdk-12" ],
   "ExposedPorts" : {
     "9200/tcp" : { },
     "9300/tcp" : { }
   },
   "Hostname" : "8d8c72fac5a9",
   "Image" : "{REDACTED CONTAINER MIRROR URL}/open-source-mirror/dockerhub/elasticsearch:6.7.0",
   "Labels" : {
     "license" : "Elastic License",
     "org.label-schema.build-date" : "20190305",
     "org.label-schema.license" : "GPLv2",
     "org.label-schema.name" : "elasticsearch",
     "org.label-schema.schema-version" : "1.0",
     "org.label-schema.url" : "https://www.elastic.co/products/elasticsearch",
     "org.label-schema.vcs-url" : "https://github.com/elastic/elasticsearch",
     "org.label-schema.vendor" : "Elastic",
     "org.label-schema.version" : "6.7.0",
     "org.testcontainers" : "true",
     "org.testcontainers.sessionId" : "671d3a01-4780-4f87-af21-ac568bafdc6f"
   },
   "MacAddress" : null,
   "NetworkDisabled" : null,
   "OnBuild" : null,
   "OpenStdin" : false,
   "PortSpecs" : null,
   "StdinOnce" : false,
   "Tty" : false,
   "User" : "",
   "Volumes" : null,
   "WorkingDir" : "/usr/share/elasticsearch",
   "Healthcheck" : null
 },
 "Created" : "2021-10-20T20:26:52.934006535Z",
 "Driver" : "overlay2",
 "ExecDriver" : null,
 "HostConfig" : {
   "Binds" : [ ],
   "BlkioWeight" : 0,
   "BlkioWeightDevice" : null,
   "BlkioDeviceReadBps" : null,
   "BlkioDeviceWriteBps" : null,
   "BlkioDeviceReadIOps" : null,
   "BlkioDeviceWriteIOps" : null,
   "MemorySwappiness" : null,
   "NanoCpus" : 0,
   "CapAdd" : null,
   "CapDrop" : null,
   "ContainerIDFile" : "",
   "CpuPeriod" : 0,
   "CpuRealtimePeriod" : 0,
   "CpuRealtimeRuntime" : 0,
   "CpuShares" : 0,
   "CpuQuota" : 0,
   "CpusetCpus" : "",
   "CpusetMems" : "",
   "Devices" : null,
   "DeviceCgroupRules" : null,
   "DeviceRequests" : null,
   "DiskQuota" : null,
   "Dns" : null,
   "DnsOptions" : null,
   "DnsSearch" : null,
   "ExtraHosts" : [ ],
   "GroupAdd" : null,
   "IpcMode" : "shareable",
   "Cgroup" : "",
   "Links" : null,
   "LogConfig" : {
     "Type" : "json-file",
     "Config" : { }
   },
   "LxcConf" : null,
   "Memory" : 0,
   "MemorySwap" : 0,
   "MemoryReservation" : 0,
   "KernelMemory" : 0,
   "NetworkMode" : "default",
   "OomKillDisable" : false,
   "Init" : null,
   "AutoRemove" : false,
   "OomScoreAdj" : 0,
   "PortBindings" : {
     "9200/tcp" : [ {
       "HostIp" : "",
       "HostPort" : ""
     } ],
     "9300/tcp" : [ {
       "HostIp" : "",
       "HostPort" : ""
     } ]
   },
   "Privileged" : false,
   "PublishAllPorts" : false,
   "ReadonlyRootfs" : false,
   "RestartPolicy" : {
     "MaximumRetryCount" : 0,
     "Name" : ""
   },
   "Ulimits" : null,
   "CpuCount" : 0,
   "CpuPercent" : 0,
   "IOMaximumIOps" : 0,
   "IOMaximumBandwidth" : 0,
   "VolumesFrom" : [ ],
   "Mounts" : null,
   "PidMode" : "",
   "Isolation" : null,
   "SecurityOpt" : null,
   "StorageOpt" : null,
   "CgroupParent" : "",
   "VolumeDriver" : "",
   "ShmSize" : 67108864,
   "PidsLimit" : null,
   "Runtime" : "runc",
   "Tmpfs" : null,
   "UTSMode" : "",
   "UsernsMode" : "",
   "Sysctls" : null,
   "ConsoleSize" : [ 0, 0 ]
 },
 "HostnamePath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hostname",
 "HostsPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hosts",
 "LogPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457-json.log",
 "Id" : "8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457",
 "SizeRootFs" : null,
 "Image" : "sha256:02982be5777dea7bbf9573e1993c936b8add775d303bafd638924cf9635e8c38",
 "MountLabel" : "",
 "Name" : "/focused_roentgen",
 "RestartCount" : 0,
 "NetworkSettings" : {
   "Bridge" : "",
   "SandboxID" : "786b67295fe24a75cd4edcf66b69383e72e2972f80bf6b6a801f9c020d14bf27",
   "HairpinMode" : false,
   "LinkLocalIPv6Address" : "",
   "LinkLocalIPv6PrefixLen" : 0,
   "Ports" : {
     "9200/tcp" : [ {
       "HostIp" : "0.0.0.0",
       "HostPort" : "49154"
     }, {
       "HostIp" : "::",
       "HostPort" : "49154"
     } ],
     "9300/tcp" : [ {
       "HostIp" : "0.0.0.0",
       "HostPort" : "49153"
     }, {
       "HostIp" : "::",
       "HostPort" : "49153"
     } ]
   },
   "SandboxKey" : "/run/user/1000/docker/netns/786b67295fe2",
   "SecondaryIPAddresses" : null,
   "SecondaryIPv6Addresses" : null,
   "EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
   "Gateway" : "172.17.0.1",
   "PortMapping" : null,
   "GlobalIPv6Address" : "",
   "GlobalIPv6PrefixLen" : 0,
   "IPAddress" : "172.17.0.2",
   "IPPrefixLen" : 16,
   "IPv6Gateway" : "",
   "MacAddress" : "02:42:ac:11:00:02",
   "Networks" : {
     "bridge" : {
       "IPAMConfig" : null,
       "Links" : null,
       "Aliases" : null,
       "NetworkID" : "aaeb65fa1c11b6e4e4350d4b68b34d38796f2170d26073bdb0282fe14f4dc6a6",
       "EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
       "Gateway" : "172.17.0.1",
       "IPAddress" : "172.17.0.2",
       "IPPrefixLen" : 16,
       "IPv6Gateway" : "",
       "GlobalIPv6Address" : "",
       "GlobalIPv6PrefixLen" : 0,
       "MacAddress" : "02:42:ac:11:00:02"
     }
   }
 },
 "Path" : "/usr/local/bin/docker-entrypoint.sh",
 "ProcessLabel" : "",
 "ResolvConfPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/resolv.conf",
 "ExecIDs" : null,
 "State" : {
   "oomkilled" : false,
   "pidLong" : 873,
   "exitCodeLong" : 0,
   "Status" : "running",
   "Running" : true,
   "Paused" : false,
   "Restarting" : false,
   "OOMKilled" : false,
   "Dead" : false,
   "Pid" : 873,
   "ExitCode" : 0,
   "Error" : "",
   "StartedAt" : "2021-10-20T20:26:53.468541843Z",
   "FinishedAt" : "0001-01-01T00:00:00Z",
   "Health" : null
 },
 "Volumes" : null,
 "VolumesRW" : null,
 "Node" : null,
 "Mounts" : [ ],
 "GraphDriver" : {
   "Name" : "overlay2",
   "Data" : {
     "RootDir" : null,
     "DeviceId" : null,
     "DeviceName" : null,
     "DeviceSize" : null,
     "dir" : null
   }
 },
 "Platform" : "linux",
 "Path" : "/usr/local/bin/docker-entrypoint.sh",
 "Args" : [ "eswrapper" ],
 "ProcessLabel" : "",
 "Mounts" : [ ],
 "Platform" : "linux",
 "Config" : {
   "Hostname" : "8d8c72fac5a9",
   "Domainname" : "",
   "User" : "",
   "AttachStdin" : false,
   "AttachStdout" : false,
   "AttachStderr" : false,
   "ExposedPorts" : {
     "9200/tcp" : { },
     "9300/tcp" : { }
   },
   "Tty" : false,
   "OpenStdin" : false,
   "StdinOnce" : false,
   "Env" : [ "discovery.type=single-node", "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ELASTIC_CONTAINER=true", "JAVA_HOME=/opt/jdk-12" ],
   "Cmd" : [ "eswrapper" ],
   "Image" : "{REDACTED CONAINER MIRROR URL}/open-source-mirror/dockerhub/elasticsearch:6.7.0",
   "Volumes" : null,
   "WorkingDir" : "/usr/share/elasticsearch",
   "Entrypoint" : [ "/usr/local/bin/docker-entrypoint.sh" ],
   "OnBuild" : null,
   "Labels" : {
     "license" : "Elastic License",
     "org.label-schema.build-date" : "20190305",
     "org.label-schema.license" : "GPLv2",
     "org.label-schema.name" : "elasticsearch",
     "org.label-schema.schema-version" : "1.0",
     "org.label-schema.url" : "https://www.elastic.co/products/elasticsearch",
     "org.label-schema.vcs-url" : "https://github.com/elastic/elasticsearch",
     "org.label-schema.vendor" : "Elastic",
     "org.label-schema.version" : "6.7.0",
     "org.testcontainers" : "true",
     "org.testcontainers.sessionId" : "671d3a01-4780-4f87-af21-ac568bafdc6f"
   }
 },
 "Driver" : "overlay2",
 "AppArmorProfile" : "",
 "HostsPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hosts",
 "HostnamePath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hostname",
 "GraphDriver" : {
   "Data" : {
     "LowerDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5-init/diff:/home/rootless/.local/share/docker/overlay2/1e71332ad2a429a4d06d157b6c4e2c4401da2b44b640d412557feb3c39452010/diff:/home/rootless/.local/share/docker/overlay2/1302968f5de6c81580eadb218555c0bc49bd3af7c766e77cbdb0ea9f02371d2a/diff:/home/rootless/.local/share/docker/overlay2/3b2b6c22a55172df81ac6af6f439aa3ec1cfe0576269d9fab8c25772e49a65ca/diff:/home/rootless/.local/share/docker/overlay2/42a8e273145d92d47a2c6827fa4d276800e98189da117ca69b25808ce1013edc/diff:/home/rootless/.local/share/docker/overlay2/be84a76283d29eac19bb5e262a4b26578218877d2302cea775fc57c29e195fb0/diff:/home/rootless/.local/share/docker/overlay2/05a94b779ac49607562a956de5a274e5cd000747ef966e612f156ba3f5018cb3/diff:/home/rootless/.local/share/docker/overlay2/57967e2c3b2027a4b67e5d97544d36b11573228ae195fb1cb0799b0ca954c669/diff",
     "MergedDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/merged",
     "UpperDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/diff",
     "WorkDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/work"
   },
   "Name" : "overlay2"
 },
 "Image" : "sha256:02982be5777dea7bbf9573e1993c936b8add775d303bafd638924cf9635e8c38",
 "Created" : "2021-10-20T20:26:52.934006535Z",
 "Name" : "/focused_roentgen",
 "RestartCount" : 0,
 "NetworkSettings" : {
   "Bridge" : "",
   "SandboxID" : "786b67295fe24a75cd4edcf66b69383e72e2972f80bf6b6a801f9c020d14bf27",
   "HairpinMode" : false,
   "LinkLocalIPv6Address" : "",
   "LinkLocalIPv6PrefixLen" : 0,
   "Ports" : {
     "9200/tcp" : [ {
       "HostIp" : "0.0.0.0",
       "HostPort" : "49154"
     }, {
       "HostIp" : "::",
       "HostPort" : "49154"
     } ],
     "9300/tcp" : [ {
       "HostIp" : "0.0.0.0",
       "HostPort" : "49153"
     }, {
       "HostIp" : "::",
       "HostPort" : "49153"
     } ]
   },
   "SandboxKey" : "/run/user/1000/docker/netns/786b67295fe2",
   "SecondaryIPAddresses" : null,
   "SecondaryIPv6Addresses" : null,
   "EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
   "Gateway" : "172.17.0.1",
   "GlobalIPv6Address" : "",
   "GlobalIPv6PrefixLen" : 0,
   "IPAddress" : "172.17.0.2",
   "IPPrefixLen" : 16,
   "IPv6Gateway" : "",
   "MacAddress" : "02:42:ac:11:00:02",
   "Networks" : {
     "bridge" : {
       "IPAMConfig" : null,
       "Links" : null,
       "Aliases" : null,
       "NetworkID" : "aaeb65fa1c11b6e4e4350d4b68b34d38796f2170d26073bdb0282fe14f4dc6a6",
       "EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
       "Gateway" : "172.17.0.1",
       "IPAddress" : "172.17.0.2",
       "IPPrefixLen" : 16,
       "IPv6Gateway" : "",
       "GlobalIPv6Address" : "",
       "GlobalIPv6PrefixLen" : 0,
       "MacAddress" : "02:42:ac:11:00:02",
       "DriverOpts" : null
     }
   }
 },
 "MountLabel" : "",
 "State" : {
   "Status" : "running",
   "Running" : true,
   "Paused" : false,
   "Restarting" : false,
   "OOMKilled" : false,
   "Dead" : false,
   "Pid" : 873,
   "ExitCode" : 0,
   "Error" : "",
   "StartedAt" : "2021-10-20T20:26:53.468541843Z",
   "FinishedAt" : "0001-01-01T00:00:00Z"
 },
 "ResolvConfPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/resolv.conf",
 "ExecIDs" : null,
 "HostConfig" : {
   "Binds" : [ ],
   "ContainerIDFile" : "",
   "LogConfig" : {
     "Type" : "json-file",
     "Config" : { }
   },
   "NetworkMode" : "default",
   "PortBindings" : {
     "9200/tcp" : [ {
       "HostIp" : "",
       "HostPort" : ""
     } ],
     "9300/tcp" : [ {
       "HostIp" : "",
       "HostPort" : ""
     } ]
   },
   "RestartPolicy" : {
     "Name" : "",
     "MaximumRetryCount" : 0
   },
   "AutoRemove" : false,
   "VolumeDriver" : "",
   "VolumesFrom" : [ ],
   "CapAdd" : null,
   "CapDrop" : null,
   "CgroupnsMode" : "host",
   "Dns" : null,
   "DnsOptions" : null,
   "DnsSearch" : null,
   "ExtraHosts" : [ ],
   "GroupAdd" : null,
   "IpcMode" : "shareable",
   "Cgroup" : "",
   "Links" : null,
   "OomScoreAdj" : 0,
   "PidMode" : "",
   "Privileged" : false,
   "PublishAllPorts" : false,
   "ReadonlyRootfs" : false,
   "SecurityOpt" : null,
   "UTSMode" : "",
   "UsernsMode" : "",
   "ShmSize" : 67108864,
   "Runtime" : "runc",
   "ConsoleSize" : [ 0, 0 ],
   "Isolation" : "",
   "CpuShares" : 0,
   "Memory" : 0,
   "NanoCpus" : 0,
   "CgroupParent" : "",
   "BlkioWeight" : 0,
   "BlkioWeightDevice" : null,
   "BlkioDeviceReadBps" : null,
   "BlkioDeviceWriteBps" : null,
   "BlkioDeviceReadIOps" : null,
   "BlkioDeviceWriteIOps" : null,
   "CpuPeriod" : 0,
   "CpuQuota" : 0,
   "CpuRealtimePeriod" : 0,
   "CpuRealtimeRuntime" : 0,
   "CpusetCpus" : "",
   "CpusetMems" : "",
   "Devices" : null,
   "DeviceCgroupRules" : null,
   "DeviceRequests" : null,
   "KernelMemory" : 0,
   "KernelMemoryTCP" : 0,
   "MemoryReservation" : 0,
   "MemorySwap" : 0,
   "MemorySwappiness" : null,
   "OomKillDisable" : false,
   "PidsLimit" : null,
   "Ulimits" : null,
   "CpuCount" : 0,
   "CpuPercent" : 0,
   "IOMaximumIOps" : 0,
   "IOMaximumBandwidth" : 0,
   "MaskedPaths" : [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ],
   "ReadonlyPaths" : [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ]
 },
 "Id" : "8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457",
 "LogPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457-json.log"
}

Some additional log lines which might help:

INFO  org.testcontainers.containers.wait.strategy.HttpWaitStrategy - /focused_roentgen: Waiting for 120 seconds for URL: http://172.17.0.1:49154/ (where port 49154 maps to container port 9200)

Log output from the failed container:

[INFO ][o.e.e.NodeEnvironment    ] [dP6fGhj] using [1] data paths, mounts [[/ (overlay)]], net usable_space [79.4gb], net total_space [242.2gb], types [overlay]
[INFO ][o.e.e.NodeEnvironment    ] [dP6fGhj] heap size [990.7mb], compressed ordinary object pointers [true]
[INFO ][o.e.n.Node               ] [dP6fGhj] node name derived from node ID [dP6fGhj8TES6gRpq72sI0Q]; set [node.name] to override
[INFO ][o.e.n.Node               ] [dP6fGhj] version[6.7.0], pid[1], build[default/docker/8453f77/2019-03-21T15:32:29.844721Z], OS[Linux/5.4.0-1028-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[INFO ][o.e.n.Node               ] [dP6fGhj] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2290863965764670913, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [aggs-matrix-stats]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [analysis-common]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [ingest-common]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [ingest-geoip]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [ingest-user-agent]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [lang-expression]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [lang-mustache]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [lang-painless]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [mapper-extras]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [parent-join]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [percolator]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [rank-eval]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [reindex]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [repository-url]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [transport-netty4]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [tribe]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-ccr]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-core]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-deprecation]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-graph]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-ilm]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-logstash]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-ml]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-monitoring]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-rollup]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-security]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-sql]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-upgrade]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] loaded module [x-pack-watcher]
[INFO ][o.e.p.PluginsService     ] [dP6fGhj] no plugins loaded
[INFO ][o.e.x.s.a.s.FileRolesStore] [dP6fGhj] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[INFO ][o.e.x.m.p.l.CppLogMessageHandler] [dP6fGhj] [controller/88] [Main.cc@109] controller (64 bit): Version 6.7.0 (Build d74ae2ac01b10d) Copyright (c) 2019 Elasticsearch BV
[INFO ][o.e.d.DiscoveryModule    ] [dP6fGhj] using discovery type [single-node] and host providers [settings]
[INFO ][o.e.n.Node               ] [dP6fGhj] initialized
[INFO ][o.e.n.Node               ] [dP6fGhj] starting ...
[INFO ][o.e.t.TransportService   ] [dP6fGhj] publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}
[INFO ][o.e.h.n.Netty4HttpServerTransport] [dP6fGhj] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[INFO ][o.e.n.Node               ] [dP6fGhj] started
[2021-10-20T20:28:53.705Z]     [2021-10-20T20:27:00,528][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [dP6fGhj] Failed to clear cache for realms [[]]
[INFO ][o.e.g.GatewayService     ] [dP6fGhj] recovered [0] indices into cluster_state
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.watches] for index patterns [.watches*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[INFO ][o.e.l.LicenseService     ] [dP6fGhj] license [2dcb2824-6c74-4c3a-b8ac-d06233221f44] mode [basic] - valid

When I run a similar test for postgres, it passes but that’s only because the postgres testcontainer uses a log watching wait strategy and doesn’t try to connect to the exposed port. If I try to actually connect to the postgres container it fails as well.

My kubernetes podspec is like this:

    podSpec:
      spec:
        containers:
        - name: main
          image: <redacted, but my image is based on debian>
          command:
            - cat
          tty: true
          volumeMounts:
            - mountPath: /var/run
              name: docker-socket
          env:
            - name: TESTCONTAINERS_RYUK_DISABLED
              value: true
        - name: dind-daemon
          image: docker:20.10.9-dind-rootless
          securityContext:
            privileged: true
            readOnlyRootFilesystem: false
          volumeMounts:
            - name: docker-socket
              mountPath: /run/user/1000
        volumes:
          - name: docker-socket
            emptyDir: {}

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:1
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
jameslaminecommented, Oct 21, 2021

Setting TESTCONTAINERS_HOST_OVERRIDE to localhost fixed the issue. So it’s not automatically detecting that correctly. The fix worked with testcontainers 1.16.1. I didn’t test it with 1.16.0.

0reactions
alpeshjikadracommented, Mar 2, 2022

I am also facing the similar issue with testcontainers 1.16.0, 1.16.1, 1.16.2 and 1.16.3. With kubernetes version 1.20 and 1.21.9.And adding TESTCONTAINERS_HOST_OVERRIDE=localhost is not working

Solution work for me is : I set testcontainers gateway ip address instead of localhost, something like TESTCONTAINERS_HOST_OVERRIDE=172.17.0.1.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Docker can't connect to Container exposed port - Stack Overflow
Check the docker inspect or docker ps results to ensure that you have your port exposed correctly · Try to connect to it...
Read more >
Can't connect to container on localhost with port mapping
I'm trying torun a windows container on my windows machine. and would like to connect to it via localhost. For example the container...
Read more >
4 Reasons Why Your Docker Containers Can't Talk to Each ...
Containers can only communicate with each other if they share a network. Containers that don't share a network cannot communicate with one another....
Read more >
Connection refused? Docker networking and how it impacts ...
Can't connect to the server running in your container? Let's see why, and how to fix it, starting with an example.
Read more >
How to Resolve the “cannot connect to the Docker daemon ...
How to Resolve the “cannot connect to the Docker daemon” Error · Method 1: Check the Docker Engine · Method 2: Assign Ownership...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Timed out waiting for container port to open

Next page

After upgrade to 1.16.1 JUnit tests fail on AWS CodeBuild