TestContainers Hang After First Run

I added a second TestContainer into our class. On the first run, everything will pass with no problems. The second run will hang on the command to permit Docker to bind. In order for it to work, I have to stop the java process, restart docker, and rerun it. At that point, it will work on the first run but not the second run again.

Here are the logs where I got the issue from:

09:01:42.591 [main] DEBUG 🐳 [localstack/localstack:latest] - Starting container: localstack/localstack:latest
09:01:42.592 [main] DEBUG 🐳 [localstack/localstack:latest] - Trying to start container: localstack/localstack:latest (attempt 1/1)
09:01:42.592 [main] DEBUG 🐳 [localstack/localstack:latest] - Starting container: localstack/localstack:latest
09:01:42.592 [main] INFO 🐳 [localstack/localstack:latest] - Creating container for image: localstack/localstack:latest
09:01:42.592 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: localstack/localstack:latest at registry: index.docker.io
09:01:42.592 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Cached auth found: [AuthConfig{username=XXXX, password=hidden non-blank value, auth=blank, email=null, registryAddress=index.docker.io, registryToken=blank}]
09:01:42.592 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [AuthConfig{username=XXXX, password=hidden non-blank value, auth=blank, email=null, registryAddress=index.docker.io, registryToken=blank}]
09:01:42.596 [main] DEBUG org.testcontainers.utility.PrefixingImageNameSubstitutor - No prefix is configured
09:01:42.596 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for alpine:3.5 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
09:01:42.596 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: alpine:3.5
09:01:42.602 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: alpine:3.5 at registry: index.docker.io
09:01:42.602 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Cached auth found: [AuthConfig{username=XXXX, password=hidden non-blank value, auth=blank, email=null, registryAddress=index.docker.io, registryToken=blank}]
09:01:42.602 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [AuthConfig{username=XXXX, password=hidden non-blank value, auth=blank, email=null, registryAddress=index.docker.io, registryToken=blank}]
09:01:42.611 [main] DEBUG org.testcontainers.utility.MountableFile - Copying classpath resource(s) from jar:file:/Users/XXXXX/.m2/repository/org/testcontainers/testcontainers/1.15.3/testcontainers-1.15.3.jar!/org/testcontainers/utility/ResourceReaper.class to /tmp/.testcontainers-tmp-3628711180189158291 to permit Docker to bind
09:01:42.611 [main] DEBUG org.testcontainers.utility.MountableFile - Copying resource org/testcontainers/utility/ResourceReaper.class from JAR file /Users/XXXXX/.m2/repository/org/testcontainers/testcontainers/1.15.3/testcontainers-1.15.3.jar
09:01:42.626 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl@27d57a2c[name=<null>,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,healthcheck=<null>,argsEscaped=<null>,entrypoint=<null>,image=alpine:3.5,volumes=com.github.dockerjava.api.model.Volumes@2ff95fc6,workingDir=<null>,macAddress=<null>,onBuild=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@9d1a267,stopSignal=<null>,stopTimeout=<null>,hostConfig=HostConfig(binds=[/private/tmp/.testcontainers-tmp-3628711180189158291:/dummy:ro], blkioWeight=null, blkioWeightDevice=null, blkioDeviceReadBps=null, blkioDeviceWriteBps=null, blkioDeviceReadIOps=null, blkioDeviceWriteIOps=null, memorySwappiness=null, nanoCPUs=null, capAdd=null, capDrop=null, containerIDFile=null, cpuPeriod=null, cpuRealtimePeriod=null, cpuRealtimeRuntime=null, cpuShares=null, cpuQuota=null, cpusetCpus=null, cpusetMems=null, devices=null, deviceCgroupRules=null, deviceRequests=null, diskQuota=null, dns=null, dnsOptions=null, dnsSearch=null, extraHosts=null, groupAdd=null, ipcMode=null, cgroup=null, links=[], logConfig=LogConfig(type=null, config=null), lxcConf=null, memory=null, memorySwap=null, memoryReservation=null, kernelMemory=null, networkMode=null, oomKillDisable=null, init=null, autoRemove=null, oomScoreAdj=null, portBindings=null, privileged=null, publishAllPorts=null, readonlyRootfs=null, restartPolicy=null, ulimits=null, cpuCount=null, cpuPercent=null, ioMaximumIOps=null, ioMaximumBandwidth=null, volumesFrom=null, mounts=null, pidMode=null, isolation=null, securityOpts=null, storageOpt=null, cgroupParent=null, volumeDriver=null, shmSize=null, pidsLimit=null, runtime=null, tmpFs=null, utSMode=null, usernsMode=null, sysctls=null, consoleSize=null),labels={org.testcontainers=true, org.testcontainers.sessionId=d5b6c43c-6fcc-4ea6-aa82-0b897b40917e},shell=<null>,networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=AuthConfig(username=XXXX, email=null, registryAddress=index.docker.io, stackOrchestrator=null)]
09:01:42.701 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 2e6a5329860ca4325e1ea4c4238a43569a83772ad36caeedd2b8bd2bdb7f886e

• Docker Desktop: 3.5.1
• Docker Engine: 20.10.7
• TestContainers Version: 1.15.3

Here is a snippet of our class

@SpringBootTest
@Testcontainers
class RandomTest {
    private static final DockerImageName localstackImage = DockerImageName.parse("localstack/localstack:latest");

    @Container
    private static final LocalStackContainer localStackContainer = new LocalStackContainer(localstackImage)
            .withServices(LocalStackContainer.Service.SQS);
    
    @Container
    private static final MongoDBContainer mongoDBContainer = new MongoDBContainer(DockerImageName.parse("mongo:4.4.6-xenial"));

    @DynamicPropertySource
    static void properties(DynamicPropertyRegistry dynamicPropertyRegistry) {
        dynamicPropertyRegistry.add("SQS_SERVICE_ENDPOINT", () -> localStackContainer.getEndpointConfiguration(LocalStackContainer.Service.SQS).getServiceEndpoint());
    }
}

[bsideup]

For some reason the container running the alpine images stays in its CREATED state

This sounds like a bug in Docker, not Testcontainers, as we do not control the transitioning from CREATED to RUNNING. If CreateContainerCmd gets stuck, this would mean that there is something in Docker’s API that blocks it from returning container’s information, and we cannot proceed without it.

Attempting to delete the container does not work

This is also a strong signal that something is terribly wrong inside the Docker daemon (deadlock? File system-related issues?)

Have you reported it to Docker yet?

[ShawnRG]

Update: I have completely purged and re-installed Docker from my system using the uninstall feature in Docker Desktop and removing it via Homebrew.

Now I do not get the issue anoymore, I have tested two previously failing test scenarios and they seem to be working consistently now.