Using `DockerComposeContainer` as a base class with spring boot not working across multiple test classes

I’m using the DockerComposeContainer in a base class with Spring Boot. The docker compose file contains Kafka/Zookeeper (please don’t ask why I’m not just using the KafkaContainer - I have my reasons and I cannot change to it).

My base class looks like this:

@DirtiesContext
@Testcontainers
public abstract class DockerComposeBase {
	@Container
	static final DockerComposeContainer<?> DOCKER_COMPOSE =
		new DockerComposeContainer<>(new File("docker-compose.yaml"))
			.withExposedService("zookeeper", 1, 2181, Wait.forListeningPort())
			.withExposedService("kafka", 1, 9092, Wait.forListeningPort());

	@DynamicPropertySource
	static void registerDynamicProperties(DynamicPropertyRegistry registry) {
		registry.add("spring.cloud.stream.kafka.binder.brokers", DockerComposeBase::getKafkaBrokers);
	}

	private static String getKafkaBrokers() {
		return String.format("%s:%s", DOCKER_COMPOSE.getServiceHost("kafka", 9092), DOCKER_COMPOSE.getServicePort("kafka", 9092));
	}
}

This is the docker-compose.yaml file:

version: '2'
services:
  zookeeper:
    image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
    command: [
      "sh", "-c",
      "bin/zookeeper-server-start.sh config/zookeeper.properties"
    ]
    ports:
      - "2181:2181"
    environment:
      LOG_DIR: /tmp/logs

  kafka:
    image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
    command: [
      "sh", "-c",
      "bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}"
    ]
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      LOG_DIR: "/tmp/logs"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

And then I’m extending this class in several other classes. The issue is that after the 1st test class runs & the 2nd test class tries to run it seems that it attempts to re-use the old container.

Here are some logs

1st test class cleaning up

2022-06-01 10:31:54.465  INFO 70769 --- [           main] 🐳 [docker/compose:1.29.2]               : Docker Compose container is running for command: down -v
2022-06-01 10:31:54.467  INFO 70769 --- [ream--704437905] 🐳 [docker/compose:1.29.2]               : STDERR: Stopping c2lkjr47cacj_kafka_1     ... 
2022-06-01 10:31:54.467  INFO 70769 --- [ream--704437905] 🐳 [docker/compose:1.29.2]               : STDERR: Stopping c2lkjr47cacj_zookeeper_1 ... 
2022-06-01 10:31:54.467  INFO 70769 --- [ream--704437905] 🐳 [docker/compose:1.29.2]               : STDERR: Stopping c2lkjr47cacj_kafka_1     ... done
2022-06-01 10:31:54.467  INFO 70769 --- [ream--704437905] 🐳 [docker/compose:1.29.2]               : STDERR: Stopping c2lkjr47cacj_zookeeper_1 ... done
2022-06-01 10:31:54.467  INFO 70769 --- [ream--704437905] 🐳 [docker/compose:1.29.2]               : STDERR: Removing c2lkjr47cacj_kafka_1     ... 
2022-06-01 10:31:54.468  INFO 70769 --- [ream--704437905] 🐳 [docker/compose:1.29.2]               : STDERR: Removing c2lkjr47cacj_zookeeper_1 ... 
2022-06-01 10:31:54.468  INFO 70769 --- [ream--704437905] 🐳 [docker/compose:1.29.2]               : STDERR: Removing c2lkjr47cacj_zookeeper_1 ... done
2022-06-01 10:31:54.468  INFO 70769 --- [ream--704437905] 🐳 [docker/compose:1.29.2]               : STDERR: Removing c2lkjr47cacj_kafka_1     ... done
2022-06-01 10:31:54.468  INFO 70769 --- [ream--704437905] 🐳 [docker/compose:1.29.2]               : STDERR: Removing network c2lkjr47cacj_default
2022-06-01 10:31:54.470  INFO 70769 --- [           main] 🐳 [docker/compose:1.29.2]               : Docker Compose has finished running

2nd test class starting up

2022-06-01 10:32:01.344  INFO 70769 --- [           main] 🐳 [docker/compose:1.29.2]               : Docker Compose container is running for command: up -d
2022-06-01 10:32:01.346  INFO 70769 --- [ream--492279488] 🐳 [docker/compose:1.29.2]               : STDERR: Creating network "c2lkjrlr6um5_default" with the default driver
2022-06-01 10:32:01.346  INFO 70769 --- [ream--492279488] 🐳 [docker/compose:1.29.2]               : STDERR: Creating c2lkjrlr6um5_zookeeper_1 ... 
2022-06-01 10:32:01.346  INFO 70769 --- [ream--492279488] 🐳 [docker/compose:1.29.2]               : STDERR: Creating c2lkjrlr6um5_zookeeper_1 ... done
2022-06-01 10:32:01.346  INFO 70769 --- [ream--492279488] 🐳 [docker/compose:1.29.2]               : STDERR: Creating c2lkjrlr6um5_kafka_1     ... 
2022-06-01 10:32:01.346  INFO 70769 --- [ream--492279488] 🐳 [docker/compose:1.29.2]               : STDERR: Creating c2lkjrlr6um5_kafka_1     ... done
2022-06-01 10:32:01.346  INFO 70769 --- [           main] 🐳 [docker/compose:1.29.2]               : Docker Compose has finished running
2022-06-01 10:32:01.347  INFO 70769 --- [           main] 🐳 [alpine/socat:1.7.4.3-r0]             : Creating container for image: alpine/socat:1.7.4.3-r0
2022-06-01 10:32:01.350 ERROR 70769 --- [           main] 🐳 [alpine/socat:1.7.4.3-r0]             : Could not start container

org.testcontainers.containers.ContainerLaunchException: Aborting attempt to link to container c2lkjr47cacj_kafka_1 as it is not running
        at org.testcontainers.containers.GenericContainer.applyConfiguration(GenericContainer.java:812)
        at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:374)
        at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:340)
        at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
        at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:338)
        at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:326)
        at org.testcontainers.containers.DockerComposeContainer.startAmbassadorContainers(DockerComposeContainer.java:331)
        at org.testcontainers.containers.DockerComposeContainer.start(DockerComposeContainer.java:177)
        at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:242)
        at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.access$200(TestcontainersExtension.java:229)
        at org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$null$1(TestcontainersExtension.java:59)
        at org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$getOrComputeIfAbsent$4(ExtensionValuesStore.java:86)
        at org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.computeValue(ExtensionValuesStore.java:223)
        at org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.get(ExtensionValuesStore.java:211)
        at org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.evaluate(ExtensionValuesStore.java:191)
        at org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.access$100(ExtensionValuesStore.java:171)
        at org.junit.jupiter.engine.execution.ExtensionValuesStore.getOrComputeIfAbsent(ExtensionValuesStore.java:89)
        at org.junit.jupiter.engine.execution.NamespaceAwareStore.getOrComputeIfAbsent(NamespaceAwareStore.java:53)
        at org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$beforeAll$2(TestcontainersExtension.java:59)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at org.testcontainers.junit.jupiter.TestcontainersExtension.beforeAll(TestcontainersExtension.java:59)
        at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$10(ClassBasedTestDescriptor.java:381)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:381)
        at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:205)
        at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:80)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:148)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
        at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
        at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
        at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
        at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

You can see that the 2nd run is attempting to connect to the c2lkjr47cacj_kafka_1 container, which was the container name from the 1st run. Instead it should be trying to connect to c2lkjrlr6um5_kafka_1.

Is there some setting i’m missing in the configuration?

[kiview]

I don’t see a reason why it should not work with a non-confluent Kafka image, that follows a similar structure as the Confluent Kafka image. And since we are Testcontainers maintainers, it’s fine to at least try it out if we recommend it, the docs just mention the default image that will be used (and against which we test and integrate).

In Spring-Boot, but can use Testcontainers in a very convenient way for running your app locally following this pattern: https://bsideup.github.io/posts/local_development_with_testcontainers/

If you absolutely have to use Docker Compose, you have to somehow code your own abstraction that ensures the same stability as the KafkaContainer implementation (you can check the implementation: https://github.com/testcontainers/testcontainers-java/blob/main/modules/kafka/src/main/java/org/testcontainers/containers/KafkaContainer.java#L104).

[edeandrea]

That may be a winner!