`confluentinc/cp-kafka:6.2.2` and `7.0.1` don't work with `KafkaContainer`

This issue has been originally reported by @simonbasle in our Testcontianers Slack.

While we initially thought this is related to GitHub Actions, it seems this is a general incompatibility of KafkaContainer with cp-kafka images >= 6.2.2.

During container startup, our InternalCommandPortListeningCheck will fail with logs along the lines of:

17:57:13.528 WARN  org.testcontainers.containers.wait.internal.InternalCommandPortListeningCheck - An exception while executing the internal check: Container.ExecResult(exitCode=137, stdout=, stderr=Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connection to 127.0.0.1 failed: Connection refused.
Ncat: Trying next address...
Ncat: Cannot assign requested address.
/bin/bash: connect: Cannot assign requested address
/bin/bash: /dev/tcp/localhost/9093: Cannot assign requested address

Subsequently, our exec is failing:

17:57:13.532 INFO  🐳 [confluentinc/cp-kafka:6.2.2] - Container confluentinc/cp-kafka:6.2.2 started in PT2.1713743S
17:57:13.532 DEBUG org.testcontainers.containers.ExecInContainerPattern - /hungry_newton: Running "exec" command: kafka-configs --alter --bootstrap-server BROKER://994010ec6ac8:9092 --entity-type brokers --entity-name 1 --add-config advertised.listeners=[PLAINTEXT://127.0.0.1:62036,BROKER://994010ec6ac8:9092]
17:57:13.669 ERROR 🐳 [confluentinc/cp-kafka:6.2.2] - Could not start container
com.github.dockerjava.api.exception.ConflictException: Status 409: {"message":"Container 994010ec6ac8c3cb2ff554a849fdba13b8e38ec6798585d648a766b601fbfd6c is not running"}

[sergey-morenets]

Hi

Confirmed that TestContainers 16.3 fails to start even GenericContainer for cp-kafka:7.1.1:

kafka = new GenericContainer<>(DockerImageName.parse("confluentinc/cp-kafka:7.1.1"))
	.withEnv("KAFKA_PROCESS_ROLES", "broker,controller").withEnv("KAFKA_NODE_ID", "1")
	.withEnv("KAFKA_LISTENER_SECURITY_PROTOCOL_MAP", "CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT")
	.withEnv("KAFKA_ADVERTISED_LISTENERS", "PLAINTEXT://kafka:9092")
	.withEnv("KAFKA_CONTROLLER_QUORUM_VOTERS", "1@kafka:9093")
	.withEnv("KAFKA_LISTENERS", "PLAINTEXT://kafka:9092,CONTROLLER://kafka:9093")
	.withEnv("KAFKA_CONTROLLER_LISTENER_NAMES", "CONTROLLER")
	.withExposedPorts(9092);

Log output:

[testcontainers-wait-0] WARN org.testcontainers.containers.wait.internal.InternalCommandPortListeningCheck - An exception while executing the internal check: Container.ExecResult(exitCode=137, stdout=, stderr=Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connection to 127.0.0.1 failed: Connection refused.
Ncat: Trying next address...
Ncat: Cannot assign requested address.
/bin/bash: connect: Cannot assign requested address
/bin/bash: /dev/tcp/localhost/9092: Cannot assign requested address
)
[main] INFO 🐳 [confluentinc/cp-kafka:7.1.1] - Container confluentinc/cp-kafka:7.1.1 started in PT0.6145648S

[simonbasle]

Thanks for your help in debugging this @kiview !

On a side note, about the Ncat log, there is still some such logs visible if you enable trace logging, even with 6.2.1… That’s because of some polling aspect, I’m assuming.

So right after these logs, we get a success with 6.2.1:

    /bin/bash: connect: Cannot assign requested address
    /bin/bash: /dev/tcp/localhost/9093: Cannot assign requested address

    [2021-12-13 12:57:56,321] [testcontainers-wait-0] [org.testcontainers.containers.wait.internal.InternalCommandPortListeningCheck] TRACE Check for [9093, 2181] took PT8.449S. Result code '0', stdout message: '/proc/net/tcp:   3: 00000000:2385 00000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 36725 1 0000000000000000 100 0 0 10 0                     
    /proc/net/tcp:   4: 00000000:0885 00000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 36445 1 0000000000000000 100 0 0 10 0                     
    /proc/net/tcp:   5: 0100007F:AC38 0100007F:0885 01 00000000:00000000 00:00000000 00000000  1000        0 36641 1 0000000000000000 20 4 10 10 -1                    
    /proc/net/tcp:   6: 0100007F:0885 0100007F:AC38 01 00000000:00000000 00:00000000 00000000  1000        0 36642 1 0000000000000000 20 4 21 10 -1       

But with 6.2.2 we get an error:

    /bin/bash: connect: Cannot assign requested address
    /bin/bash: /dev/tcp/localhost/9093: Cannot assign requested address

    [2021-12-13 12:19:46,251] [testcontainers-wait-0] [org.testcontainers.containers.wait.internal.InternalCommandPortListeningCheck] TRACE Check for [9093, 2181] took PT3.331S. Result code '137', stdout message: ''
    [2021-12-13 12:19:46,251] [testcontainers-wait-0] [org.testcontainers.containers.wait.internal.InternalCommandPortListeningCheck] WARN An exception while executing the internal check: Container.ExecResult(exitCode=137, stdout=, stderr=Ncat: Version 7.70 ( nmap.org/ncat )