HiveMQContainer with reuse on macOS M1 intermittently fails to start

Intermittently, when the HiveMQContainer is already running, our reused container fails to start with the following exception.

...
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hiveMqContainer' defined in class path resource [app/site/config/HiveMqContainerConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.hivemq.HiveMQContainer]: Factory method 'hiveMqContainer' threw exception; nested exception is org.testcontainers.containers.ContainerLaunchException: Container startup failed
	at app//org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
	at app//org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486)
	at app//org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
	at app//org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
	at app//org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
	at app//org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at app//org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at app//org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at app//org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at app//org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at app//org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
	at app//org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389)
	at app//org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309)
	at app//org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
	at app//org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
	... 106 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.hivemq.HiveMQContainer]: Factory method 'hiveMqContainer' threw exception; nested exception is org.testcontainers.containers.ContainerLaunchException: Container startup failed
	at app//org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at app//org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
	... 120 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed
	at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:345)
	at app//org.testcontainers.containers.GenericContainer.start(GenericContainer.java:326)
	at app//app.site.config.HiveMqContainerConfig$Companion.createContainer(InfrastructureConfig.kt:130)
	at app//app.site.config.HiveMqContainerConfig.hiveMqContainer(InfrastructureConfig.kt:137)
	at app//app.site.config.HiveMqContainerConfig$$EnhancerBySpringCGLIB$$f88740c.CGLIB$hiveMqContainer$0(<generated>)
	at app//app.site.config.HiveMqContainerConfig$$EnhancerBySpringCGLIB$$f88740c$$FastClassBySpringCGLIB$$87ad679a.invoke(<generated>)
	at app//org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
	at app//org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
	at app//app.site.config.HiveMqContainerConfig$$EnhancerBySpringCGLIB$$f88740c.hiveMqContainer(<generated>)
	at java.base@17.0.2/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base@17.0.2/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base@17.0.2/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base@17.0.2/java.lang.reflect.Method.invoke(Method.java:568)
	at app//org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 121 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
	at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:338)
	... 134 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at app//org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:537)
	at app//org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:340)
	at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	... 135 more
Caused by: org.rnorth.ducttape.TimeoutException: java.util.concurrent.TimeoutException
	at app//org.rnorth.ducttape.timeouts.Timeouts.callFuture(Timeouts.java:70)
	at app//org.rnorth.ducttape.timeouts.Timeouts.doWithTimeout(Timeouts.java:60)
	at app//org.testcontainers.containers.wait.strategy.WaitAllStrategy.waitUntilReady(WaitAllStrategy.java:53)
	at app//org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:926)
	at app//org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:480)
	... 137 more
Caused by: java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
	at org.rnorth.ducttape.timeouts.Timeouts.callFuture(Timeouts.java:65)
	... 141 more

We haven’t been able to pinpoint a clear cause. Third party MQTT clients (https://mqttx.app/) remain connected just fine, while the HiveMQContainer code fails. Vital stats follow.

build.gradle.kts plugins:

plugins {
    id("org.springframework.boot") version "2.6.7"
    id("io.spring.dependency-management") version "1.0.11.RELEASE"
    id("org.jlleitschuh.gradle.ktlint") version "10.3.0"
    id("org.jetbrains.dokka") version "1.6.21"
    jacoco
    kotlin("jvm") version "1.6.21"
    kotlin("plugin.spring") version "1.6.21"
    kotlin("plugin.allopen") version "1.6.21"
    kotlin("kapt") version "1.6.21"
}

build.gradle.kts testcontainer dependencies:

    // normally, these would be testImplementation, but this allows us to run locally
    // without having to launch containers manually
    implementation(platform("org.testcontainers:testcontainers-bom:1.17.2"))
    implementation("org.testcontainers:testcontainers")
    implementation("org.testcontainers:mongodb")
    implementation("org.testcontainers:postgresql")
    implementation("org.testcontainers:hivemq")

Docker version:

$ docker version
Client:
 Cloud integration: v1.0.24
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:49:20 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.8.2 (79419)
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:45:44 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

macOS version: 12.3.1 (21E258) (Monterey) MBP Model Name: MacBook Pro (16-inch, 2021) MBP Model Number: A2485 MBP Model Identifier: MacBookPro18,2 RAM: 64 Gb

Spring Boot Kotlin config class:

@Configuration
@Profile("default") // should only be activated locally on developers' machines
class HiveMqContainerConfig {
    companion object {
        val LOG = LoggerFactory.getLogger(HiveMqContainerConfig::class.java)

        val REUSE_ID = "app:site-microservice:hivemq"
        val IMAGE = DockerImageName.parse("hivemq/hivemq-ce:2021.3")
        var CONTAINER: HiveMQContainer? = null

        fun createContainer(): HiveMQContainer {
            if (CONTAINER === null) {
                CONTAINER = HiveMQContainer(IMAGE).apply {
                    withLogLevel(Level.DEBUG)
                    // NOTE: you have to set
                    // testcontainers.reuse.enable=true in local file ~/.testcontainers.properties
                    // for containers to actually be reused!
                    withReuse(true)
                    withLabel("reuse.id", REUSE_ID)
                }
                LOG.info("starting hivemq container -- you're prolly a dev, right?")
                CONTAINER!!.start()
            }
            return CONTAINER!!
        }
    }

    @Bean(destroyMethod = "")
    fun hiveMqContainer(): HiveMQContainer = createContainer()

    @Bean
    fun hiveMqClient(
        @Value("\${mqtt.client.identifier}") identifier: String,
        container: HiveMQContainer
    ): Mqtt5AsyncClient {
        return MqttClient
            .builder()
            .useMqttVersion5()
            .identifier(identifier)
            .serverHost(container.host)
            .serverPort(container.mqttPort)
            // TODO: other stuff...
            .buildAsync()
    }
}

[YannickWeber]

Hello @matthewadams!

The exception you present looks like the HiveMQ docker container is unable to start in time. It seems like you are running on arm64, please note that the image hivemq/hivemq-ce:2021.3 only comes for amd64. Normally, it is still possible to run amd64 images on arm64 machines, but from experience we know that this is rather unstable because of the underlying emulation.

As HiveMQ CE is open source there are unofficial arm64 images on dockerhub, that you can use: https://hub.docker.com/r/yannickweber/hivemq-ce.

val IMAGE = DockerImageName.parse("yannickweber/hivemq-ce").asCompatibleSubstituteFor("hivemq/hivemq-ce")

Anyhow, official arm64 support will come in the very near future.

If your problem persists please share the full log output of the container.

[eddumelendez]

We should follow https://github.com/hivemq/hivemq4-docker-images/issues/38 and wait for an arm image due to is already known that it will not work well on that architecture.