Images "Failed to create cache dir" after upgrading to version 1.10.0

Recently, we upgraded our hono installation from version 1.6.0 to 1.10.0 on our OpenShift platform, but now most of the images don’t start any more because of the failed permission to create the tmp folder. It seems that on start the container tries to create a folder in the root directory instead inside of the /tmp folder, because he is building a string containing of the /tmp folder path and a UUID (/tmp-{UUID} as seen in the example below) instead of how it was the case in version 1.6.0 /tmp/file-{UUID}.

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
13:18:40.843 [main] ERROR o.s.boot.SpringApplication - Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'application': Unsatisfied dependency expressed through method 'setVertx' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vertx' defined in org.eclipse.hono.commandrouter.spring.ApplicationConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.vertx.core.Vertx]: Factory method 'vertx' threw exception; nested exception is java.lang.IllegalStateException: Failed to create cache dir: /tmp-d9297304-40a8-4fe6-b51e-936ceccd8d84
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.resolveMethodArguments(AutowiredAnnotationBeanPostProcessor.java:768)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:720)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1413)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:771)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:763)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:438)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:339)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318)
	at org.eclipse.hono.commandrouter.spring.Application.main(Application.java:110)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vertx' defined in org.eclipse.hono.commandrouter.spring.ApplicationConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.vertx.core.Vertx]: Factory method 'vertx' threw exception; nested exception is java.lang.IllegalStateException: Failed to create cache dir: /tmp-d9297304-40a8-4fe6-b51e-936ceccd8d84
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1380)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1300)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.resolveMethodArguments(AutowiredAnnotationBeanPostProcessor.java:760)
	... 20 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.vertx.core.Vertx]: Factory method 'vertx' threw exception; nested exception is java.lang.IllegalStateException: Failed to create cache dir: /tmp-d9297304-40a8-4fe6-b51e-936ceccd8d84
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
	... 33 common frames omitted
Caused by: java.lang.IllegalStateException: Failed to create cache dir: /tmp-d9297304-40a8-4fe6-b51e-936ceccd8d84
	at io.vertx.core.file.impl.FileCache.setupCacheDir(FileCache.java:59)
	at io.vertx.core.file.impl.FileCache.setupCache(FileCache.java:29)
	at io.vertx.core.file.impl.FileResolver.<init>(FileResolver.java:64)
	at io.vertx.core.impl.VertxBuilder.initFileResolver(VertxBuilder.java:316)
	at io.vertx.core.impl.VertxBuilder.init(VertxBuilder.java:248)
	at io.vertx.core.Vertx.vertx(Vertx.java:86)
	at org.eclipse.hono.commandrouter.spring.ApplicationConfig.vertx(ApplicationConfig.java:135)
	at org.eclipse.hono.commandrouter.spring.ApplicationConfig$$EnhancerBySpringCGLIB$$501e6da7.CGLIB$vertx$1(<generated>)
	at org.eclipse.hono.commandrouter.spring.ApplicationConfig$$EnhancerBySpringCGLIB$$501e6da7$$FastClassBySpringCGLIB$$ec39dc8c.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
	at org.eclipse.hono.commandrouter.spring.ApplicationConfig$$EnhancerBySpringCGLIB$$501e6da7.vertx(<generated>)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 34 common frames omitted
Caused by: java.nio.file.AccessDeniedException: /tmp-d9297304-40a8-4fe6-b51e-936ceccd8d84
	at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source)
	at java.base/java.nio.file.Files.createDirectory(Unknown Source)
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source)
	at java.base/java.nio.file.Files.createDirectories(Unknown Source)
	at io.vertx.core.file.impl.FileCache.setupCacheDir(FileCache.java:56)
	... 50 common frames omitted

The JAVA_OPTIONS start parameter -Dvertx.cacheDirBase=/tmp in the Dockerfile is the same as before, but it seems something is missing in the code now?

After some tests it seems the container is starting when the start parameter is changed to -Dvertx.cacheDirBase=/tmp/file and just for clarification -Dvertx.cacheDirBase=/tmp/ is not enough here. It also starts by adding -Dvertx.disableFileCPResolving=true to the start parameters, but not with -Dvertx.disableFileCaching=true.

As it seems that the vert.x library got rebuild on that part entirely very recent, I’m not sure if this is maybe a vert.x bug or a hono configuration failure? But after looking for a while I couldn’t find an option to change this tmp folder option anywhere in the config files?

The only solution I would think of right now would be to change the Dockerfile value and rebuild the images again with the changed values, overwrite the enviroment variable in the deployment or run the container as root (which is not really an option to be honest).

But I guess this is not a wanted default behaviour, unless of course I’m missing some important config value here?

[klankster]

@sophokles73 just tried it and, yes, it works. A folder with vertx-cache-{UUID} is created in the /tmp folder.

[klankster]

@sophokles73 Yes, I use the Helm chart for deploying and right now just hard coded the environment variable JAVA_OPTIONS = -Dvertx.cacheDirBase=/tmp/file into the deployment YAML files I need. With this, the containers are starting now again.