Hibernate Reactive: NamedQuery fails after hot reload

Describe the bug Same issue as already described by @cescoffier in #13355, but I was actually able to reproduce it reliably 😄

After the application is reloaded in dev mode due to a code change, named queries do not work anymore. You don’t even need to change something in the queried entity itself, just any change that leads to an entire reload of the application will be sufficient.

Expected behavior NamedQuery should work just as before hot reload.

Actual behavior Executing a NamedQuery leads to the following exception:

2021-02-18 10:46:11,037 ERROR [org.jbo.res.rea.ser.cor.ExceptionMapping] (vert.x-eventloop-thread-11) Request failed : java.lang.IllegalArgumentException: Type specified for TypedQuery [com.example.Fruit] is incompatible with query return type [class com.example.Fruit]
	at org.hibernate.internal.AbstractSharedSessionContract.resultClassChecking(AbstractSharedSessionContract.java:863)
	at org.hibernate.reactive.session.impl.ReactiveSessionImpl.createReactiveQuery(ReactiveSessionImpl.java:461)
	at org.hibernate.reactive.session.impl.ReactiveSessionImpl.buildReactiveQueryFromName(ReactiveSessionImpl.java:443)
	at org.hibernate.reactive.session.impl.ReactiveSessionImpl.createReactiveNamedQuery(ReactiveSessionImpl.java:430)
	at org.hibernate.reactive.mutiny.impl.MutinySessionImpl.createNamedQuery(MutinySessionImpl.java:222)
	at io.quarkus.hibernate.reactive.runtime.ReactiveSessionProducer_ProducerMethod_createMutinySession_1321d110ee9e92bda147899150401e0a136779c7_ClientProxy.createNamedQuery(ReactiveSessionProducer_ProducerMethod_createMutinySession_1321d110ee9e92bda147899150401e0a136779c7_ClientProxy.zig:997)
	at com.example.FruitDao.get(FruitDao.java:20)
	at com.example.FruitDao_ClientProxy.get(FruitDao_ClientProxy.zig:217)
	at com.example.FruitResource.get(FruitResource.java:32)
	at com.example.FruitResource_ClientProxy.get(FruitResource_ClientProxy.zig:217)
	at com.example.FruitResource$quarkusrestinvoker$get_2420412d5b210b3632823e74ede45d605c6ff069.invoke(FruitResource$quarkusrestinvoker$get_2420412d5b210b3632823e74ede45d605c6ff069.zig:33)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
	at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:7)
	at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:117)
	at org.jboss.resteasy.reactive.server.handlers.RestInitialHandler.beginProcessing(RestInitialHandler.java:47)
	at org.jboss.resteasy.reactive.server.vertx.ResteasyReactiveVertxHandler.handle(ResteasyReactiveVertxHandler.java:17)
	at org.jboss.resteasy.reactive.server.vertx.ResteasyReactiveVertxHandler.handle(ResteasyReactiveVertxHandler.java:7)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1038)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:137)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:132)
	at io.quarkus.vertx.http.runtime.StaticResourcesRecorder.lambda$start$1(StaticResourcesRecorder.java:62)
	at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1038)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:101)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:132)
	at io.vertx.ext.web.handler.impl.StaticHandlerImpl.lambda$sendStatic$1(StaticHandlerImpl.java:206)
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327)
	at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
	at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

To Reproduce

Link to a small reproducer (preferably a Maven project if the issue is not Gradle-specific).

https://github.com/markusdlugi/hibernate-reactive-hot-swap

Steps to reproduce the behavior:

1. Start PostgreSQL database using infrastructure/docker-compose.yml
2. Start application in dev mode using mvn quarkus:dev
3. Execute GET http://localhost:8080/fruits - should still work with HTTP 200
4. Make a code change to trigger hot reload, e.g. by removing line 49 in com.example.FruitDao
5. Execute GET http://localhost:8080/fruits again - application will reload in the background and fail with exception

Configuration

# Add your application.properties here, if applicable.
quarkus.datasource.db-kind=postgresql
quarkus.datasource.username=test
quarkus.datasource.password=test
quarkus.datasource.reactive.url=vertx-reactive:postgresql://localhost:5432/hibernate-reactive-hot-swap
quarkus.hibernate-orm.database.generation=create

Screenshots (If applicable, add screenshots to help explain your problem.)

Environment (please complete the following information):

• Output of uname -a or ver: Microsoft Windows [Version 10.0.17763.1757]
• Output of java -version: OpenJDK 64-Bit Server VM Corretto-11.0.10.9.1 (build 11.0.10+9-LTS, mixed mode)
• GraalVM version (if different from Java): N/A
• Quarkus version or git rev: 1.11.3.Final
• Build tool (ie. output of mvnw --version or gradlew --version): Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)

Additional context For some reason, the issue only appears if both of the following conditions are met:

• The application has a dependency to io.quarkus:quarkus-smallrye-health
• The folder META-INF/resources exists (can be empty)

If either of these is removed, the issue just disappears. No idea why.

I don’t think that these are the only conditions where this issue appears, most likely this can also happen under different circumstances, but those were the conditions where it failed in our application.

[geoand]

I’m glad it worked!

To be honest, I was kinda hoping it would fix this issue as well, but I had no idea 😃

[markusdlugi]

Thanks once again for checking.

I tried the reproducer project again and it seems like you are right, removing the dependency to quarkus-smallrye-health does not change the behavior. That’s strange because I was fairly sure it did back when I created the issue, but seems like I was wrong. Sorry about that.

What still does affect the behavior is removing the META-INF/resources folder - once you remove that, the issue cannot be reproduced anymore, NamedQueries are working fine, no exceptions.

I also tried the quickstart you mentioned and can confirm that the issue is also reproducible there. Similarly, if you remove the META-INF/resources folder there, issue also disappears for the quickstart.

I had another look at some heap dumps, and actually noticed that the Fruit entity being present twice is true both in the failing case and in the working case (so without META-INF/resources). So the classloader leak does not seem to be the main root cause of the exception - Quarkus is leaking the classloaders in any case (mainly due to the contextClassLoaders of the Vert.x threads, as it seems). But in one case, Hibernate is able to handle that, while in the other case, for some reason, it isn’t. I still haven’t found a reason why Hibernate behaves differently in the two cases.