org.hibernate.AnnotationException happens randomly

Describe the bug I’m trying to showcase the execution of an Axon application (https://github.com/AxonFramework) using Quarkus and JPA/Hibernate as an event store.

When I run the application I sometimes get the following error, sometimes not:

Caused by: org.hibernate.AnnotationException: Property org.axonframework.eventhandling.AbstractEventEntry.metaData has an unbound type and no explicit target entity. Resolve this Generic usage issue or set an explicit target attribute (eg @OneToMany(target=) or use an explicit @Type
	at org.hibernate.cfg.PropertyContainer.assertTypesAreResolvable(PropertyContainer.java:243)
	at org.hibernate.cfg.PropertyContainer.getProperties(PropertyContainer.java:233)
	at org.hibernate.cfg.AnnotationBinder.addElementsOfClass(AnnotationBinder.java:1505)
	at org.hibernate.cfg.AnnotationBinder.fillComponent(AnnotationBinder.java:2736)
	at org.hibernate.cfg.AnnotationBinder.bindIdClass(AnnotationBinder.java:2894)
	at org.hibernate.cfg.AnnotationBinder.mapAsIdClass(AnnotationBinder.java:1063)
	at org.hibernate.cfg.AnnotationBinder.bindClass(AnnotationBinder.java:791)
	at org.hibernate.boot.model.source.internal.annotations.AnnotationMetadataSourceProcessorImpl.processEntityHierarchies(AnnotationMetadataSourceProcessorImpl.java:248)
	at org.hibernate.boot.model.process.spi.MetadataBuildingProcess$1.processEntityHierarchies(MetadataBuildingProcess.java:239)
	at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:282)
	at io.quarkus.hibernate.orm.runtime.boot.FastBootMetadataBuilder.build(FastBootMetadataBuilder.java:320)
	at io.quarkus.hibernate.orm.runtime.PersistenceUnitsHolder.createMetadata(PersistenceUnitsHolder.java:101)
	at io.quarkus.hibernate.orm.runtime.PersistenceUnitsHolder.constructMetadataAdvance(PersistenceUnitsHolder.java:73)
	at io.quarkus.hibernate.orm.runtime.PersistenceUnitsHolder.initializeJpa(PersistenceUnitsHolder.java:40)
	at io.quarkus.hibernate.orm.runtime.HibernateOrmRecorder$1.created(HibernateOrmRecorder.java:60)
	at io.quarkus.arc.runtime.ArcRecorder.initBeanContainer(ArcRecorder.java:53)
	at io.quarkus.deployment.steps.ArcProcessor$generateResources-1025303321.deploy_0(ArcProcessor$generateResources-1025303321.zig:149)
	at io.quarkus.deployment.steps.ArcProcessor$generateResources-1025303321.deploy(ArcProcessor$generateResources-1025303321.zig:40)
	at io.quarkus.runner.ApplicationImpl.<clinit>(ApplicationImpl.zig:317)

I saw the following ticket: #12228 but I’m not sure it’s actually related.

I’m using the default JPA mapping used in Axon, it works nicely with Spring Boot (comments removed for clarity):

@Entity
@Table(indexes = @Index(columnList = "aggregateIdentifier,sequenceNumber", unique = true))
public class DomainEventEntry extends AbstractSequencedDomainEventEntry<byte[]> {

    protected DomainEventEntry() {
    }
}

This entity has the following hierarchy:

@MappedSuperclass
public abstract class AbstractSequencedDomainEventEntry<T> extends AbstractDomainEventEntry<T> implements DomainEventData<T> {

    @Id
    @GeneratedValue
    @SuppressWarnings("unused")
    private long globalIndex;

    public AbstractSequencedDomainEventEntry(DomainEventMessage<?> eventMessage, Serializer serializer,
                                             Class<T> contentType) {
        super(eventMessage, serializer, contentType);
    }

    protected AbstractSequencedDomainEventEntry() {
    }
}

@MappedSuperclass
public abstract class AbstractDomainEventEntry<T> extends AbstractEventEntry<T> implements DomainEventData<T> {

    @Basic
    private String type;
    @Basic(optional = false)
    private String aggregateIdentifier;
    @Basic(optional = false)
    private long sequenceNumber;

    public AbstractDomainEventEntry(DomainEventMessage<?> eventMessage, Serializer serializer, Class<T> contentType) {
        super(eventMessage, serializer, contentType);
        type = eventMessage.getType();
        aggregateIdentifier = eventMessage.getAggregateIdentifier();
        sequenceNumber = eventMessage.getSequenceNumber();
    }

    public AbstractDomainEventEntry(String type, String aggregateIdentifier, long sequenceNumber,
                                    String eventIdentifier, Object timestamp, String payloadType,
                                    String payloadRevision, T payload, T metaData) {
        super(eventIdentifier, timestamp, payloadType, payloadRevision, payload, metaData);
        this.type = type;
        this.aggregateIdentifier = aggregateIdentifier;
        this.sequenceNumber = sequenceNumber;
    }

    protected AbstractDomainEventEntry() {
    }

    @Override
    public String getType() {
        return type;
    }

    @Override
    public String getAggregateIdentifier() {
        return aggregateIdentifier;
    }

    @Override
    public long getSequenceNumber() {
        return sequenceNumber;
    }
}

@MappedSuperclass
public abstract class AbstractEventEntry<T> implements EventData<T> {

    @Column(nullable = false, unique = true)
    private String eventIdentifier;
    @Basic(optional = false)
    private String timeStamp;
    @Basic(optional = false)
    private String payloadType;
    @Basic
    private String payloadRevision;
    @Basic(optional = false)
    @Lob
    @Column(length = 10000)
    private T payload;
    @Basic
    @Lob
    @Column(length = 10000)
    private T metaData;

    public AbstractEventEntry(EventMessage<?> eventMessage, Serializer serializer, Class<T> contentType) {
        SerializedObject<T> payload = eventMessage.serializePayload(serializer, contentType);
        SerializedObject<T> metaData = eventMessage.serializeMetaData(serializer, contentType);
        this.eventIdentifier = eventMessage.getIdentifier();
        this.payloadType = payload.getType().getName();
        this.payloadRevision = payload.getType().getRevision();
        this.payload = payload.getData();
        this.metaData = metaData.getData();
        this.timeStamp = formatInstant(eventMessage.getTimestamp());
    }

    public AbstractEventEntry(String eventIdentifier, Object timestamp, String payloadType, String payloadRevision,
                              T payload, T metaData) {
        this.eventIdentifier = eventIdentifier;
        if (timestamp instanceof TemporalAccessor) {
            this.timeStamp = formatInstant((TemporalAccessor) timestamp);
        } else {
            this.timeStamp = timestamp.toString();
        }
        this.payloadType = payloadType;
        this.payloadRevision = payloadRevision;
        this.payload = payload;
        this.metaData = metaData;
    }

    protected AbstractEventEntry() {
    }

    @Override
    public String getEventIdentifier() {
        return eventIdentifier;
    }

    @Override
    public Instant getTimestamp() {
        return DateTimeUtils.parseInstant(timeStamp);
    }

    @Override
    @SuppressWarnings("unchecked")
    public SerializedObject<T> getMetaData() {
        return new SerializedMetaData<>(metaData, (Class<T>) metaData.getClass());
    }

    @Override
    @SuppressWarnings("unchecked")
    public SerializedObject<T> getPayload() {
        return new SimpleSerializedObject<>(payload, (Class<T>) payload.getClass(),
                                            new SimpleSerializedType(payloadType, payloadRevision));
    }
}

Expected behavior The application should start

Actual behavior Sometimes the application starts fine, sometimes not (see the following logs).

Success

Listening for transport dt_socket at address: 5005
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2021-03-09 22:06:59,766 INFO  [io.quarkus] (Quarkus Main Thread) axon-quarkus 1.0.0-SNAPSHOT on JVM (powered by Quarkus 1.12.1.Final) started in 1.656s. Listening on: http://localhost:8080
2021-03-09 22:06:59,767 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2021-03-09 22:06:59,768 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [agroal, cdi, config-yaml, hibernate-orm, hibernate-validator, jdbc-h2, micrometer, mutiny, narayana-jta, rest-client, rest-client-jaxb, resteasy, resteasy-jackson, smallrye-context-propagation]

Failure

2021-03-09 22:08:17,420 INFO  [org.jbo.threads] (main) JBoss Threads version 3.2.0.Final
2021-03-09 22:08:17,717 INFO  [org.hib.Version] (build-2) HHH000412: Hibernate ORM core version 5.4.28.Final
2021-03-09 22:08:18,196 INFO  [org.hib.val.int.uti.Version] (Quarkus Main Thread) HV000001: Hibernate Validator 6.2.0.Final
2021-03-09 22:08:18,252 INFO  [org.hib.Version] (Quarkus Main Thread) HHH000412: Hibernate ORM core version 5.4.28.Final
2021-03-09 22:08:18,262 INFO  [org.hib.ann.com.Version] (Quarkus Main Thread) HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2021-03-09 22:08:18,294 INFO  [org.hib.dia.Dialect] (Quarkus Main Thread) HHH000400: Using dialect: org.hibernate.dialect.H2Dialect
2021-03-09 22:08:18,342 INFO  [io.qua.dep.dev.IsolatedDevModeMain] (main) Attempting to start hot replacement endpoint to recover from previous Quarkus startup failure
2021-03-09 22:08:18,343 ERROR [io.qua.run.boo.StartupActionImpl] (Quarkus Main Thread) Error running Quarkus: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.quarkus.runner.bootstrap.StartupActionImpl$3.run(StartupActionImpl.java:134)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ExceptionInInitializerError
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at java.base/java.lang.Class.newInstance(Class.java:584)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:65)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:42)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:119)
	at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:29)
	... 6 more
Caused by: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.<clinit>(ApplicationImpl.zig:403)
	... 15 more
Caused by: org.hibernate.AnnotationException: Property org.axonframework.eventhandling.AbstractEventEntry.metaData has an unbound type and no explicit target entity. Resolve this Generic usage issue or set an explicit target attribute (eg @OneToMany(target=) or use an explicit @Type
	at org.hibernate.cfg.PropertyContainer.assertTypesAreResolvable(PropertyContainer.java:243)
	at org.hibernate.cfg.PropertyContainer.getProperties(PropertyContainer.java:233)
	at org.hibernate.cfg.AnnotationBinder.addElementsOfClass(AnnotationBinder.java:1505)
	at org.hibernate.cfg.AnnotationBinder.fillComponent(AnnotationBinder.java:2736)
	at org.hibernate.cfg.AnnotationBinder.bindIdClass(AnnotationBinder.java:2894)
	at org.hibernate.cfg.AnnotationBinder.mapAsIdClass(AnnotationBinder.java:1063)
	at org.hibernate.cfg.AnnotationBinder.bindClass(AnnotationBinder.java:791)
	at org.hibernate.boot.model.source.internal.annotations.AnnotationMetadataSourceProcessorImpl.processEntityHierarchies(AnnotationMetadataSourceProcessorImpl.java:248)
	at org.hibernate.boot.model.process.spi.MetadataBuildingProcess$1.processEntityHierarchies(MetadataBuildingProcess.java:239)
	at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:282)
	at io.quarkus.hibernate.orm.runtime.boot.FastBootMetadataBuilder.build(FastBootMetadataBuilder.java:320)
	at io.quarkus.hibernate.orm.runtime.PersistenceUnitsHolder.createMetadata(PersistenceUnitsHolder.java:101)
	at io.quarkus.hibernate.orm.runtime.PersistenceUnitsHolder.constructMetadataAdvance(PersistenceUnitsHolder.java:73)
	at io.quarkus.hibernate.orm.runtime.PersistenceUnitsHolder.initializeJpa(PersistenceUnitsHolder.java:40)
	at io.quarkus.hibernate.orm.runtime.HibernateOrmRecorder$1.created(HibernateOrmRecorder.java:60)
	at io.quarkus.arc.runtime.ArcRecorder.initBeanContainer(ArcRecorder.java:53)
	at io.quarkus.deployment.steps.ArcProcessor$generateResources-1025303321.deploy_0(ArcProcessor$generateResources-1025303321.zig:149)
	at io.quarkus.deployment.steps.ArcProcessor$generateResources-1025303321.deploy(ArcProcessor$generateResources-1025303321.zig:40)
	at io.quarkus.runner.ApplicationImpl.<clinit>(ApplicationImpl.zig:317)
	... 15 more

To Reproduce

Here is the project were you can reproduce the issue: https://github.com/frezelth/axon-quarkus

Steps to reproduce the behavior:

1. Compile
2. Run
3. Error sometimes happens when starting

Configuration

# quarkus:
  datasource:
    db-kind: h2
    jdbc:
      url: jdbc:h2:mem:default
  hibernate-orm:
    dialect: org.hibernate.dialect.H2Dialect
    database:
      generation: drop-and-create
    packages: org.axonframework.eventsourcing.eventstore.jpa
  index-dependency:
    axon-eventsourcing:
      group-id: org.axonframework
      artifact-id: axon-eventsourcing
    axon-messaging:
      group-id: org.axonframework
      artifact-id: axon-messaging

Environment (please complete the following information):

• Output of uname -a or ver: Linux lw202054295 5.4.0-66-generic #74~18.04.2-Ubuntu SMP Fri Feb 5 11:17:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Output of java -version: openjdk version “11.0.9.1” 2020-11-04
• GraalVM version (if different from Java): OpenJDK Runtime Environment GraalVM CE 21.0.0.2 (build 11.0.10+8-jvmci-21.0-b06)
• Quarkus version or git rev: 1.12.1.Final
• Build tool (ie. output of mvnw --version or gradlew --version): maven 3.6.3

[Sanne]

For ORM this will be improved as https://hibernate.atlassian.net/browse/HHH-14499 .

Luckily this is not a blocker, as I can prevent the problem with a one-line patch in Quarkus, and it turns out it doesn’t need any change in HCANN either.

[Sanne]

It turns out this is a complex issue, because of its nature of not being consistently reproducible and affecting the very old HCANN library (https://github.com/hibernate/hibernate-commons-annotations).

I was able to narrow it down by splittin the problem into multiple problems which are interacting:

1. The metadata from entity scanning produced by Quarkus, which is then handed to Hibernate, is not consistent in each run. Of course it’s always complete but it seems the order of entities is not stable over time.
2. Hibernate ORM seems able to sort the mapping metadata so to narrow down the use of generics in this mapping, but there is a specific input order which seems to be able to trick it into an invalid sort order
3. I’ve not figured out yet why HCANN requires this specific order to be able to narrow the boundaries of the generics correctly

We’ll need to work on each of these as different issues.