Not respected References order causes schemaParser failure

In class AbstractSchemaResolver there is the resolveReferences method:

    protected Map<String, ParsedSchema<S>> resolveReferences(List<io.apicurio.registry.rest.v2.beans.ArtifactReference> artifactReferences) {
        Map<String, ParsedSchema<S>> resolvedReferences = new HashMap<>();
        artifactReferences.forEach(reference -> {
            final InputStream referenceContent = client.getArtifactVersion(reference.getGroupId(), reference.getArtifactId(), reference.getVersion());
            final List<io.apicurio.registry.rest.v2.beans.ArtifactReference> referenceReferences = client.getArtifactReferencesByCoordinates(reference.getGroupId(), reference.getArtifactId(), reference.getVersion());
            if (!referenceReferences.isEmpty()) {
                final Map<String, ParsedSchema<S>> nestedReferences = resolveReferences(referenceReferences);
                resolvedReferences.putAll(nestedReferences);
                resolvedReferences.put(reference.getName(), parseSchemaFromStream(reference.getName(), referenceContent, resolveReferences(referenceReferences)));
            } else {
                resolvedReferences.put(reference.getName(), parseSchemaFromStream(reference.getName(), referenceContent, Collections.emptyMap()));
            }
        });
        return resolvedReferences;
    }

Here it’s really important that the records in resolvedReferences are in the correct order (nestedRefences must be before the initial reference in order to avoid parser failure) and that the order will be maintained.

The problem here is that an HashMap is used but, according with Java documentation, it doesn’t guarantee a predictable order: https://docs.oracle.com/javase/6/docs/api/java/util/HashMap.html

In fact, in our projects, the schemaParser fails because It tries to parse a schema before his references.

A possible solution might be to use LinkedHashMap that, again according to Java documentation, keep the correct order: https://docs.oracle.com/javase/10/docs/api/java/util/LinkedHashMap.html

[fabiobrz]

Hi @merlor - sorry for chiming in here, but I found this issue intresting and played a bit around it.

I noticed an integration test that is expected to verify serializing/deserializing object instances that have references, i.e.: https://github.com/Apicurio/apicurio-registry/blob/master/app/src/test/java/io/apicurio/registry/noprofile/serde/JsonSchemaSerdeTest.java#L204

So basically I decided to give it a try in order to have a reproducer that would support your analysis (HashMap vs. LinkedHashMap) and came up with a modified version of such tests, that creates a 4-level references chain, see https://github.com/fabiobrz/apicurio-registry/commit/45d75b085b7f4289d6885612553397b8a8e035ea

That being said, though, I keep getting a green bar whren executing the test. Do you have or can provide a minimum viable case that could be used as a reproducer? Or did I miss your point altogether and shpould focus on some different test case?

[carlesarnal]

Hi all, sorry for being late to the party…

😃

There is indeed a problem when handling nested and complex references in Avro but, according to my testing, it is not related to the data structure being used in this particular case but more about how Avro references are handled in the Avro schema parser (the nested references and some complex scenarios were not being properly handled). The order is of course important, but the important part resides in the schema parser and the order to parse the schemas there. I’ve created PR #2716 addressing some of those issues and also improving our testing to cover those scenarios.

Exactly what I meant, thanks @carlesarnal - BTW I tried to wrap a reproducer up with Avro (similar to what you’ve done) but the test still passes in my case.

Out of curiosity, were you able to reproduce it?

Yes, if I remove the code changes and just leave the test changes, the modified test fails for me.