Fail to publish msgs on topic using async and avro on docker-compose

Hi all,

I have been testing out the new blog recently using docker-compose https://microcks.io/blog/async-features-with-docker-compose/

What i am trying to achieve is to use async API with an avro schema and onboard to schema registry. Similar to the post that was published before: https://microcks.io/documentation/guides/avro-messaging/

Step 1

What I initially wanted to do is onboard my API via the importer. I worked out how to define a scheduler to pull in both files (async and avro)

NB: This link is broken btw where it mentions the importer job in the article https://microcks.io/documentation/documentation/using/importers/#scheduled-import

See scnshots below:

As you can see i import the raw file on a schedule

https://raw.githubusercontent.com/shavo007/kafka-tutorial/main/asyncapi/src/main/resources/avro/io/confluent/examples/clients/basicavro/payment-avro-ref-asyncapi.yaml

And it picks up both files in the API which is nice. So first part complete!

Step 2

When I set this up I was initially confused by the logging as it mentions schema registry but none existed. Example of log output:

2021-09-28 05:03:16,248 INFO  [io.git.mic.min.asy.SchemaRegistry] (main) Updating Schema registry for 'Payment Avro API - 0.1.2' with 2 entries

But there is no schema registry running. So then when i looked at the docker-compose files I dont see a container running the schema registry. So I added that in:

  schema-registry:
    image: confluentinc/cp-schema-registry:6.2.0
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - zookeeper
      - kafka
    ports:
     - "8888:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: "zookeeper:2181"

And i updated the application.properties

kafka.schema.registry.url=http://localhost:8888

And then ran that:

docker-compose -f docker-compose.yml -f docker-compose-async-addon.yml up -d

Just to test this out I used my repos existing producer to send messages with an avro schema and that all worked: https://github.com/shavo007/kafka-tutorial/blob/10c93d0a9cf080705ab1c0a81bdd8e481538995a/asyncapi/src/main/java/io/confluent/examples/clients/basicavro/ProducerExample.java

I can then verify the schema exists:

curl --silent -X GET http://localhost:8888/subjects/transactions-value/versions/latest | jq .
{
  "subject": "transactions-value",
  "version": 1,
  "id": 1,
  "schema": "{\"type\":\"record\",\"name\":\"Payment\",\"namespace\":\"io.confluent.examples.clients.basicavro\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}]}"
}

But i cant see mirocks creating a new schema even tho it is publishing payment events to the topic PaymentAvroAPI-0.1.2-transactions

i can see the topic is created with events but no schema

kcat -b localhost:9092 -L
kcat -b localhost:9092 -t PaymentAvroAPI-0.1.2-transactions -o end

Am i missing something?? What i was expecting to see is microcks publishing sample events to topic and publishing the schema to the registry also.

[shavo007]

can do! ill add it to my backlog. I’m only kicking off evaluating different tools for EDA currently. I’ll reach out if we come to that decision.

[lbroudoux]

Hey @shavo007! That’s awesome! Would you be interested sharing your use-case and work with community through a blog post? We run a blog on https://microcks.io/blog/ and welcome anyone that like to share their story about using Microcks. We’d very happy!