Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
In Kafka-Connect scenario, the same schema is updated (PUT) in the registry for each table row
See original GitHub issueI am observing that the registry-client is invoking a PUT /api/artifacts/<reccname>-value for each record row and updating its version number. I am using version 1.2.3.Final and this behaviour is observed with both apicurio-registry-mem and apicurio-registry-kafka 1.2.3.Final.
I have a test table created with CREATE TABLE Persons5 (“id” int primary key, LastName varchar(255), FirstName varchar(255));
The HTTP transcription shows the following message exchange.
==============
PUT /api/artifacts/test-sqlite-jdbc-Persons5-value HTTP/1.1
Accept: application/json
X-Registry-ArtifactType: AVRO
Content-Type: */*
User-Agent: Jersey/2.28 (HttpUrlConnection 11.0.2)
Host: localhost:8080
Connection: keep-alive
Content-Length: 221
{"type":"record","name":"Persons5","fields":[{"name":"id","type":"long"},{"name":"LastName","type":["null","string"],"default":null},{"name":"FirstName","type":["null","string"],"default":null}],"connect.name":"Persons5"}
HTTP/1.1 404 Not Found
Date: Fri, 31 Jul 2020 21:15:02 GMT
Expires: Thu, 30 Jul 2020 21:15:02 GMT
Pragma: no-cache
Cache-control: no-cache, no-store, must-revalidate
Content-Type: application/json
Content-Length: 15241
{"message":"No artifact with ID 'test-sqlite-jdbc-Persons5-value' was found.","error_code":404,"detail":"io.apicurio.registry.storage.ArtifactNotFoundException: No artifact with ID 'test-sqlite-jdbc-Persons5-value' was found.\n\tat io.apicurio.registry.storage.impl.SimpleMapRegistryStorage$SimpleStorageMap.get(SimpleMapRegistryStorage.java:124)\n\tat io.apicurio.registry.storage.impl.AbstractMapRegistryStorage.getVersion2ContentMap(AbstractMapRegistryStorage.java:93)
#### the stack trace is too long and omitted in this issue report ####
\n\tat java.lang.Thread.run(Thread.java:748)\n\tat org.jboss.threads.JBossThread.run(JBossThread.java:479)\n"}
==============
POST /api/artifacts?ifExists=FAIL HTTP/1.1
Accept: application/json
X-Registry-ArtifactId: test-sqlite-jdbc-Persons5-value
X-Registry-ArtifactType: AVRO
Content-Type: */*
User-Agent: Jersey/2.28 (HttpUrlConnection 11.0.2)
Host: localhost:8080
Connection: keep-alive
Content-Length: 221
{"type":"record","name":"Persons5","fields":[{"name":"id","type":"long"},{"name":"LastName","type":["null","string"],"default":null},{"name":"FirstName","type":["null","string"],"default":null}],"connect.name":"Persons5"}
HTTP/1.1 200 OK
Date: Fri, 31 Jul 2020 21:15:03 GMT
Expires: Thu, 30 Jul 2020 21:15:03 GMT
Pragma: no-cache
Cache-control: no-cache, no-store, must-revalidate
Content-Type: application/json
Content-Length: 168
{"name":"Persons5","createdOn":1596230103600,"modifiedOn":1596230103600,"id":"test-sqlite-jdbc-Persons5-value","version":1,"type":"AVRO","globalId":0,"state":"ENABLED"}
==============
PUT /api/artifacts/test-sqlite-jdbc-Persons5-value HTTP/1.1
Accept: application/json
X-Registry-ArtifactType: AVRO
Content-Type: */*
User-Agent: Jersey/2.28 (HttpUrlConnection 11.0.2)
Host: localhost:8080
Connection: keep-alive
Content-Length: 221
{"type":"record","name":"Persons5","fields":[{"name":"id","type":"long"},{"name":"LastName","type":["null","string"],"default":null},{"name":"FirstName","type":["null","string"],"default":null}],"connect.name":"Persons5"}
HTTP/1.1 200 OK
Date: Fri, 31 Jul 2020 21:15:03 GMT
Expires: Thu, 30 Jul 2020 21:15:03 GMT
Pragma: no-cache
Cache-control: no-cache, no-store, must-revalidate
Content-Type: application/json
Content-Length: 168
{"name":"Persons5","createdOn":1596230103960,"modifiedOn":1596230103960,"id":"test-sqlite-jdbc-Persons5-value","version":2,"type":"AVRO","globalId":1,"state":"ENABLED"}
==============
PUT /api/artifacts/test-sqlite-jdbc-Persons5-value HTTP/1.1
Accept: application/json
X-Registry-ArtifactType: AVRO
Content-Type: */*
User-Agent: Jersey/2.28 (HttpUrlConnection 11.0.2)
Host: localhost:8080
Connection: keep-alive
Content-Length: 221
{"type":"record","name":"Persons5","fields":[{"name":"id","type":"long"},{"name":"LastName","type":["null","string"],"default":null},{"name":"FirstName","type":["null","string"],"default":null}],"connect.name":"Persons5"}
HTTP/1.1 200 OK
Date: Fri, 31 Jul 2020 21:15:03 GMT
Expires: Thu, 30 Jul 2020 21:15:03 GMT
Pragma: no-cache
Cache-control: no-cache, no-store, must-revalidate
Content-Type: application/json
Content-Length: 168
{"name":"Persons5","createdOn":1596230103986,"modifiedOn":1596230103986,"id":"test-sqlite-jdbc-Persons5-value","version":3,"type":"AVRO","globalId":2,"state":"ENABLED"}
==============
PUT /api/artifacts/test-sqlite-jdbc-Persons5-value HTTP/1.1
Accept: application/json
X-Registry-ArtifactType: AVRO
Content-Type: */*
User-Agent: Jersey/2.28 (HttpUrlConnection 11.0.2)
Host: localhost:8080
Connection: keep-alive
Content-Length: 221
{"type":"record","name":"Persons5","fields":[{"name":"id","type":"long"},{"name":"LastName","type":["null","string"],"default":null},{"name":"FirstName","type":["null","string"],"default":null}],"connect.name":"Persons5"}
HTTP/1.1 200 OK
Date: Fri, 31 Jul 2020 21:15:27 GMT
Expires: Thu, 30 Jul 2020 21:15:27 GMT
Pragma: no-cache
Cache-control: no-cache, no-store, must-revalidate
Content-Type: application/json
Content-Length: 168
{"name":"Persons5","createdOn":1596230127547,"modifiedOn":1596230127547,"id":"test-sqlite-jdbc-Persons5-value","version":4,"type":"AVRO","globalId":3,"state":"ENABLED"}
==============
PUT /api/artifacts/test-sqlite-jdbc-Persons5-value HTTP/1.1
Accept: application/json
X-Registry-ArtifactType: AVRO
Content-Type: */*
User-Agent: Jersey/2.28 (HttpUrlConnection 11.0.2)
Host: localhost:8080
Connection: keep-alive
Content-Length: 221
{"type":"record","name":"Persons5","fields":[{"name":"id","type":"long"},{"name":"LastName","type":["null","string"],"default":null},{"name":"FirstName","type":["null","string"],"default":null}],"connect.name":"Persons5"}
HTTP/1.1 200 OK
Date: Fri, 31 Jul 2020 21:15:27 GMT
Expires: Thu, 30 Jul 2020 21:15:27 GMT
Pragma: no-cache
Cache-control: no-cache, no-store, must-revalidate
Content-Type: application/json
Content-Length: 168
{"name":"Persons5","createdOn":1596230127572,"modifiedOn":1596230127572,"id":"test-sqlite-jdbc-Persons5-value","version":5,"type":"AVRO","globalId":4,"state":"ENABLED"}
This results in each Kafka message containing a new schema version number.
Issue Analytics
- State:
- Created 3 years ago
- Comments:6 (6 by maintainers)
Top Results From Across the Web
Using Kafka Connect with Schema Registry
Kafka Connect and Schema Registry integrate to capture schema information from connectors. Kafka Connect converters provide a mechanism for converting data ...
Read more >Database migration with Apache Kafka® and Apache ... - Aiven
During service creation we can enable Apache Kafka connect and the schema registry functionality provided by Aiven's Karapace and the ...
Read more >9 Using the Kafka Connect Handler - Oracle Help Center
Publishing messages to a single topic for multiple source tables will appear to the schema registry that the schema is evolving every time...
Read more >Integrate Apache Kafka Connect support on Azure Event Hubs ...
It provides a set of Kafka Connect connectors that tap into row-level changes in database table(s) and convert them into event streams that ......
Read more >Kafka 1.1 Documentation
Each partition has one server which acts as the "leader" and zero or more servers ... or in active/active scenarios to place data...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@EricWittmann For my original concern, this issue has been resolved. As I commented above, this behavior seems to be the intended behavior of
AutoRegisterIdStrategyand using an appropriate strategy will avoid this problem. I can close it. Thank you.Thanks @elakito - I think we have some things to improve in the serdes layer to make things easier to use/understand. That might just be better documentation, or examples. Or perhaps we can bundle up some common use-cases into configuration aliases. I’m not sure yet, but there’s a lot of potential for improvement I think! Any suggestions are always welcome - it’s good to get the perspective of someone actually trying to get things done.