Importing schemas from kafka after container restart crashes on "SELECT nextval('globalidsequence')"

Hello,

we have an issue with re-importing messages from Kafka topic back to internal h2 DB, that results into container crash. Here are the steps we followed:

1. Exporting data with exporter
2. Importing it to apicurio-registry-kafkasql:2.0.0.Final container connected to empty Kafka topic
3. Everything works fine until we re-deploy the Kubernetes deployment, whitch normally results to pods restart.
4. After restart, the pod will never run successfully again, it keeps crashing in loop.

All imports from Kafka after restart were successful, so it was not very visible in regitstry log, but when I increased DB trace level: -D%prod.quarkus.datasource.jdbc.url=jdbc:h2:mem:registry_db;TRACE_LEVEL_SYSTEM_OUT=2

after last import I’ve got this:

2021-05-15 14:05:41,632 DEBUG [io.api.reg.sto.imp.kaf.sql.KafkaSqlSink_Subclass] (KSQL Kafka Consumer Thread) Kafka message successfully processed. Notifying listeners of response.
2021-05-15 14:05:41,632 DEBUG [io.api.reg.sto.imp.kaf.sql.KafkaSqlSink_Subclass] (KSQL Kafka Consumer Thread) Processing Kafka message with UUID: ba36c16c-c4de-45b7-bea7-340df895cca1
2021-05-15 14:05:41,633 TRACE [org.jdb.v3.cor.Jdbi] (KSQL Kafka Consumer Thread) Jdbi [org.jdbi.v3.core.Jdbi@2b186595] obtain handle [org.jdbi.v3.core.Handle@cfc62d2] in 0ms
2021-05-15 14:05:41 jdbc[3]: 
/*SQL #:1*/SELECT MAX(globalId) FROM versions;
2021-05-15 14:05:41,634 INFO  [io.api.reg.sto.imp.sql.AbstractSqlRegistryStorage] (KSQL Kafka Consumer Thread) Resetting globalId sequence to 12517376
2021-05-15 14:05:41 jdbc[3]: 
/*SQL #:1 t:3*/SELECT nextval('globalidsequence');
2021-05-15 14:05:41 jdbc[3]: 
/*SQL #:1*/SELECT nextval('globalidsequence');
2021-05-15 14:05:41 jdbc[3]: 
/*SQL #:1*/SELECT nextval('globalidsequence');

... thousands of these lines ...

2021-05-15 14:05:46 jdbc[3]: 
/*SQL #:1*/SELECT nextval('globalidsequence');
2021-05-15 14:05:46 jdbc[3]: 
/*SQL #:1*/SELECT nextval('globalidsequence');
2021-05-15 14:05:46 jdbc[3]: 
/*SQL #:1*/SELECT nextval('globalidsequence');

then

2021-05-15 14:11:17 database: closing mem:registry_db from shutdown hook
2021-05-15 14:11:17 database: disconnecting session #3
2021-05-15 14:11:17 database: disconnected session #3
2021-05-15 14:11:17 database: disconnecting session #4
...
2021-05-15 14:11:17 database: disconnecting session #22
2021-05-15 14:11:17 database: disconnected session #22

and this at the end:

2021-05-15 14:11:17 database: closing mem:registry_db
2021-05-15 14:11:17 database: closed
2021-05-15 14:11:17 jdbc[3]: exception
2021-05-15 14:11:17,472 TRACE [org.jdb.v3.cor.Handle] (KSQL Kafka Consumer Thread) Handle [org.jdbi.v3.core.Handle@35dc7564] released
2021-05-15 14:11:17 jdbc[3]: exception
2021-05-15 14:11:17 jdbc[3]: exception
2021-05-15 14:11:17,491 DEBUG [io.api.reg.sto.imp.kaf.sql.KafkaSqlSink_Subclass] (KSQL Kafka Consumer Thread) Registry exception detected: org.jdbi.v3.core.result.ResultSetException: Could not get result set [statement:"SELECT nextval('globalidsequence')", arguments:{positional:{}, named:{}, finder:[]}]
2021-05-15 14:11:17,491 DEBUG [io.api.reg.sto.imp.kaf.sql.KafkaSqlSink_Subclass] (KSQL Kafka Consumer Thread) Processing Kafka message with UUID: c363bc84-dbd6-4546-a998-a427c0d3e2d3
2021-05-15 14:11:17,491 DEBUG [io.api.reg.sto.imp.kaf.sql.KafkaSqlSink_Subclass] (KSQL Kafka Consumer Thread) Unexpected exception detected: Error injecting javax.transaction.TransactionManager io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.transactionManager

I think the issue is connected with with the high globalId sequence number: 2021-05-15 14:05:41,634 INFO [io.api.reg.sto.imp.sql.AbstractSqlRegistryStorage] (KSQL Kafka Consumer Thread) Resetting globalId sequence to 12517376

as our glogalIds of imported schemas are following:

1
65536
458753
...
1769472
1769473
...
4259841
4325377
4456448
4456449
4915201
4980736
...
6815745
6881281
6946817
7012353
7077889
...
12320768
12517376

Is it tpossible, that the import api can handle these globalIds (as it is first time imported successfully), but import after container restart is different approach and cannot handle if global IDs are not in sequence? Could you please suggest how can we handle this in version 2.0.0.Final?

[EricWittmann]

I think we can close this with Fabian’s recent fix for the sequence reset. As for the next version, we’ll be cherry-picking bug fixes from master into a new 2.0.x branch (to be created) so that we can do patch releases. I could have a 2.0.1.Final release done next week probably.

[EricWittmann]

( closing this as fixed by https://github.com/Apicurio/apicurio-registry/pull/1507 )