[Enhancement] Upon kctr deletion wait until the connector is stopped

Is your feature request related to a problem? Please describe. In our case we create one connector (kctr1) to do some initial work, later we delete kctr1 and create kctr2 to continue. The operator doesn’t wait until a connector is stopped and removed from the Kafka Connect cluster. This results in kctr1 and kctr2 overlapping and producing invalid data. This forces us after issuing kctr1 deletion to call Kafka Connect API to see if the connector has been stopped and deleted.

Describe the solution you’d like Ideally the operator should consult GET /connectors/<connector name>/status on whether the connector has been stopped and deleted from the cluster before deleting the corresponding Kubernetes resource. This would allow for kubectl delete kctr <connector name> --wait=true to behave correcly. Currently this command waits only untill the connector’s configuration is removed from Kafka Connect cluster which doesn’t mean the connector itself is gone.

[scholzj]

The additional check for the status should be in the 0.21.0 release.

[morozov]

[…] why are you deleting and creating connector instead of just updating it for example.

Our use case is that we create a Debezium connector pointed to an existing production database. In order to avoid the full DB lock during the snapshot, we make a clone of the DB, run the snapshot reader against the clone and then run the binlog reader against the actual DB.

The snapshot and the binlog readers cannot be represented by the same connector because they cannot share the config, offsets, etc. and should be run consequently.

[…] what exactly is the overlap […]

I don’t remember exactly. We experienced some Kafka Producer level errors that pointed out that there is an overlap possible between the connectors.

But assuming it is true fixing it should for example make sure a new reconciliation does not start before the old one is finished (=> the old connector properly deleted). So that alone could help.

Would you consider implementing it in Strimzi? Is it possible that despite the real order of events: “old deleted”, “new created”, Strimzi will receive “new created” first, start reconciliation, and only then will receive “old deleted”?