Debezium connector is broken in 2.3.0

Steps to reproduce

docker run -it -p 6650:6650 -p 8080:8080 apachepulsar/pulsar-all:2.3.0 /pulsar/bin/pulsar standalone

Error Log

00:06:01.073 [main] WARN  org.apache.pulsar.functions.utils.io.ConnectorUtils - Failed to load connector from /pulsar/./connectors/pulsar-io-debezium-2.3.0.nar
java.nio.file.NoSuchFileException: /tmp/pulsar-nar/pulsar-io-debezium-2.3.0.nar-unpacked/META-INF/services/pulsar-io.yaml
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:1.8.0_181]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_181]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_181]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:1.8.0_181]
	at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_181]
	at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_181]
	at java.nio.file.Files.readAllBytes(Files.java:3152) ~[?:1.8.0_181]
	at org.apache.pulsar.common.nar.NarClassLoader.getServiceDefinition(NarClassLoader.java:198) ~[org.apache.pulsar-pulsar-common-2.3.0.jar:2.3.0]
	at org.apache.pulsar.functions.utils.io.ConnectorUtils.getConnectorDefinition(ConnectorUtils.java:103) ~[org.apache.pulsar-pulsar-functions-utils-2.3.0.jar:2.3.0]
	at org.apache.pulsar.functions.utils.io.ConnectorUtils.searchForConnectors(ConnectorUtils.java:123) [org.apache.pulsar-pulsar-functions-utils-2.3.0.jar:2.3.0]
	at org.apache.pulsar.functions.worker.ConnectorsManager.<init>(ConnectorsManager.java:34) [org.apache.pulsar-pulsar-functions-worker-2.3.0.jar:2.3.0]
	at org.apache.pulsar.functions.worker.WorkerService.start(WorkerService.java:159) [org.apache.pulsar-pulsar-functions-worker-2.3.0.jar:2.3.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1042) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:441) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0]
	at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:305) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0]
	at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:102) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0]

[sijie]

@aahmed-se I don’t think you are using the right connector. you need to use pulsar-io-kafka-connect-adaptor instead of using pulsar-io-debezium. The documentation is here : http://pulsar.apache.org/docs/en/io-cdc/

Additionally I think debezium module should not be packaged as a NAR. it is just a normal module for keeping debezium related code changes and a dependency for pulsar-io-kafka-connect-adaptor.

[sijie]

Sent pull request #3656

In any case, shouldn’t that be debezium connector rather that be named Kafka connect?

It is a generic kafka-connect adaptor connector that invokes debezium. because debezium is too tied to Kafka connect.

We can add a few wrappers to wrap kafka-connect-adaptor to a debezium connector if needed. /cc @jiazhai to see if he can help with that. However I think keeping a generic kafka-connect-adaptor connector will have its own value.