`cub kafka-ready` fails with non-plaintext and config.provider

When a config.providers is specified and connecting with a security.protocol other than PLAINTEXT then cub kafka-ready fails.

When using non-PLAINTEXT, the generated kafka-connect.properties which contains the relevant config.providers properties. The following error is thrown:

[main] ERROR org.apache.kafka.clients.admin.AdminClientConfig - ClassNotFoundException exception occurred: org.ggt.kafka.config.provider.KafkaEnvConfigProvider
[main] ERROR io.confluent.admin.utils.cli.KafkaReadyCommand - Error while running kafka-ready.
org.apache.kafka.common.config.ConfigException: Invalid value java.lang.ClassNotFoundException: org.ggt.kafka.config.provider.KafkaEnvConfigProvider for configuration Invalid config:org.ggt.kafka.config.provider.KafkaEnvConfigProvider ClassNotFoundException exception occurred
	at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:537)
	at org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:477)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:107)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
	at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:208)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:69)
	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
	at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:138)

Relevant code in the ensure file: https://github.com/confluentinc/cp-docker-images/blob/5.4-preview/debian/kafka-connect-base/include/etc/confluent/docker/ensure#L24-L28

The config provider used works just fine with the strimzi images and works with PLAINTEXT.

[jennybrown8]

I seem to also have this issue with a different config provider (AWSSecretProvider from lenses.io). In my case, the cub classpath does not impact the results, but changing CONNECT_SECURITY_PROTOCOL from SASL_SSL to PLAINTEXT does resolve it (even without touching any classpaths). I shouldn’t keep it on plaintext for production use, so I still hope to see a deeper fix for this. I hope this helps to further isolate the problem for fixes.

[mubeta06]

Update: The issue appears to be associated with cub’s inability to load up the config providers jar file. The work around for us was to append to CUB_CLASSPATH the path to the org.ggt.kafka.config.provider.KafkaEnvConfigProvider jar file. e.g. CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar:/usr/local/share/java/kafka-connect/plugins/kafka-env-config-provider/kafka-env-config-provider-0.0.1.jar