cp-kafka-connect: Custom ConfigProvider ClassNotFoundExceptions on startup

Hey, I just tried the cp-kafka-connect docker image and I want to install some custom ConfigProviders. The Confluent documentation clearly describes how to proceed with that:

To install the custom ConfigProvider implementation, add a new subdirectory containing the JAR files to the directory that is in Connect’s plugin.path and (re)start the Connect workers. When the Connect worker starts up it instantiates all ConfigProvider implementations specified in the worker configuration.

Source: https://docs.confluent.io/platform/current/connect/security.html

Unfortunately I noticed that this does not work, even though the startup logs show that the .jar files are indeed loaded. I’m wondering whether this correlates with the issue described in https://github.com/confluentinc/cp-docker-images/issues/815 . I tried all the suggested solutions but none worked for me. The problem seems to be common around the community as there are already blogposts describing the issues and workarounds (again none worked for me):

• https://www.novatec-gmbh.de/en/blog/using-a-secret-provider-with-kafka-connect/
• https://help.lenses.io/knowledge-base/secret-provider/secret-provider-on-confluent-images/

---

To reproduce:

Dockerfile I used:

FROM confluentinc/cp-kafka-connect:6.2.0

ENV KUBERNETES_CONFIG_PROVIDER_VERSION=0.1.0
ENV KUBERNETES_CONFIG_PROVIDER_DOWNLOAD=https://github.com/strimzi/kafka-kubernetes-config-provider/releases/download/${KUBERNETES_CONFIG_PROVIDER_VERSION}/kafka-kubernetes-config-provider-${KUBERNETES_CONFIG_PROVIDER_VERSION}.tar.gz
ENV KUBERNETES_CONFIG_PROVIDER_TARGET_PATH=/usr/share/java/strimzi-kubernetes-config-provider

ENV ENV_CONFIG_PROVIDER_VERSION=0.1.0
ENV ENV_CONFIG_PROVIDER_DOWNLOAD=https://github.com/strimzi/kafka-env-var-config-provider/releases/download/${ENV_CONFIG_PROVIDER_VERSION}/kafka-env-var-config-provider-${ENV_CONFIG_PROVIDER_VERSION}.tar.gz
ENV ENV_CONFIG_PROVIDER_TARGET_PATH=/usr/share/java/strimzi-env-var-config-provider

RUN set -e; \
    echo "===> Installing Kubernetes Config provider"; \
    mkdir ${KUBERNETES_CONFIG_PROVIDER_TARGET_PATH}; \
    curl -sLS ${KUBERNETES_CONFIG_PROVIDER_DOWNLOAD} | tar -xvz -C ${KUBERNETES_CONFIG_PROVIDER_TARGET_PATH} --strip-components=2;

RUN set -e; \
    echo "===> Installing Env Config provider"; \
    mkdir ${ENV_CONFIG_PROVIDER_TARGET_PATH}; \
    curl -sLS ${ENV_CONFIG_PROVIDER_DOWNLOAD} | tar -xvz -C ${ENV_CONFIG_PROVIDER_TARGET_PATH} --strip-components=2;

Jar files end up being where they are supposed to be:

And then start with a sth like command = ["/bin/connect-distributed", "/tmp/workers.properties"]

workers.properties:

bootstrap.servers=broker:29092
security.protocol=PLAINTEXT
sasl.mechanism=PLAIN
client.id=connect
producer.bootstrap.servers=broker:29092
producer.security.protocol=PLAINTEXT
producer.sasl.mechanism=PLAIN
producer.compression.type=gzip
group.id=connect
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=connect_configs
offset.storage.topic=connect_offsets
status.storage.topic=connect_statuses
config.providers=env
# config.providers.secrets.class=io.strimzi.kafka.KubernetesSecretConfigProvider
# config.providers.configmaps.class=io.strimzi.kafka.KubernetesConfigMapConfigProvider
config.providers.env.class=io.strimzi.kafka.EnvVarConfigProvider
plugin.path=/usr/share/java,/usr/share/confluent-hub-components

---

Relevant log messages during startup & failure:

[2021-09-10 19:39:22,503] INFO Loading plugin from: /usr/share/java/strimzi-kubernetes-config-provider (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:246)
[2021-09-10 19:39:22,902] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/strimzi-kubernetes-config-provider/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:269)


ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:86)
org.apache.kafka.common.config.ConfigException: Invalid value java.lang.ClassNotFoundException: io.strimzi.kafka.KubernetesConfigMapConfigProvider for configuration Invalid config:io.strimzi.kafka.KubernetesConfigMapConfigProvider ClassNotFoundException exception occurred

[C0urante]

I was able to reproduce locally (thanks for the detailed steps, @weeco!) and encountered the same issue.

After investigating, it appears that the issue is that the Strimzi config provider isn’t packaged with the required service loader provider configuration files that Connect uses (via the Java ServiceLoader API) to identify and load config providers.

See the Connect scanning logic for config providers and this PR for where the classloading logic was modified to use the ServiceLoader mechanism instead of the blanket class scanning approach used for connectors, transforms, etc.

See this file for an example of how the out-of-the-box FileConfigProvider is packaged with a provider configuration file so that Connect can discover it during plugin scanning.

See the source for the Strimzi config provider and an examination of the kafka-env-var-config-provider-0.1.0.jar artifact in the reproduction Docker container I set up:

[appuser@c8d149487882 java]$ jar tf /usr/share/java/strimzi-env-var-config-provider/kafka-env-var-config-provider-0.1.0.jar | sort
META-INF/
META-INF/MANIFEST.MF
META-INF/maven/
META-INF/maven/io.strimzi/
META-INF/maven/io.strimzi/kafka-env-var-config-provider/
META-INF/maven/io.strimzi/kafka-env-var-config-provider/pom.properties
META-INF/maven/io.strimzi/kafka-env-var-config-provider/pom.xml
io/
io/strimzi/
io/strimzi/kafka/
io/strimzi/kafka/EnvVarConfigProvider.class

for evidence that no provider configuration file is packaged with their config provider.

@weeco can you reach out to Strimzi and see if they agree with this assessment?

[vutkin]

Hi @weeco , so finally it works or still not? I have same problem even with version 0.1.1 Found a workaround: https://github.com/confluentinc/cp-docker-images/issues/828#issuecomment-720533576