cp-kafka-connect: Custom ConfigProvider ClassNotFoundExceptions on startup
See original GitHub issueHey, I just tried the cp-kafka-connect docker image and I want to install some custom ConfigProviders. The Confluent documentation clearly describes how to proceed with that:
To install the custom ConfigProvider implementation, add a new subdirectory containing the JAR files to the directory that is in Connect’s plugin.path and (re)start the Connect workers. When the Connect worker starts up it instantiates all ConfigProvider implementations specified in the worker configuration.
Source: https://docs.confluent.io/platform/current/connect/security.html
Unfortunately I noticed that this does not work, even though the startup logs show that the .jar files are indeed loaded. I’m wondering whether this correlates with the issue described in https://github.com/confluentinc/cp-docker-images/issues/815 . I tried all the suggested solutions but none worked for me. The problem seems to be common around the community as there are already blogposts describing the issues and workarounds (again none worked for me):
- https://www.novatec-gmbh.de/en/blog/using-a-secret-provider-with-kafka-connect/
- https://help.lenses.io/knowledge-base/secret-provider/secret-provider-on-confluent-images/
To reproduce:
Dockerfile I used:
FROM confluentinc/cp-kafka-connect:6.2.0
ENV KUBERNETES_CONFIG_PROVIDER_VERSION=0.1.0
ENV KUBERNETES_CONFIG_PROVIDER_DOWNLOAD=https://github.com/strimzi/kafka-kubernetes-config-provider/releases/download/${KUBERNETES_CONFIG_PROVIDER_VERSION}/kafka-kubernetes-config-provider-${KUBERNETES_CONFIG_PROVIDER_VERSION}.tar.gz
ENV KUBERNETES_CONFIG_PROVIDER_TARGET_PATH=/usr/share/java/strimzi-kubernetes-config-provider
ENV ENV_CONFIG_PROVIDER_VERSION=0.1.0
ENV ENV_CONFIG_PROVIDER_DOWNLOAD=https://github.com/strimzi/kafka-env-var-config-provider/releases/download/${ENV_CONFIG_PROVIDER_VERSION}/kafka-env-var-config-provider-${ENV_CONFIG_PROVIDER_VERSION}.tar.gz
ENV ENV_CONFIG_PROVIDER_TARGET_PATH=/usr/share/java/strimzi-env-var-config-provider
RUN set -e; \
echo "===> Installing Kubernetes Config provider"; \
mkdir ${KUBERNETES_CONFIG_PROVIDER_TARGET_PATH}; \
curl -sLS ${KUBERNETES_CONFIG_PROVIDER_DOWNLOAD} | tar -xvz -C ${KUBERNETES_CONFIG_PROVIDER_TARGET_PATH} --strip-components=2;
RUN set -e; \
echo "===> Installing Env Config provider"; \
mkdir ${ENV_CONFIG_PROVIDER_TARGET_PATH}; \
curl -sLS ${ENV_CONFIG_PROVIDER_DOWNLOAD} | tar -xvz -C ${ENV_CONFIG_PROVIDER_TARGET_PATH} --strip-components=2;
Jar files end up being where they are supposed to be:
And then start with a sth like command = ["/bin/connect-distributed", "/tmp/workers.properties"]
workers.properties:
bootstrap.servers=broker:29092
security.protocol=PLAINTEXT
sasl.mechanism=PLAIN
client.id=connect
producer.bootstrap.servers=broker:29092
producer.security.protocol=PLAINTEXT
producer.sasl.mechanism=PLAIN
producer.compression.type=gzip
group.id=connect
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=connect_configs
offset.storage.topic=connect_offsets
status.storage.topic=connect_statuses
config.providers=env
# config.providers.secrets.class=io.strimzi.kafka.KubernetesSecretConfigProvider
# config.providers.configmaps.class=io.strimzi.kafka.KubernetesConfigMapConfigProvider
config.providers.env.class=io.strimzi.kafka.EnvVarConfigProvider
plugin.path=/usr/share/java,/usr/share/confluent-hub-components
Relevant log messages during startup & failure:
[2021-09-10 19:39:22,503] INFO Loading plugin from: /usr/share/java/strimzi-kubernetes-config-provider (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:246)
[2021-09-10 19:39:22,902] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/strimzi-kubernetes-config-provider/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:269)
ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed:86)
org.apache.kafka.common.config.ConfigException: Invalid value java.lang.ClassNotFoundException: io.strimzi.kafka.KubernetesConfigMapConfigProvider for configuration Invalid config:io.strimzi.kafka.KubernetesConfigMapConfigProvider ClassNotFoundException exception occurred
Issue Analytics
- State:
- Created 2 years ago
- Comments:5
I was able to reproduce locally (thanks for the detailed steps, @weeco!) and encountered the same issue.
After investigating, it appears that the issue is that the Strimzi config provider isn’t packaged with the required service loader provider configuration files that Connect uses (via the Java
ServiceLoader
API) to identify and load config providers.See the Connect scanning logic for config providers and this PR for where the classloading logic was modified to use the
ServiceLoader
mechanism instead of the blanket class scanning approach used for connectors, transforms, etc.See this file for an example of how the out-of-the-box
FileConfigProvider
is packaged with a provider configuration file so that Connect can discover it during plugin scanning.See the source for the Strimzi config provider and an examination of the
kafka-env-var-config-provider-0.1.0.jar
artifact in the reproduction Docker container I set up:for evidence that no provider configuration file is packaged with their config provider.
@weeco can you reach out to Strimzi and see if they agree with this assessment?
Hi @weeco , so finally it works or still not? I have same problem even with version 0.1.1 Found a workaround: https://github.com/confluentinc/cp-docker-images/issues/828#issuecomment-720533576