Passing sasl.jaas.config property seems impossible when starting apicurio-registry-storage-kafka-1.2.3.Final with Docker

This is the follow up question raised in https://github.com/Apicurio/apicurio-registry/issues/10#issuecomment-657489360

It is still not clear to me if it is possible to run apicurio-registry-storage-kafka-1.2.3.Final in docker and setting sasl.jaas.config over the JAVA_OPTIONS environment variable.

I tried several variants.

1. First, a smoke test without passing JAVA_OPTIONS.

$ docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' apicurio/apicurio-registry-kafka:1.2.3.Final
exec java -Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar

__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2020-07-31 12:38:47,448 WARN  [io.qua.config] (main) Unrecognized configuration key "quarkus.datasource.username" was provided; it will be ignored
...
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
...

1. Setting JAVA_OPTIONS but omitting the required sasl.jaas.config property.

$ docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e 'JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN' apicurio/apicurio-registry-kafka:1.2.3.Final

exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
...
	sasl.mechanism = PLAIN
	security.protocol = SASL_PLAINTEXT
...
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:820)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666)
	at io.apicurio.registry.utils.kafka.ConsumerContainer.consumerLoop(ConsumerContainer.java:143)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
...

One can see the container is started but complains about the missing jaas.config as expected.

1. Setting JAVA_OPTIONS with the jaas.config (variant-1 using 'JAVA_OPTIONS=... ...sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret"')

docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e 'JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret"' apicurio/apicurio-registry-kafka:1.2.3.Final

exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
Error: Could not find or load main class required

It seems it didn’t even parse the properties correctly.

1. Setting JAVA_OPTIONS with the jaas.config (variant-2 'JAVA_OPTIONS=... ...sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret";"' )

docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e 'JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";"' apicurio/apicurio-registry-kafka:1.2.3.Final

exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
Error: Could not find or load main class required

the same error.

5 Setting JAVA_OPTIONS with the jaas.config (variant-3 "JAVA_OPTIONS=... ...sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\"")

docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e "JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\"" apicurio/apicurio-registry-kafka:1.2.3.Final

exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
Error: Could not find or load main class required

the same error

Could someone tell me if it not possible or if it is possible how I can start the docker container?

[jsenko]

We’ve discussed this issue and decided on supporting management of environment variables in the Apicurio CR like so:

spec:
 configuration:
   persistence: kafkasql
   kafkasql:
     bootstrapServers: "my-cluster-kafka-bootstrap.registry-example-kafkasql-scram.svc:9093"
 env:
    - key: REGISTRY_KAFKA_COMMON_SECURITY_PROTOCOL
      value: SASL_SSL
    - key: REGISTRY_KAFKA_COMMON_SASL_JAAS_CONFIG
      value: "software.amazon.msk.auth.iam.IAMLoginModule required;"
    - key: REGISTRY_KAFKA_COMMON_SASL_MECHANISM
      value: AWS_MSK_IAM
    - key: REGISTRY_KAFKA_COMMON_SASL_CLIENT_CALLBACK_HANDLER_CLASS
      value: software.amazon.msk.auth.iam.IAMClientCallbackHandler

Solving https://github.com/Apicurio/apicurio-registry-operator/issues/167 and adding documentation about available env. variables.

[forsberg]

@PlugaruT - glad you made it work! 👍

As a general observation, sometimes having a look at /proc/<pid> to see where /proc/<pid>/pwd points to can be enlightening when it comes to the real working directory of a program.