Kafka SSL Native test on FIPS enforcing system: sun.security.pkcs11.SunPKCS11 on image heap

Describe the bug

Running with a FIPS aware HotSpot on a FIPS enforcing system, the Kafka SSL test passes just fine:

[INFO] Quarkus - Integration Tests - Kafka SSL ............ SUCCESS [  2.338 s]

Attempt at running it in Native fails though:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on native-image 21.3.1.0-Final Mandrel Distribution (Java Version 11.0.14.1+1-LTS)
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /home/karm/fips-mandrel-java11-21.3.1.0-Final/bin/native-image -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 -H:-ParseOnce -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+JNI -H:+AllowFoldMethods -J-Djava.awt.headless=true -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:NativeLinkerOption=-no-pie -H:-UseServiceLoaderFeature -H:+StackTrace -H:AdditionalSecurityProviders=com.sun.security.sasl.Provider,org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientProvider,org.apache.kafka.common.security.scram.internals.ScramSaslClientProvider quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner -jar quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner.jar
[quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner:38119]    classlist:   3,552.37 ms,  0.96 GB
[quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner:38119]        (cap):     590.14 ms,  1.18 GB
[quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner:38119]        setup:   2,681.96 ms,  1.18 GB
14:13:42,005 INFO  [org.jbo.threads] JBoss Threads version 3.4.2.Final
[quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner:38119]     (clinit):     818.61 ms,  3.98 GB
[quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner:38119]   (typeflow):   4,891.22 ms,  3.98 GB
[quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner:38119]    (objects):  41,738.06 ms,  3.98 GB
[quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner:38119]   (features):   2,062.13 ms,  3.98 GB
[quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner:38119]     analysis:  51,434.72 ms,  3.98 GB
Error: No instances of sun.security.pkcs11.SunPKCS11 are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=sun.security.pkcs11.SunPKCS11.
Detailed message:
Trace: Object was reached by 
	reading field java.security.Provider$Service.provider of
		constant sun.security.pkcs11.SunPKCS11$P11Service@62a54948 reached by 
	reading field sun.security.jca.ProviderList$ServiceList.firstService of
		constant sun.security.jca.ProviderList$ServiceList@3dc4f539 reached by 
	reading field sun.security.jca.ProviderList$ServiceList$1.this$1 of
		constant sun.security.jca.ProviderList$ServiceList$1@5b3f6188 reached by 
	reading field java.security.KeyFactory.serviceIterator of
		constant java.security.KeyFactory@56c281df reached by 
	indexing into array
		constant java.security.KeyFactory[]@49b9d1aa reached by 
	reading field java.util.Arrays$ArrayList.a of
		constant java.util.Arrays$ArrayList@2eb5d4ae reached by 
	scanning method org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:522)
Call path from entry point to org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(String, char[]): 
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:501)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.createKeyStoreFromPem(DefaultSslEngineFactory.java:462)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.<init>(DefaultSslEngineFactory.java:435)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:284)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:161)
	at org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.getAssignorInstances(ConsumerPartitionAssignor.java:287)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:769)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:664)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:645)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:625)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint.createConsumer(SslKafkaEndpoint.java:63)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint.create(SslKafkaEndpoint.java:36)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint_Bean.create(Unknown Source)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.RequestContext.getIfActive(RequestContext.java:70)
	at io.quarkus.arc.impl.RequestContext.get(RequestContext.java:78)
	at io.quarkus.micrometer.runtime.binder.kafka.KafkaEventObserver_Observer_onStop_a7d706cbdf99711cb5dda67189ced7f36529b6b8.notify(Unknown Source)
	at io.quarkus.arc.impl.EventImpl$DeferredEventNotification.run(EventImpl.java:460)
	at io.micrometer.core.instrument.binder.jvm.JvmHeapPressureMetrics$$Lambda$1720/0x00000007c1a9e440.accept(Unknown Source)
	at java.base@11.0.14.1/java.util.ArrayList.forEach(ArrayList.java:1541)
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST_ArrayList_forEach_5467a6ce4be9f47657a3fc17ababe734349fec61(generated:0)

com.oracle.svm.core.util.UserError$UserException: No instances of sun.security.pkcs11.SunPKCS11 are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=sun.security.pkcs11.SunPKCS11.
Detailed message:
Trace: Object was reached by 
	reading field java.security.Provider$Service.provider of
		constant sun.security.pkcs11.SunPKCS11$P11Service@62a54948 reached by 
	reading field sun.security.jca.ProviderList$ServiceList.firstService of
		constant sun.security.jca.ProviderList$ServiceList@3dc4f539 reached by 
	reading field sun.security.jca.ProviderList$ServiceList$1.this$1 of
		constant sun.security.jca.ProviderList$ServiceList$1@5b3f6188 reached by 
	reading field java.security.KeyFactory.serviceIterator of
		constant java.security.KeyFactory@56c281df reached by 
	indexing into array
		constant java.security.KeyFactory[]@49b9d1aa reached by 
	reading field java.util.Arrays$ArrayList.a of
		constant java.util.Arrays$ArrayList@2eb5d4ae reached by 
	scanning method org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:522)
Call path from entry point to org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(String, char[]): 
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:501)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.createKeyStoreFromPem(DefaultSslEngineFactory.java:462)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.<init>(DefaultSslEngineFactory.java:435)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:284)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:161)
	at org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.getAssignorInstances(ConsumerPartitionAssignor.java:287)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:769)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:664)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:645)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:625)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint.createConsumer(SslKafkaEndpoint.java:63)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint.create(SslKafkaEndpoint.java:36)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint_Bean.create(Unknown Source)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.RequestContext.getIfActive(RequestContext.java:70)
	at io.quarkus.arc.impl.RequestContext.get(RequestContext.java:78)
	at io.quarkus.micrometer.runtime.binder.kafka.KafkaEventObserver_Observer_onStop_a7d706cbdf99711cb5dda67189ced7f36529b6b8.notify(Unknown Source)
	at io.quarkus.arc.impl.EventImpl$DeferredEventNotification.run(EventImpl.java:460)
	at io.micrometer.core.instrument.binder.jvm.JvmHeapPressureMetrics$$Lambda$1720/0x00000007c1a9e440.accept(Unknown Source)
	at java.base@11.0.14.1/java.util.ArrayList.forEach(ArrayList.java:1541)
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST_ArrayList_forEach_5467a6ce4be9f47657a3fc17ababe734349fec61(generated:0)

	at com.oracle.svm.core.util.UserError.abort(UserError.java:87)
	at com.oracle.svm.hosted.FallbackFeature.reportAsFallback(FallbackFeature.java:233)
	at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:759)
	at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:529)
	at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:488)
	at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:403)
	at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:569)
	at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:122)
	at com.oracle.svm.hosted.NativeImageGeneratorRunner$JDK9Plus.main(NativeImageGeneratorRunner.java:599)
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: No instances of sun.security.pkcs11.SunPKCS11 are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=sun.security.pkcs11.SunPKCS11.
Detailed message:
Trace: Object was reached by 
	reading field java.security.Provider$Service.provider of
		constant sun.security.pkcs11.SunPKCS11$P11Service@62a54948 reached by 
	reading field sun.security.jca.ProviderList$ServiceList.firstService of
		constant sun.security.jca.ProviderList$ServiceList@3dc4f539 reached by 
	reading field sun.security.jca.ProviderList$ServiceList$1.this$1 of
		constant sun.security.jca.ProviderList$ServiceList$1@5b3f6188 reached by 
	reading field java.security.KeyFactory.serviceIterator of
		constant java.security.KeyFactory@56c281df reached by 
	indexing into array
		constant java.security.KeyFactory[]@49b9d1aa reached by 
	reading field java.util.Arrays$ArrayList.a of
		constant java.util.Arrays$ArrayList@2eb5d4ae reached by 
	scanning method org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:522)
Call path from entry point to org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(String, char[]): 
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:501)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.createKeyStoreFromPem(DefaultSslEngineFactory.java:462)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.<init>(DefaultSslEngineFactory.java:435)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:284)
	at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:161)
	at org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.getAssignorInstances(ConsumerPartitionAssignor.java:287)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:769)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:664)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:645)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:625)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint.createConsumer(SslKafkaEndpoint.java:63)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint.create(SslKafkaEndpoint.java:36)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint_Bean.create(Unknown Source)
	at io.quarkus.it.kafka.ssl.SslKafkaEndpoint_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.RequestContext.getIfActive(RequestContext.java:70)
	at io.quarkus.arc.impl.RequestContext.get(RequestContext.java:78)
	at io.quarkus.micrometer.runtime.binder.kafka.KafkaEventObserver_Observer_onStop_a7d706cbdf99711cb5dda67189ced7f36529b6b8.notify(Unknown Source)
	at io.quarkus.arc.impl.EventImpl$DeferredEventNotification.run(EventImpl.java:460)
	at io.micrometer.core.instrument.binder.jvm.JvmHeapPressureMetrics$$Lambda$1720/0x00000007c1a9e440.accept(Unknown Source)
	at java.base@11.0.14.1/java.util.ArrayList.forEach(ArrayList.java:1541)
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST_ArrayList_forEach_5467a6ce4be9f47657a3fc17ababe734349fec61(generated:0)

	at com.oracle.graal.pointsto.constraints.UnsupportedFeatures.report(UnsupportedFeatures.java:126)
	at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:756)
	... 6 more
[quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner:38119]      [total]:  57,975.38 ms,  3.98 GB
# Printing build artifacts to: /home/karm/quarkus/integration-tests/kafka-ssl/target/quarkus-integration-test-kafka-ssl-999-SNAPSHOT-native-image-source-jar/quarkus-integration-test-kafka-ssl-999-SNAPSHOT-runner.build_artifacts.txt

Expected behavior

Passes both for HotSpot and Native.

Actual behavior

Native fails.

How to Reproduce?

On a FIPS enforcing system, using FIPS aware native-image:

./mvnw clean install -Dquickly -pl '!devtools/gradle,!devtools/gradle/gradle-model,!devtools/gradle/gradle-extension-plugin,!devtools/gradle/gradle-application-plugin,!integration-tests/gradle'
./mvnw verify -f integration-tests/pom.xml --fail-at-end --batch-mode -Dno-format -DfailIfNoTests=false -Dnative -pl kafka-ssl

Output of uname -a or ver

Linux rhel9fips 5.14.0-63.el9.x86_64

Output of java -version

Red Hat build of OpenJDK 64-Bit Server VM 18.9 (build 11.0.14.1+1-LTS, mixed mode)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

95cc838f

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[jerboaa]

@jerboaa @zakkak I have it patched and working with FIPS +1

--- a/extensions/kafka-client/deployment/src/main/java/io/quarkus/kafka/client/deployment/KafkaProcessor.java
+++ b/extensions/kafka-client/deployment/src/main/java/io/quarkus/kafka/client/deployment/KafkaProcessor.java
@@ -519,6 +519,8 @@ public class KafkaProcessor {
                 new RuntimeInitializedClassBuildItem("org.apache.kafka.common.security.authenticator.SaslClientAuthenticator"));
         producer.produce(new RuntimeInitializedClassBuildItem(
                 "org.apache.kafka.common.security.oauthbearer.internals.expiring.ExpiringCredentialRefreshingLogin"));
+        producer.produce(new RuntimeInitializedClassBuildItem(
+                "org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore"));
     }

Karm@4f5ce01

I don’t think we should fix this on the quarkus side. It should be fixed in native-image instead. At least to get to the same state where JDK in FIPS mode is.

[cescoffier]

This is expected. The user is responsible for passing a SslEngineFactory in the kafka consumer/producer configuration. So, you need to define the kafka.ssl.engine.factory.class property and point to a valid (FIPS-aware) SSL Engine Factory (org.apache.kafka.common.security.auth.SslEngineFactory).

To be clear, while I’m aware of this feature, I never had to use it.