Need advice for getting Datahub to connect to Kafka Schema Registry using SSL

Hi,

Need some advice, getting crazy here.

I am already having the following env variable set in pods (cut from kubectl describe pod…):

Environment:

  KAFKA_BOOTSTRAP_SERVER:                                           kafka.confluent.svc.cluster.local:9092
  KAFKA_SCHEMAREGISTRY_URL:                                         https://schemaregistry.confluent.svc.cluster.local:8443
  GMS_HOST:                                                         kindred-datahub-datahub-gms
  GMS_PORT:                                                         8080
  SPRING_KAFKA_PROPERTIES_SCHEMA_REGISTRY_SSL_KEYSTORE_LOCATION:    /mnt/kindred-datahub/certs/kindred-datahub.d2-pt.aws.kindredgroup.com.keystore.jks
  SPRING_KAFKA_PROPERTIES_SCHEMA_REGISTRY_SSL_TRUSTSTORE_LOCATION:  /mnt/kindred-datahub/certs/kindred-datahub.d2-pt.aws.kindredgroup.com.truststore.jks
  SPRING_KAFKA_PROPERTIES_SCHEMA_REGISTRY_URL:                      https://schemaregistry.confluent.svc.cluster.local:8443
  SPRING_KAFKA_PROPERTIES_SECURITY_PROTOCOL:                        SSL
  SPRING_KAFKA_PROPERTIES_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:    https
  SPRING_KAFKA_PROPERTIES_SSL_KEYSTORE_LOCATION:                    /mnt/kindred-datahub/certs/kindred-datahub.d2-pt.aws.kindredgroup.com.keystore.jks
  SPRING_KAFKA_PROPERTIES_SSL_KEYSTORE_TYPE:                        JKS
  SPRING_KAFKA_PROPERTIES_SSL_PROTOCOL:                             TLS
  SPRING_KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION:                  /mnt/kindred-datahub/certs/kindred-datahub.d2-pt.aws.kindredgroup.com.truststore.jks
  SPRING_KAFKA_PROPERTIES_SSL_TRUSTSTORE_TYPE:                      JKS
  SPRING_KAFKA_PROPERTIES_SCHEMA_REGISTRY_SSL_KEY_PASSWORD:         <set to the key 'kindred-datahub.d2-pt.aws.kindredgroup.com.KeyPass' in secret 'kindred-datahub-certs'>         Optional: false
  SPRING_KAFKA_PROPERTIES_SCHEMA_REGISTRY_SSL_KEYSTORE_PASSWORD:    <set to the key 'kindred-datahub.d2-pt.aws.kindredgroup.com.KeyStorePass' in secret 'kindred-datahub-certs'>    Optional: false
  SPRING_KAFKA_PROPERTIES_SCHEMA_REGISTRY_SSL_TRUSTSTORE_PASSWORD:  <set to the key 'kindred-datahub.d2-pt.aws.kindredgroup.com.TrustStorePass' in secret 'kindred-datahub-certs'>  Optional: false
  SPRING_KAFKA_PROPERTIES_SSL_KEY_PASSWORD:                         <set to the key 'kindred-datahub.d2-pt.aws.kindredgroup.com.KeyPass' in secret 'kindred-datahub-certs'>         Optional: false
  SPRING_KAFKA_PROPERTIES_SSL_KEYSTORE_PASSWORD:                    <set to the key 'kindred-datahub.d2-pt.aws.kindredgroup.com.KeyStorePass' in secret 'kindred-datahub-certs'>    Optional: false
  SPRING_KAFKA_PROPERTIES_SSL_TRUSTSTORE_PASSWORD:                  <set to the key 'kindred-datahub.d2-pt.aws.kindredgroup.com.TrustStorePass' in secret 'kindred-datahub-certs'>  Optional: false
Mounts:
  /mnt/kindred-datahub/certs from kindred-datahub-certs-dir (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kindred-datahub-datahub-mce-consumer-token-ml4ph (ro)

But I am still hitting the following exceptions: kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition MetadataChangeEvent_v4-0 at offset 22. If needed, please seek past the record to continue consumption. kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 395 kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:208) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:252) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:482) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:475) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getSchemaByIdFromRegistry(CachedSchemaRegistryClient.java:153) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getBySubjectAndId(CachedSchemaRegistryClient.java:232) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getById(CachedSchemaRegistryClient.java:211) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:116) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:88) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:55) kindred-datahub-datahub-mce-consumer-76b967cfbc-xcqrc datahub-mce-consumer at org.apache.kafka.common.serialization.Deserializer.deserialize(Deserializer.java:58) What am I missing here? Thank in advance!

[grantatspothero]

Yes, this is probably the same issue, all the schema registry configuration was being ignored due to the old client library.

I will open a PR today

[themightylaz]

Hi @mars-lan , app is now building fine with avro serde bumped up to 5.5.1. Now will test and then create PR