[Question] ...S3 CamelSinkConnector is not dumping data into S3 bucket

Using the following documents as reference: https://ibm-cloud-architecture.github.io/refarch-eda/scenarios/connect-s3/ and https://developers.redhat.com/blog/2020/02/14/using-secrets-in-apache-kafka-connect-configuration/

I can successfully created Strimzi kafka connect cluster and S3 sink, but the data is not dumped into S3. What Am I missing here??!!

Here is my image Docker file

FROM strimzi/kafka:0.18.0-kafka-2.5.0
USER root:root
COPY ./my-plugins/ /opt/kafka/plugins/
USER 1001

Here is my plugins folder

Here is my cluster deployment yaml file

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
  name: my-connect-cluster
spec:
  image: selumalai/selumalai-s3-kafkaconnect
  replicas: 1
  bootstrapServers: 34.214.12.113:9092
  externalConfiguration:
    volumes:
      - name: aws-credentials
        secret:
          secretName: aws-credentials
  config:
    config.providers: file
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false

Here is my KafkaConnector yaml file

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: s3-sink-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.camel.kafkaconnector.CamelSinkConnector
  tasksMax: 1
  config:
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    topics: my-replicated-topic 
    camel.sink.url: aws-s3://selumalai-kafka-s3?keyName=${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}
    camel.sink.maxPollDuration: 10000
    camel.component.aws-s3.configuration.autocloseBody: false
    camel.component.aws-s3.accessKey: ${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_access_key_id}
    camel.component.aws-s3.secretKey: ${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_secret_access_key}
    camel.component.aws-s3.region: US_WEST_2

output of kubectl get kctr

ML-C02Z605SLVDQ:selumalai-k8s-s3-connect e192270$ kubectl get kctr --selector strimzi.io/cluster=my-connect-cluster -o name
kafkaconnector.kafka.strimzi.io/s3-sink-connector

Here is the last log of the pod

… 2020-05-28 02:42:34,925 WARN [Worker clientId=connect-1, groupId=connect-cluster] Catching up to assignment’s config offset. (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-05-28 02:42:34,925 INFO [Worker clientId=connect-1, groupId=connect-cluster] Current config state offset -1 is behind group assignment 2, reading to end of config log (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-05-28 02:42:35,635 INFO [Worker clientId=connect-1, groupId=connect-cluster] Finished reading to end of log and updated config snapshot, new config log offset: 2 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-05-28 02:42:35,635 INFO [Worker clientId=connect-1, groupId=connect-cluster] Starting connectors and tasks using config offset 2 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-05-28 02:42:35,635 INFO [Worker clientId=connect-1, groupId=connect-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1] 2020-05-28 02:42:36,927 INFO [Worker clientId=connect-1, groupId=connect-cluster] Session key updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [KafkaBasedLog Work Thread - connect-cluster-configs] 2020-05-28 02:43:19,890 INFO JVM Runtime does not support Modules (org.eclipse.jetty.util.TypeUtil) [qtp818785904-49] …

Why data is not pushed to S3.?? I am unable to find pod logs showing the data is getting consumed. Any help will be appreciated!!

[scholzj]

Normally the SSL for connecting to Kafka is configured in Connect and not in the connector. The S3 connector should IMHO not need its own connection to Kafka, it just passes the messages to/from Connect framework.

[cristianburca]

I think I found the answer here https://github.com/confluentinc/kafka-connect-storage-cloud/issues/180#issuecomment-607882088 I missed the SLL params definition for consumer and producer.

Connect config

CONNECT_BOOTSTRAP_SERVERS: “kafka1:9094,kafka2:9094,kafka3:9094” CONNECT_SECURITY_PROTOCOL: SSL CONNECT_SSL_TRUSTSTORE_LOCATION: /root/kafka_credentials/truststore.jks CONNECT_SSL_TRUSTSTORE_PASSWORD: PASSWORD CONNECT_SSL_KEYSTORE_LOCATION: /root/kafka_credentials/client-keystore.jks CONNECT_SSL_KEYSTORE_PASSWORD: PASSWORD CONNECT_SSL_KEY_PASSWORD: PASSWORD

To solve this issue I added this ones too

Producer config

CONNECT_PRODUCER_SECURITY_PROTOCOL: SSL CONNECT_PRODUCER_SSL_TRUSTSTORE_LOCATION: /root/kafka_credentials/truststore.jks CONNECT_PRODUCER_SSL_TRUSTSTORE_PASSWORD: <PASSWORD> CONNECT_PRODUCER_SSL_KEYSTORE_LOCATION: /root/kafka_credentials/client-keystore.jks CONNECT_PRODUCER_SSL_KEYSTORE_PASSWORD: <PASSWORD> CONNECT_PRODUCER_SSL_KEY_PASSWORD: <PASSWORD>

Consumer Config

CONNECT_CONSUMER_SECURITY_PROTOCOL: SSL CONNECT_CONSUMER_SSL_TRUSTSTORE_LOCATION: /root/kafka_credentials/truststore.jks CONNECT_CONSUMER_SSL_TRUSTSTORE_PASSWORD: <PASSWORD> CONNECT_CONSUMER_SSL_KEYSTORE_LOCATION: /root/kafka_credentials/client-keystore.jks CONNECT_CONSUMER_SSL_KEYSTORE_PASSWORD: <PASSWORD> CONNECT_CONSUMER_SSL_KEY_PASSWORD: <PASSWORD>