S3 Sink connector, error during offset commit

What version of the Stream Reactor are you reporting this issue for?

kafka-connect-aws-s3-3.0.1-2.5.0-all.tar.gz

What is the expected behaviour?

Records to written to AWS S3

What was observed?

WARN [sink-s3|task-0] WorkerSinkTask{id=sink-s3-0} Offset commit failed during close (org.apache.kafka.connect.runtime.WorkerSinkTask:390)
ERROR [sink-s3|task-0] WorkerSinkTask{id=sink-s3-0} Commit of offsets threw an unexpected exception for sequence number 1: null (org.apache.kafka.connect.runtime.WorkerSinkTask:267)
java.lang.UnsupportedOperationException: empty.maxBy

What is your Connect cluster configuration (connect-avro-distributed.properties)?

- name: CONNECT_REST_PORT
  value: "8083"
- name: CONNECT_BOOTSTRAP_SERVERS
  value: "blockchain-kafka-kafka-0.default.svc.cluster.local:9092"
- name: CONNECT_GROUP_ID
  value: "kafka-connect"
- name: CONNECT_CONFIG_STORAGE_TOPIC
  value: "_connect-configs"
- name: CONNECT_OFFSET_STORAGE_TOPIC
  value: "_connect-offsets"
- name: CONNECT_STATUS_STORAGE_TOPIC
  value: "_connect-status"
- name: CONNECT_KEY_CONVERTER
  value: "org.apache.kafka.connect.storage.StringConverter"
- name: CONNECT_VALUE_CONVERTER
  value: "org.apache.kafka.connect.json.JsonConverter"
- name: CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE
  value: "false"
- name: CONNECT_REST_ADVERTISED_HOST_NAME
  value: "kafka-connect"
- name: CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN
  value: "[%d] %p %X{connector.context}%m (%c:%L)%n"
- name: CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR
  value: "1"
- name: CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR
  value: "1"
- name: CONNECT_STATUS_STORAGE_REPLICATION_FACTOR
  value: "1"
- name: AWS_ACCESS_KEY_ID

What is your connector properties configuration (my-connector.properties)?

 {
"connector.class": "io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector",
"topics": "test_yordan_kafka_connect_single_partition",
"tasks.max": "1",
"aws.auth.mode": "Default",
"topics": "test_yordan_kafka_connect_single_partition",
"connect.s3.kcql": "insert into yordan-flink-savepoints-test-hz-stage:test-bucket select * from test_yordan_kafka_connect_single_partition `json` WITH_FLUSH_COUNT = 5000",
"connect.s3.aws.region": "eu-central-1",
"timezone": "UTC",
"errors.log.enable":true
}

Please provide full log files (redact and sensitive information)

kafka-connect.log

[davidsloan]

@YordanPavlov This is a common issue.

The jClouds library gives us problems here because it tries to work out the region but sometimes gets it wrong. Either

1. add the configurations: connect.s3.aws.client=AWS connect.s3.aws.region=eu-west-1 (Change the region to the relevant one.

OR

1. Define a custom URL to target a specific region like this: connect.s3.custom.endpoint=https://s3.eu-west-1.amazonaws.com/ This second is approach mentioned in the README here: https://github.com/lensesio/stream-reactor/blob/master/kafka-connect-aws-s3/README-sink.md#connects3customendpoint

[YordanPavlov]

Thank you for your support, the connector works for me now.