Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Setting max.request.size for Producer

See original GitHub issue

Similarly to #2592, I’m setting the value for max.request.size in the KafkaConnect CRD:

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
  name: connect-cluster
spec:
  version: 2.4.0
  replicas: 1
  bootstrapServers: kafka-cluster-kafka-bootstrap:9093
  ...
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
    offset.flush.timeout.ms: 50000
    max.request.size: 8388608

The log shows the correct value at first:

# Provided configuration
..
max.request.size=8388608
..

Until I start my postgres connector via the REST interface:

2020-03-18 20:40:20,082 INFO Instantiated connector postgres-connector with version 1.0.2.Final of type class io.debezium.connector.postgresql.PostgresConnector (org.apache.kafka.connect.runtime.Worker) [pool-6-thread-1]
2020-03-18 20:40:20,086 INFO ProducerConfig values:
...
	max.request.size = 1048576
...

Is there any way to prevent max.request.size from being reset to the default value?

I’m using:

Strimzi 0.16.2
A custom Kafka Connect image based on strimzi/kafka:0.16.2-kafka-2.4.0

Issue Analytics

State:
Created 4 years ago
Comments:6 (5 by maintainers)

Top GitHub Comments

4reactions

rsolanocommented, Mar 19, 2020

Thanks for the quick response @scholzj.

It turns out I just needed to set producer.max.request.size to ensure the producer worker picked it up.

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
  name: connect-cluster
spec:
  ...
  config:
    ..
    max.request.size: 8388608
    producer.max.request.size: 8388608

Not sure why my consumer doesn’t require a consumer.max.request.size counterpart, but it works.

0reactions

tombentleycommented, Mar 19, 2020

@scholzj I can’t explain the observations in #2592, unless the config was being overridden for that connector. The producer configs get created in Worker#producerConfigs certainly doesn’t do anything which could explain what you saw.