"group.id" of ConsumerPropertie be replaced by "kafka-supervisor-%s" in kafka-indexing-service

Affected Version

druid-0.16.0-incubating

Description

In the class org.apache.druid.indexing.kafka.KafkaRecordSupplier

the function getKafkaConsumer use default consumerConfigs to overwrite the user’s consumerProperties

private KafkaConsumer<byte[], byte[]> getKafkaConsumer() { final Map<String, Object> consumerConfigs = KafkaConsumerConfigs.getConsumerProperties(); final Properties props = new Properties(); addConsumerPropertiesFromConfig(props, sortingMapper, consumerProperties); props.putAll(consumerConfigs);

consumerConfigs is public class KafkaConsumerConfigs { public static Map<String, Object> getConsumerProperties() { final Map<String, Object> props = new HashMap<>(); props.put("metadata.max.age.ms", "10000"); props.put("group.id", StringUtils.format("kafka-supervisor-%s", RandomIdUtils.getRandomId())); props.put("auto.offset.reset", "none"); props.put("enable.auto.commit", "false"); props.put("isolation.level", "read_committed"); return props; } }

so the group.id of user’s ConsumerProperties will be repalced by the consumerConfigs from KafkaConsumerConfigs.getConsumerProperties()

I think exchange the place of the 2 line will solve the problem

addConsumerPropertiesFromConfig(props, sortingMapper, consumerProperties); props.putAll(consumerConfigs);

the code addConsumerPropertiesFromConfig(props, sortingMapper, consumerProperties); should be placed after the code props.putAll(consumerConfigs);

[FrankChen021]

I have not dived into Kafka to see which requests from consumer are limited by kafka brokers. As far as I know, poll operation of KafkaComsumer does not send group.id to kafka brokers which means that operation could not be restricted.

Since kafka-acl allows ACLs to be configured on a specific consumer group, I think it makes sense to make group.id configurable.

[asdf2014]

@phuchuynh9994 Make sense to me. Otherwise, the Kafka business data with Consumer authentication enabled cannot be written to the Druid cluster. Maybe we should make the group.id option is configurable, but by default, we still generate it randomly.

cc @gianm