Kafka Streams Binder error using interceptor.classes for Consumer or Producer

I’ve found a problem upgrading to Spring Cloud Hoxton.SR11 (and 2020.0.2).

When using spring.cloud.stream.kafka.streams.binder. consumerProperties.interceptor.classes=xxx or producerProperties.interceptor.classes=xxx with Kafka Streams Binder a java.lang.ClassCastException is thrown.

The changes made for #997 in KafkaStreamsBinderSupportAutoConfiguration are putting both interceptor.classes entries in the global properties map, and I suspect that’s the problem.

The producer property overrides the consumer one, when creating the kafka consumer it tries to cast the producer interceptor to org.apache.kafka.clients.consumer.ConsumerInterceptor throwing a ClassCastException.
If I only use a consumer interceptor, eventually when creating the kafka producer it reads the global interceptor.classes entry and tries to cast the consumer interceptor to org.apache.kafka.clients.producer.ProducerInterceptor also throwing a ClassCastException.

I’ve made a change to kafka-streams-word-count sample project to reproduce the problem: kafka-streams-word-count. Just run the test class KafkaStreamsWordCountApplicationTests to see the exception being thrown.

ps.: also tried setting the interceptors using spring.cloud.stream.kafka.binder prefix with the same outcome.

[ceccone]

Thanks for the feedback @garyrussell.

I was digging into KafkaStreams source code and issue tracker to find the problem and if it was ever reported.

Then, I came across org.apache.kafka.streams.StreamsConfig, which says that we can use prefixes to isolate consumer/producer configs from other client configs. For instance, we have these options there:

public static final String CONSUMER_PREFIX = "consumer.";
public static final String MAIN_CONSUMER_PREFIX = "main.consumer.";
public static final String RESTORE_CONSUMER_PREFIX = "restore.consumer.";
public static final String GLOBAL_CONSUMER_PREFIX = "global.consumer.";
public static final String PRODUCER_PREFIX = "producer.";

And StreamsConfig’s methods: getCommonConsumerConfigs, getMainConsumerConfigs, getRestoreConsumerConfigs, getGlobalConsumerConfigs and getProducerConfigs take these prefixes into account.

With this in mind, I was able to make it work by using the prefixes:

spring.cloud.stream.kafka.streams.binder.consumer-properties.consumer.interceptor.classes=MyConsumerInterceptor
spring.cloud.stream.kafka.streams.binder.producer-properties.producer.interceptor.classes=MyProducerInterceptor

Tested in the sample project kafka-streams-word-count and it worked.

I still think that the change d1c62bb could have used the prefixes (if they weren’t present) to avoid the ClassCastException.

[sobychacko]

@ceccone @garyrussell. We could certainly add the prefixes if they are not present. We will use this issue to address it. Thanks for reporting this.