Pause Kafka Consumer with Functional Style

Dear Developers,

Please allow me to apologize of posting question here. I know this isn’t the right place however given that I’ve lost 100 of my stackoverflow reputation for bounty for this very same question https://stackoverflow.com/questions/66123464/pause-kafka-consumer-with-spring-cloud-stream-and-functional-style, I believe that I earned some credit for posting such question here.

I’m trying to build a retry mechanism which pauses the consumer on the retry topic based on the timestamp and retry count from the incoming message. This requires me to get the header information that includes the consumer, topic and partition information.

With classic style I can do the following.

@StreamListener(Sink.INPUT)
public void in(String in, @Header(KafkaHeaders.CONSUMER) Consumer<?, ?> consumer) {
    System.out.println(in);
    consumer.pause(Collections.singleton(new TopicPartition("myTopic", 0)));
}

However I cannot find any example or documentation showing the functional style. I have tried the following but it doesn’t work.

@Bean
public Function<Message<?>, KStream<String, String>> process() {
    message -> {
        Consumer<?, ?> consumer = message.getHeaders().get(KafkaHeaders.Consumer, Consumer.class);
        String topic = message.getHeaders().get(KafkaHeaders.Topic, String.class);
        Integer partitionId = message.getHeaders().get(KafkaHeaders.RECEIVED_PARTITION_ID, Integer.class);
        CustomPayload payload = (CustomPayload) message.getPayload();
        if (payload.getRetryTime() < System.currentTimeMillis()) {
            consumer.pause(Collections.singleton(new TopicPartition(topic, partitionId)));
        }
    }
}

I got such exception.

Caused by: java.lang.IllegalStateException: No factory found for binding target type: org.springframework.messaging.Message among registered factories: channelFactory,messageSourceFactory,kStreamBoundElementFactory,kTableBoundElementFactory,globalKTableBoundElementFactory
    at org.springframework.cloud.stream.binding.AbstractBindableProxyFactory.getBindingTargetFactory(AbstractBindableProxyFactory.java:82)
    at org.springframework.cloud.stream.binder.kafka.streams.function.KafkaStreamsBindableProxyFactory.bindInput(KafkaStreamsBindableProxyFactory.java:191)
    at org.springframework.cloud.stream.binder.kafka.streams.function.KafkaStreamsBindableProxyFactory.afterPropertiesSet(KafkaStreamsBindableProxyFactory.java:111)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1853)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1790)
    ... 96 more

It would be great if someone can point me at the right direction of solving this problem. I really liked the functional style and I don’t want to mix the code with both functional and classic binding (which also has many annotations marked as deprecation).

Much appreciated!

[AttitudeL]

This solution works like a charm! @sobychacko thank you again for all the help! I’m closing this ticket.

[sobychacko]

Using the StreamBridge API is now preferred to BinderAwareChannelResolver. See the reference docs for that. Basically, when you resume the consumer, you can use StreamBridge to send the record to an outbound destination. If you were using a Function instead of a Consumer, then the outbound sending is implicitly handled by the framework.