Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add batching support for consumer applications

See original GitHub issue

As a user, I’d like to have an option to control the batch-size at the consumer level, so I can buffer until a certain limit and consume the messages in bulk.

Note: spring-kafka and spring-integration-kafka supports a BatchMessageListener and the binder can adapt to it to expose an overridable property similar to producer.

Acceptance:

I can enable batching via a property at consumer (i.e., listenerMode)
I can use the listenerMode property to override the consumer behavior with tow options: record and batch (If record, each Message will contain byte[] from a single data record. If batch, each Message will contain a List<byte[]>)
Document the feature with an example
Tests included

Issue Analytics

State:
Created 7 years ago
Reactions:14
Comments:6 (2 by maintainers)

Top GitHub Comments

5reactions

sabbyanandancommented, May 14, 2019

Hi, everyone! My apologies for the super delayed response. Given the number of votes and the many comments on this issue, we are very much interested in adding batching support for the consumers.

For Kafka in particular, the core semantics and the infrastructure to enable this is already built out in Spring Kafka and Spring Integration Kafka projects - we will build on top of this and line them up nicely as an easy to override the property in Kafka binder. We will have to think about what this means for the Kafka Streams binder, though.

As an FYI, @garyrussell recently added similar support in Spring AMQP, which will then bubble up to Spring Integration AMQP, and likewise, we will provide an option to enable this for the Rabbit binder, too.

As another FYI, @artembilan has had this support in the Kinesis binder since the first milestone. See listenerMode docs here.

With all that said, we are sure to build this out and add support for both Kafka and Rabbit binders as part of the Spring Cloud stream v2.3 / Horsham release-train. This train will be dependent on Spring Boot’s 2.2 GA timeline.

In the meantime, if you have any other thoughts/comments regarding this feature, feel free to add as comments here.

Thank you for your patience and continued interest.

0reactions

olegzcommented, Jan 6, 2020

@vinothkarthik Please do not comment on closed issues especially with questions. For questions we use Stack Overflow channel.