Add support for parallel processing of batches / bulk sets of messages

Currently the interface only provides the user a way to process single messages in parallel. There are use cases however where the user may want to processes batches of messages in parallel as well.

For example, say the API they’re calling out to actually supports batches of requests, for example Elastic Search batch API that can take several documents at once.

The API might look like:

void pollBatch(int batchLevel, Consumer<List<ConsumerRecord<K, V>>> usersVoidConsumptionFunction);

So the user could request batches of 5 records, which could then be transformed into a single HTTP request. These batches of 5 messages would all be run in parallel, up to the usual concurrency limits. The same could be added to the vertx module.

Further from this, options could be added to specify requirements for the batch - for example, give me 5 messages in a batch that all have the same key, are all in the same partition etc.

Feature comes out of the backlog, but was also prompted immediately from initial user feedback upon announcement: https://twitter.com/derekm00r3/status/1323375584701984768

[astubbs]

FYI ok, there’s a couple test failures still - i think they need updating. But the batching feature seems feature complete now (lol). Need to implement some test for the vertx and reactor versions, but should have this merged early next week.

[astubbs]

@raphaelauv yes I think it should be. I’m currently working on another project atm, but hope to get back to PC 1.0 release soon.