Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add support for parallel processing of batches / bulk sets of messages

See original GitHub issue

Currently the interface only provides the user a way to process single messages in parallel. There are use cases however where the user may want to processes batches of messages in parallel as well.

For example, say the API they’re calling out to actually supports batches of requests, for example Elastic Search batch API that can take several documents at once.

The API might look like:

void pollBatch(int batchLevel, Consumer<List<ConsumerRecord<K, V>>> usersVoidConsumptionFunction);

So the user could request batches of 5 records, which could then be transformed into a single HTTP request. These batches of 5 messages would all be run in parallel, up to the usual concurrency limits. The same could be added to the vertx module.

Further from this, options could be added to specify requirements for the batch - for example, give me 5 messages in a batch that all have the same key, are all in the same partition etc.

Feature comes out of the backlog, but was also prompted immediately from initial user feedback upon announcement:

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:2
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

astubbscommented, Feb 25, 2022

FYI ok, there’s a couple test failures still - i think they need updating. But the batching feature seems feature complete now (lol). Need to implement some test for the vertx and reactor versions, but should have this merged early next week.

astubbscommented, Nov 22, 2021

@raphaelauv yes I think it should be. I’m currently working on another project atm, but hope to get back to PC 1.0 release soon.

Read more comments on GitHub >

github_iconTop Results From Across the Web

The Salesforce Bulk API -
Processing batches serially means running them one after another, and processing batches in parallel means running multiple batches at the same time.
Read more >
Spring Batch Parallel Processing and Scaling 101 - Hevo Data
Spring Batch Parallel Processing is each chunk in its own thread by adding a task executor to the step. If there are a...
Read more >
Scaling and Parallel Processing - Spring
The simplest way to start parallel processing is to add a TaskExecutor to your Step configuration. ... In this example, the taskExecutor is...
Read more >
Use DMS batch apply feature to improve CDC replication ...
I'm running a full load and a change data capture (CDC) AWS Database Migration Service (AWS DMS) task. The source latency isn't high, ......
Read more >
How to use batching to improve Azure SQL Database and ...
The second set of tests ran from a cloud service and database that both ... processing multiple batches in parallel might be more...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found