Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Find records in batches

See original GitHub issue

Issue type:

[ ] bug report [x] feature request [ ] documentation issue

Constantly, we have to load records in batches, to process them without consuming a lot of memory. Having some Rails background, I am a big fond of find_in_batches, which receives a callback which will be called with each batch of records. I’d like to have something like this:

repository.findInBatches({ batchSize: 100, concurrency: 3, where: { ... } }, (batch) => { 
  // process batch ...
})

The function is responsible for loading the records, limited by batchSize and order by primary key, to make the query fast. The concurrency options would control the number of batches which can be processed concurrently.

I want to know opinions about it. In case the community is positive about it, I can open a PR with the feature.

Issue Analytics

State:
Created 3 years ago
Reactions:14
Comments:5 (2 by maintainers)

Top GitHub Comments

4reactions

imnotjamescommented, Oct 2, 2020

Would returning an async iterator be a better path for more use case?

2reactions

zivarichacommented, Oct 25, 2020

+1 for that, i also come from ruby background and this is one of the first things i found missing in typeorm, would be really helpful to add it

Top Results From Across the Web

ActiveRecord::Batches - Rails API

The find_each method uses find_in_batches with a batch size of 1000 (or as ... Yields each batch of records that was found by...

find_in_batches (ActiveRecord::Batches) - APIdock

Yields each batch of records that was found by the find options as an array. Person.where("age > 21").find_in_batches do |group| sleep(50) # Make...

Rails — find_each v.s find_in_batches v.s in_batches - Medium

Rails provides find_each , find_in_batches , and in_batches these three public methods to work with the records in batches, which helps reduce memory ......

Run a rails query in batches - Stack Overflow

You need to use find_each with the option batch_size. A.where(:name => "John").find_each(batch_size: 1000) do |a| # your code end.

Method: ActiveRecord::Batches#in_batches - RubyDoc.info

You can make worker 1 handle all the records between id 1 and 9999 and worker 2 handle from 10000 and beyond by...