Which of these methods to limit how fast things get piped?
See original GitHub issueIn the example below, I’m downloading a gzipped file that’s several hundred megabytes (millions of rows), then piping it through highland so I can use the batch
feature. This way, instead of inserting 1 row into the db at a time (csvStream emits one ‘data’ event for one row), I can do it for 10000 rows at a time in the ‘data’ event handler you see below.
download.stream(url)
.pipe(zlib.createGunzip())
.pipe(csvStream)
.pipe(highland())
.batch(10000)
.on('data', async (data) => {
await insertIntoDB(data)
})
But when running this I get out of memory errors and the system starts to slow down significantly. I think it’s because the data is coming in too fast to the ‘data’ event. The csvStream’s finish
event happens in a couple of minutes but the program runs for up to another hour, which indicates that the whole csv file has been read into memory, rather than being piped downstream piece by piece as the data
event consumes the batches.
I’m new to highland and looking through the documentation I can’t tell which of the various methods would be the most appropriate in this case. http://highlandjs.org/#backpressure seems like it’s most relevant to this situation but I can’t tell how to use it in this code. http://highlandjs.org/#parallel looks good too.
Can I configure highland so that at any time there’s only for example 3 batches (where batch
is 10000) worth of rows that have been read? And it only reads another 10000 rows when one of those 3 batches is complete.
Issue Analytics
- State:
- Created 7 years ago
- Comments:25 (1 by maintainers)
I’m not super familiar with the async functions spec, but I think the code you’ve posted will just execute as many insertIntoDB as necessary to get through your input data, all in parallel. The fact that you
await
inside your async function doesn’t actually cause it to block thedata
call. You’d get the same behavior if you passedinsertIntoDB
toon('data')
directly.You’re right that backpressure is what you want though. In Highland, you get it for free as long as you don’t opt-out. Using
on('data')
is opting-out. You shouldn’t really need to use theon
method ever, really. It’s why we don’t document the events on the website.To do what you want, you need to use
parallel
ormergeWithLimit
. They’re more or less the same with the exception thatparallel
preserves order andmergeWithLimit
doesn’t. The way you use them is by constructing a stream of streams. What’s usually done is that you usemap
to perform some asynchronous action, and return a stream that waits for that action to complete and then emits the result. The stream here functions much like a promise. It doesn’t contain any data; it’s just a handle for later getting at your data. Then you callparalel
ormergeWithLimit
, which will handle the backpressure for you.For example,
The reason this works is the laziness and backpressure features of Highland. Nothing happens until you call
done
, which is a consumption operator.done
will ask for data from themergeWithLimit
stage.mergeWithLimit
will ask for streams from themap
stage, stopping once it has gotten 3 of them. Once it has finished consuming one of the three, it will ask for more. This is backpressure.map
only executes whenmergeWithLimit
asks for data. This is laziness. SincemergeWithLimit
will only ask for 3 at a time and won’t ask for more until a previous stream (i.e., task) has completed, you will only have 3 tasks in-flight at any one time.map
of course exerts backpressure on the previous stages, which keeps memory usage from blowing up.download
should return a promise that resolves once the download completes. Both of your options will work. Use the one that you like better.