Stream behavior with errors

If the call to the download function below fails with an error, stopOnError will cause the Promise to reject. Since this is doing parallel(10), will this code wait for the other 9 download operations to finish, or will it terminate them right away when it encounters the first error?

How will the behavior be different if I use mergeWithLimit(10) instead of parallel(10)?

  await new Promise((res, rej) => highland(range)
    .map(page => highland(download(page)))
    .parallel(10)
    .stopOnError(rej)
    .done(res));

[vqvu]

Is it fine to have errors and stopOnError in the same stream?

Yes.

Or as you said, done will throw if it handles an unhandled error, so maybe stopOnError(rej) isn’t necessary? But then how would I reject the Promise?

You can call rej inside errors instead of pushing the error along. Like this

    .errors((err, push) => {
      if (err instanceof MyError) {
        log(err)
      } else {
        rej(err) // Reject the promise.
        push(null, highland.nil) // stopOnError.
      }
    })

It’s important to call rej(err) first, before push nil. Pushing nil first may cause the stream to end and done(res) to execute before you get a chance to reject the promise, causing you to end up with a resolved promise instead of a rejected one. This is a quirk of the current implementation.

That said, I personally think it’s too easy to accidentally transpose the two lines, and it’s not immediately obvious that order matters, so I wouldn’t recommend doing this. Your solution using stopOnError(rej) is less error-prone.

is it fine to have an errors function without any push(null, myData)? In my example after doing log(err) would I need to do any kind of push? (The errors would already be removed from the stream and logged so I wouldn’t want to push any values for them back onto the stream.)

It’s perfectly fine to not push anything. That’s one of the primary uses of errors—to handle certain errors and filter them out from the stream.

[vqvu]

parallel will hold back errors the same way that it holds back stream values due to its ordering guarantee. mergeWithLimit will emit the error as soon as it gets it. stopOnError will stop the stream from consuming new data (and thus stop new operations from being started), but it does not cancel download operations that have already started.

Let’s say you have 15 download operations to process, your parallelism factor is 10, and the 3rd download fails.

With parallel

parallel will emit the error in order, so when your promise is rejected,

• The first two download operations are guaranteed to have completed.
• The 3rd download will have failed.
• Up to the next 10 downloads (downloads 4-13) will have been started. “Up to” because the actual number depends on how fast the other downloads complete. These downloads will not be stopped, and you know nothing about them. In particular, any number of these downloads may have already completed.

With mergeWithLimit

mergeWithLimit will emit the error as soon as possible, so when your promise is rejected,

• The 3rd download will have failed.
• You know nothing about the other downloads. Depending on how fast the 3rd download executed, the actual number of started and completed downloads can be anything from 0 to 14.
• The number of started but not yet completed downloads are guaranteed to be less than or equal to 10.

Using merge doesn’t tell you much about the download operations on errors. That’s the price you pay for the better memory usage and more parallelism. That said, if you want to know which downloads completed before the error, you can resolve the Promise that download returns with some id (say the page number) when the download completes. These values will be emitted before the error if the download completed before the error.

Note that you do not have to use stopOnError. There is also errors. It’s just there as an example, because you want to handle errors somehow (done will throw if it encounters an unhandled error). But it’s up to you how you want to handle them.