HTTP requests best practice

I am making a number of HTTP requests using the Request module. The responses from these requests need to be unzipped and written to a file in the order the URLs were in.

So far I have the following:

function download(urls, path) {
  return new Promise((resolve, reject) => {
    const tmpPath = path + '-tmp';
    const unzip = zlib.createGunzip();
    const output = fs.createWriteStream(tmpPath);

    _(urls)
      .map(request)
      .map(_)
      .parallel(5)
      .pipe(unzip)
      .on('error', (err) => { return reject(err); })
      .pipe(output)
      .on('finish', () => {
        fs.rename(tmpPath, path, (err) => {
          if (err) { return reject(err); }
          return resolve(path);
        });
      })
      .on('error', (err) => { return reject(err); });
  });
}

This works however I have a couple of questions about the implementation.

1. Is parallel the best option here? I guess there is a trade off between holding more data in memory or calling the URLs in sequence using flatMap. I am after speed so downloading in parallel and using a bit more memory is ok for me.
2. Is there any way to capture events on the request. If I was using request on its own I would have something like:

request(url)
  .on('response', (response) => {
    if (response.statusCode === 404) {
      return reject(...);
    }
  })
 .on('error', (err) => { return reject(err); });

Is there a way of listening for these events when using Highland?

Any advice greatly received.

[moose56]

Thanks for your help @vqvu, here are all the bits combined in case it helps anyone else:

function download (urls, path) {
  const tmpPath = path + '-tmp';
  const unzip = zlib.createGunzip();
  const output = fs.createWriteStream(tmpPath);

  const getUrl = url => {
    return _((push, next) => {
      request(url)
        .on('response', resp => {
          if (resp.statusCode === 200) {
            push(null, _(resp));
            push(null, _.nil);
          } else {
            push(new Error('Something went wrong'));
            push(null, _.nil);
          }
        });
    });
  };

  let error;

  return new Promise((resolve, reject) => {
    _(urls)
      .flatMap(getUrl)
      .parallel(5)
      .through(unzip)
      .stopOnError(err => error = err)
      .pipe(output)
      .on('finish', () => {
        console.log('finish');
        if (error) return reject(error);

        fs.rename(tmpPath, path, (err) => {
          if (err) { return reject(err); }
          return resolve(path);
        });
      })
      .on('error', reject);
  });
}

[vqvu]

Is parallel the best option here? I guess there is a trade off between holding more data in memory or calling the URLs in sequence using flatMap. I am after speed so downloading in parallel and using a bit more memory is ok for me.

Yep. That’s the trade off. If you’re after download speed, then parallel is the right thing to do here. Tune the parallelism factor as necessary.

Is there a way of listening for these events when using Highland?

Yes, but it’s a bit more complicated. By default, doing _(request(url)) will already listen for the error event and propagate it correctly. But if you want to listen to multiple custom events, you’ll need to write your own stream generator.

Something like this. Note that I’m not calling reject until fairly late in the pipeline, and I’m using through instead of pipe. It’s generally more convenient to keep everything as a Highland stream for as long as possible, since Highland does nice things like error-propagation for you.

In this case, it’s also important to call push(error) instead of reject(error) in getUrl, since otherwise, you will have rejected the promise but not stop the stream.

function getUrl(url) {
  return _((push, next) => {
    request(url)
        .on('response', (response) => {
          if (response.statusCode === 200) {
            push(null, _(response));
            push(null, _.nil);
          } else {
            push(new Error('...'));
            push(null, _.nil);
          }
        })
        .on('error', (err) => {
          push(err);
          push(null, _.nil);
        });
  });
}

_(urls)
    .map(getUrl)
    .parallel(5)
    .through(unzip)  // Using through instead of pipe to keep things a Highland stream.
    .stopOnError(reject)  // Stop the steam so that no new URLs are fetched.
    .pipe(output)
    .on('finish', ...)
    .on('error', reject);