asyncBuffer docs are misleading

I made a mistake when I got rid of asyncMapParallel and other parallel functions. I thought I could instead have parallelism be created with asyncBuffer. asyncBuffer is still useful and has a correct implementation and example, however the docs suggest (as I was thinking) that the method can create parallelism, which it cannot. Making multiple requests to an async generator simply queues them.

Here is an example of code that I just wrote in macrome where I want parallelism and I believe I have implemented it correctly:

const initialPaths = await traverse(this.projectRoot, { ignored });

const operations = await execPipe(
  initialPaths,
  filter((path) => !!this.generatorInstances.find(({ generator }) => matches(path, generator))),
  map((path) =>
    this.parserFor(path)
      .hasHeader(path)
      .then((h) => (h ? { path, operation: ADD } : null)),
  ),
  asyncBuffer(4),
  asyncFilter((op) => op !== null),
  arrayFromAsync,
);

this.initialProcess(operations);

[coder-mike]

Tangent here, but what are you using the library for? I’m always curious since I don’t get feedback any other way.

I’ve used twice, in completely different projects.

The first one was an alerting engine.

• Given a sequence of events from another server, determine whether there’s an issue that needs to be raised as an alert (e.g. email someone).
• Needed to define various guarantees, such as “if the server goes down in the middle of processing an event, it should continue where it left off”. This was implemented using a persistent bookmark/cursor that recorded the last successfully processed event. Storing the bookmark is itself an asynchronous process.
• For performance reasons, it would have been crazy to process one event before requesting the next (the data source had hundreds of events per second), so we needed to have a processing pipeline that could process hundreds of events in parallel if necessary.

The second use case (which I’m working on now) is polling a legacy SQL database for new records in a particular table and then acting on them in an asynchronous manner.

• The records are being fetched in batches using an async generator which performs a round-trip to the database.
• This is followed by an asyncBuffer so that we’re requesting the next batch while processing the previous (we don’t need batches to be requested in parallel, but just need the next batch to be “ready for us” when we’re finished processing the previous).
• For each record, we’re contacting a web server, which is a round-trip that we don’t want to wait for so this requires parallelism in the pipeline
• And again, we need to keep track of the “progress” through the records so that even if the server dies in the middle, it’s still guaranteed to process each record at least once. So there is a round-trip database update at the end of each item

[conartist6]

Invariant 3 is already pretty much a guarantee in the iterator spec. Any iterators implemented with generator functions will force effectively serial calls to next(). Generators do this by buffering the calls to next(). That is one thing that makes it difficult to come up with inventive solutions. I really do not want to break with generator semantics.