Proposal: metaprogramming with babel macros

I think in the long term we’re going to have to do something about the amount of code duplication in iter-tools. The bulk of it stems from the fact that most of our implementations have almost identical copies for the async forms.

I do not think that we can or should want to try to consolidate the sync vs async methods to share code at runtime. I can expand more on my reasons for that, but perf is high among them, given the complexity that would be involved. We could however share code between implementations at compile time, and I think we should.

Babel macros are pretty cool, and a relatively recent feature of babel: they allow you to import a transformation into a source as a symbol. The macro is then passed the AST node that is called on, and given an option to inspect and transform it. Here’s what it might look like:

import a from "./macros/async";
a(function someMethod() {
  a(for(const item of iterable) {
    ...
  })
})

Which, when transpiled by babel, would be split into two outputs:

function someMethod() {
  for(const item of iterable) {
    ...
  }
}

and

async function someMethod() {
  for await (const item of iterable) {
    ...
  }
}

Fortunately in order to keep ourselves sane we already make sure that our sync and async implementations looks as similar to each other as we can make them. But this method of working gives us a lot of flexibility as well. For one it is opt-in not opt-out, so we can use it only where it really makes sense. It would even allow us to insert slightly different custom code into each method if we needed to:

import isAsync from "./macros/is-async";

if (isAsync) {
  // Only the contents of one block or the other would be present in output
  // ...
} else {
  // ...
}

Some more thoughts:

• Babel macros are supported on astexplorer.net, making it relatively easy to test them out and figure out what the correct transformation code to apply is.
• Because the macros have resolvable imports it will be far easier for new contributors to the project to discover the relevant transformation code and understand how the final code is generated.
• Eventually (though it would not be necessary immediately) we could develop macros that would help us manage our tests for sync and async version of a function. This could help considerably. As low hanging fruit, the macros could easily allow us to generate real inputs which were sync for sync tests and async for async tests, instead of only testing the code paths where async methods are called on synchronous iterables. Consolidating the tests to the extent that they validate basic method behavior for both sync and async impls would also give us more energy to write tests for what may be genuinely different between the two versions.

[sithmel]

I can see we have some degree of repetition, but I am not sure it is worth to add such a complex element. I am afraid this will be raising the (already high) bar, for new contributors.

[conartist6]

I am closing this as this technique is now deployed in the codebase. Work is outstanding still on the typescript types, but that will be tracked in #237.