Functions taking multiple iterables should take an iterable of iterables

In this ticket I proposed that functions taking multiple iterables should take instead an iterable of iterables (or an iterable of async iterables). They are:

• concat
• compress
• zipAll
• zip
• merge/collate

This is a follow up from this ticket: https://github.com/sithmel/iter-tools/issues/185

Here’s why I am proposing this:

• consistency with the other functions and curry that works the same way
• interoperability with functions that return an iterable of iterables (fork, partition etc.)

I am registering here different opinions and my answesr: @conartist6

I can see the theoretical elegance of changing the functions which take multiple iterables to take an iterable of iterables, but it seems a bit like sacrificing the common use case for the benefit of the uncommon one.

My answer: uncommon is subjective. In https://github.com/sithmel/iter-tools/pull/167#issuecomment-464402212 I wrote: The api should be consistent within itself, to ease function composition. You have functions returning a iterable of iterables (fork, partition, split etc.) and therefore you need corresponding functions that take an iterable of iterables as argument.

@conartist6 in this comment https://github.com/sithmel/iter-tools/issues/185#issuecomment-460717480 wrote:

I still think that this idea parallelism in data processing is best expressed as code which is written in columns is not robust, and I am particularly unhappy about the idea of making sacrifices from our most basic use cases, which I imagine to have at least 20 times more potential audience than code-in-columns data pipelines. It is because I believe that this library is currently only reaching a tiny fraction of the programmers who it should interest that I have invested time and effort in it. There is at present a bit of a conflict between my interest in serving that yet-unrealized constituency and your interest in refining your existing use cases […]

What you call “code in column” is function composition. I can argue that is a well spread pattern (in scheme and clojure for example) and I don’t see why you say that is not robust. Also there are plenty of APIs that take an iterable (Promise.all, Map, Set for example) and I don’t see why this should scare away users.

Alternatively, you could make a helper function based on your currying helper function, which would turn a method from varargs to iterable-of-iterables style. Usage might look like collectParams(zip). I cannot think of any reason a correct implementation would not be possible.

I can consider (as a last resort) creating alternate version of these functions taking an iterable instead of multiple arguments.

@conartist6 again on https://github.com/sithmel/iter-tools/pull/174#issuecomment-466792130

I think TC39 has spoken on variadic functions. By offering call spread and arguments spread they embraced the pattern, and it makes sense too. Most of the time users of these variadic functions will have separate expressions to pass to each argument. In the less common case that they have an iterable, all they have to do is write .... In the even less common case that we are in a pipeline and working exclusively with higher order functions, we now have apply.

I would not take the presence of the spread operator as an endorsement. It is just a more practical of using apply

The variadic arguments arrangement has many desirable properties for the common case. For one, no extra syntax is needed. Another: it is the most optimized, as it passes the variables on the stack instead of having to create an Array.

No extra syntax: not clear what you mean. “most optimized” I won’t be surprised if the array is created in the background. But we don’t really need to argue on this. We have a benchmark suite, we can just measure it.

Finally Typescript endorses the pattern. Typescript has an awful time with tuples, and has a tendency to lose their types unless two things happen: 1) the tuple-typed array is defined in exactly the location that it is consumed by a type which supports tuple-typed arrays. and 2) that API has an overload for tuples and an overload for Iterable, since there is no concept of iterables which return a different type at each iteration. This means that in order for the common case type information to not be lost, the user has to make a call of the structure: variadicFn(arg, [iterableA, iterableB]). The [] cannot go anywhere else, and thus is just dead weight; we may as well have allowed the user to call the function in a variadic manner.

I don’t see where the problem is in TS. It should be possible to define an iterator returning iterators. And also I am not sure it is necessary to overspecify the content of the iterables returned by the iterable. I don’t see this as worst than the inconsistency on the curry.

[conartist6]

OK. I’m going to try to get the currying PR merge-ready.

[conartist6]

By the way that example is also an excellent validation of our earlier decision not to treat objects as being coercible to iterables.