Foldby not combining in parallel.

Hey guys.

Been using dask for ad-hoc big data analysis creating custom reports while processing a few terabytes of json data.

I’ve noticed that the foldby operation has a huge sequential task that is computed after it has applied the binop to each partition. In theory, if a combine function is supplied couldn’t this be done in parallel by taking the output of each partition and combining them in a pairwise fashion?

Does this make sense or did I miss something?

Sample code:

(
    bag
    .filter(predicate)
    .map(js.loads)
    .map(transform)
    .foldby(key, binop, initial, combine, initial)
    .compute()
)

Cheers!

[mrocklin]

There are a few other tree reduction implementations around in the codebase. You might want to search for split_every in the codebase.

[mrocklin]

I’m not sure why dictitems is not applied in the second case. Do any of the tests fail with its addition? That might be a useful way to see an unintended change.

My bad, I’ve been using open-source software for about a little over a year and this is my first time contributing. Still a n00b.

Different groups operate differently, but most dask devs congregate on github rather than gitter.