BUG: parallel crashes when array too big.

I am not sure whether this is a bug or a bad parametrization on my part, but I get an error when passing a unique large arrays to Parallel with max_nbytes=None, and n_jobs > 1.

from joblib import Parallel, delayed
import numpy as np

def func(X):
    pass

X = np.random.rand(841, 306, 1445)
chunks = map(list, zip(*[np.array_split(X, 1)]))  # note n_jobs > n_splits

parallel = Parallel(-1, max_nbytes=None)
p_func = delayed(func)

out = parallel(p_func(chunk) for chunk in chunks)

However, it works is max_nbytes isn’t provided, or when the array is smaller.

This issue is replicable on ubuntu 14.04 + anaconda build with joblib 0.9.4 and numpy 1.11.0

It may be related to https://github.com/joblib/joblib/issues/138

[lesteve]

So this is bleeding edge from pickle? Shall we add an explicit error when more than 2**31 bytes are sent through the socket?

It’s not pickle but multiprocessing. You can have a look at https://bugs.python.org/issue17560 for more details. I think this quote extracted from it pretty much sums it up:

But I can’t help feeling that multigigabyte arrays should be transferred using shared mmaps rather than serialization.

[kingjr]

ok thanks @lesteve & @ogrisel