Can joblib.Parallel work in interactive environments (e.g., IPython Notebook) in Windows?

The doc example

from joblib import Parallel, delayed
Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))

worked with importable functions like sqrt, but not with those defined in interactive environments including IPython Notebook and IPython console:

def myfunc(x):
    return sqrt(x)
Parallel(n_jobs=2)(delayed(myfunc)(i ** 2) for i in range(10))

and raised the following error:

AttributeError: 'module' object has no attribute 'myfunc'

I’m running Python 2.7.7, Anaconda 2.0.1 (64-bit), Windows 7 SP1 (64-bit), IPython 2.1.0, joblib 0.8.4. The same works smoothly in Linux.

[ogrisel]

[lesteve]

AFAIR this is not expected to work under Windows neither with Python 2 nor Python 3.

@GaelVaroquaux 2.7. It can be solved by putting the method into module and then import them. But that really complicated the code sometimes…

Putting the method in a module is the cleanest work-around. If that helps, you have a built-in editor in Jupyter these days. If you insist of having all your code in the same notebook you can use a combination of %%file to write into an external file and imp.reload to make sure you always use the latest version of the module.

For completeness, the long-term fix for this is to use dill to serialize the function inside Parallel. The idea would be to just take #240 and only use dill to serialise the function inside Parallel. This is something we really want to do so that we can switch to fork_server as the default joblib start method. We just haven’t had the time to do it … PR more than welcome !