Crash when passing named function as an argument

Hello,

I am using Python 3.6.5 with joblib 0.11.1.dev0 (master).

Consider the two files:

# test_imp.py
def foo_imp(i):
    return i+2

# test.py
from joblib import Parallel, delayed

def foo(i):
    return i+1
def wrap(i, func):
    return func(i)

# runs fine
from test_imp import foo_imp
res = Parallel(n_jobs=2)([delayed(wrap)(i, foo_imp) for i in range(10)])
print(res)

# crashes
res = Parallel(n_jobs=2)([delayed(wrap)(i, foo) for i in range(10)])
print(res)

When passing an imported function foo_imp as an argument, the code runs fine. For the locally defined function foo however, I get the following error:

Traceback (most recent call last):
  File "/python/site-packages/joblib/externals/loky/process_executor.py", line 376, in _process_worker
    call_item = call_queue.get(block=True, timeout=timeout)
  File "/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'foo' on <module 'joblib.externals.loky.backend.popen_loky_posix' from '/python/site-packages/joblib/externals/loky/backend/popen_loky_posix.py'>

Is this intended behavior? It might be related to #600.

[tomMoral]

I have pushed a fix for this issue in master. It should be faster too when issuing many short tasks.

[tomMoral]

You could introspect the object using a function like

def is_in_main(obj):
    mod = getattr(obj, "__module__", "")
    return "__main__" in mod

Be careful that you need to go a bit further to ensure that the unpickling is done properly. You could for instance wrap the objects from main in a class such as

class WrapMainModuleObject:
    def __init__(self, obj):
        self.pickled_obj = dumps(obj)
    def __reduce__(self):
        return loads, (self.pickled_obj,)

Edit: Sorry I ended-up creating the pull request. Feel free to test it and report if you see bad behaviors.