Inconsistent joblib.Parallel behaviour between v0.12 and v0.14 when running from within an imported module?

I have a joblib.Parallel statement which is run from within an imported module. From v0.12 to v0.12.5 it works nicely, but with v0.13+ it goes serial with no errors.

The schema of my code (which spans several files) follows. Input arguments are omitted for the sake of readability.

# main file
import f1
for _ in range(n):
    f1()

# file containing f1
import f2
def f1():
    # do something with f2
    return something

# file containing f2
def f2():
    def help():
        return something
    joblib.Parallel(delayed(help))    # parallel with loki backend
    return something

I tried to find some information online regarding joblib.Parallel and its use on nested functions, with no luck.

However, if I change my workflow as follows

# main file
import f1
joblib.Parallel(delayed(f1))     # parallel with loki backend

# file containing f1
import f2
def f1():
    # do something with f2
    return something

# file containing f2
def f2():
    def help():
        return something
    for _ in range():
        # run help() in serial with a plain for loop
    return something

everything starts working also with v0.14.

What am I missing? And why with joblib v0.12-0.12.5 it works fine but with joblib v0.13+ it goes serial?

[Alessi0X]

@ogrisel

Going serial when it should not is what I called being a bug.

Sorry, since you were mentioning errors or tracebacks I thought you meant bugs as errors/crashes.

I will try my best to provide a minimal working example.

[Alessi0X]

Hi @pierreglaser I have updated my code snippet with info on joblib’s backends.

I will try to craft a minimal working example, but this ain’t going to be easy. Unfortunately, things are quite nested here. I’ll try to work it out anyway