Wrong number of workers reported by effective_n_jobs

I want to highlight something which looks weird to me (not sure yet if this is a bug). I made a snippet which is a pattern that we use in scikit-learn.

from joblib import Parallel, delayed, effective_n_jobs, parallel_backend
from joblib.parallel import get_active_backend


def func(x):
    return x ** 2


class Klass:
    def __init__(self, n_jobs=None):
        self.n_jobs = n_jobs

    def func(self, x):
        n_jobs = effective_n_jobs(self.n_jobs)
        print(
            f"Number of effective jobs found: {effective_n_jobs(self.n_jobs)}"
        )
        backend, backend_n_jobs = get_active_backend()
        print(
            f"Current backend used: {backend}"
        )
        print(
            f"Number of jobs in the backend: {backend_n_jobs}"
        )
        return Parallel(n_jobs=n_jobs)(delayed(func)(i) for i in x)


n_jobs = 4
with parallel_backend("threading", n_jobs=n_jobs):
    r = Klass().func(range(10))

Number of effective jobs found: 1
Current backend used: <joblib._parallel_backends.ThreadingBackend object at 0x7f946468d810>
Number of jobs in the backend: 4

If I am not wrong, one would expect to make the Parallel call with n_jobs=4. However, effective_n_jobs will report n_jobs=1 due to:

https://github.com/joblib/joblib/blob/1753c3bcb1d8430439ce48a1ef4e866166f34296/joblib/_parallel_backends.py#L232-L235

I would have expected parallel_backend to overwrite the number of jobs if not specified in the estimator but I am not sure about it.

NB: found during investigating https://github.com/scikit-learn/scikit-learn/issues/15978

[glemaitre]

Let’s try to implement @glemaitre in a PR

Not me but the solution 😃

Shall I do it?

I can do it with a unit test.

[GaelVaroquaux]

Let's try to implement @glemaitre in a PR

Not me but the solution 😃

I would so much like to have @glemaitre in joblib. It would be a major feature!