Library specific global options are not passed to jobs

Library such as pandas and scikit-learn have global config options. These options are not passed to jobs spawned by joblib (with a multiprocess backend):

from joblib import Parallel, delayed
from sklearn import get_config, config_context

def get_working_memory():
    return get_config()['working_memory']

with config_context(working_memory=123):
    results = Parallel(n_jobs=2)(
        delayed(get_working_memory)() for _ in range(2)
    )

results
# [1024, 1024]

Related: https://github.com/joblib/joblib/issues/1064 Scikit-learn specific fix: https://github.com/scikit-learn/scikit-learn/pull/17634

[thomasjpfan]

I think it would not be possible for joblib to tell “where the configuration are stored and should be passed into the spawn processes” unless the library somehow registers the how to set and get the configuration from the library.

For scikit-learn it could be something like:

import joblib

joblib.register_global_config(getter=sklearn.get_config,
                              setter=sklearn.set_config)

[lesteve]

I would think the culprit is very likely cloupickle if I had to do an extra wild-guess (feeling a bit adventurous this morning 😉)