Using joblib from Matlab

I’m trying to use functions MNE-Python and scikit-learn from MATLAB. It works well, except for some issues with the standard streams (especially on windows) and with joblib: https://github.com/mne-tools/mne-python/pull/8515#issuecomment-726917486

As a minimal example, I used the one from the joblib documentation: joblib.Parallel(n_jobs=2)(joblib.delayed(math.sqrt)(i**2) for i in range(10))

This works well from a Python 3.8 console on Windows 10.2004. It also works with MATLAB 2020b / WSL-Ubuntu18 / Python 3.8 / joblib 0.17.0.

But it crashes when calling it from MATLAB 2020b / Windows 10.2004 / same Python 3.8 environment / joblib 0.17.0. Here are the Matlab console outputs for 1 job (always working) and 2 jobs (crashing with backends loky and multiprocessing)

>> py.exec('import joblib, math; print(joblib.Parallel(n_jobs=1, backend="loky")(joblib.delayed(math.sqrt)(i**2) for i in range(10)))', py.dict())
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

>> py.exec('import joblib, math; print(joblib.Parallel(n_jobs=2, backend="loky")(joblib.delayed(math.sqrt)(i**2) for i in range(10)))', py.dict())
Error using resource_tracker>_send (line 209)
Python Error: OSError: [Errno 22] Invalid argument
Error in resource_tracker>_check_alive (line 182)
Error in resource_tracker>ensure_running (line 102)
Error in resource_tracker>register (line 190)
Error in _memmapping_reducer>register_folder_finalizer (line 590)
Error in _memmapping_reducer>register_new_context (line 560)
Error in _memmapping_reducer>set_current_context (line 535)
Error in _memmapping_reducer>__init__ (line 531)
Error in executor>get_memmapping_executor (line 42)
Error in executor>get_memmapping_executor (line 20)
Error in _parallel_backends>configure (line 494)
Error in parallel>_initialize_backend (line 740)
Error in parallel>__call__ (line 973)
Error in <string>><module> (line 1)
 
>> py.exec('import joblib, math; print(joblib.Parallel(n_jobs=2, backend="multiprocessing")(joblib.delayed(math.sqrt)(i**2) for i in range(10)))', py.dict())
Error using resource_tracker>_send (line 209)
Python Error: OSError: [Errno 22] Invalid argument
Error in resource_tracker>_check_alive (line 182)
Error in resource_tracker>ensure_running (line 102)
Error in resource_tracker>register (line 190)
Error in _memmapping_reducer>register_folder_finalizer (line 590)
Error in _memmapping_reducer>register_new_context (line 560)
Error in _memmapping_reducer>set_current_context (line 535)
Error in _memmapping_reducer>__init__ (line 531)
Error in pool>__init__ (line 303)
Error in _parallel_backends>configure (line 470)
Error in parallel>_initialize_backend (line 740)
Error in parallel>__call__ (line 973)
Error in <string>><module> (line 1)
 
>> py.exec('import joblib, math; print(joblib.Parallel(n_jobs=2, backend="sequential")(joblib.delayed(math.sqrt)(i**2) for i in range(10)))', py.dict())
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

>> py.exec('import joblib, math; print(joblib.Parallel(n_jobs=2, backend="threading")(joblib.delayed(math.sqrt)(i**2) for i in range(10)))', py.dict())
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

This is most likely due to limitations in the MATLAB-Python integration on Windows. Would you have any suggestion to get around these limitations?

[ftadel]

I don’t understand how it could work before on Windows then…

It has probably never worked 😃

>> py.exec("from concurrent.futures import ProcessPoolExecutor; ProcessPoolExecutor(max_workers=2).submit(print, 'ok').result()", py.dict())
Error using _base>__get_result (line 388)
Python Error: BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Error in _base>result (line 439)

Error in <string>><module> (line 1)

This triggers a licensing error: I do have the parallel processing toolbox installed, but forking the Matlab process might never be allowed…

>> py.exec("from multiprocessing import Pool; Pool(4).apply(print, ('ok',))", py.dict())

Same, but much worse, because it restarts the job when they fail, which makes Matlab difficult to kill from the Windows process manager…

[ftadel]

It is up to the user to set up a Python environment on the system, Matlab doesn’t ship (yet?) its own Python. The user has to point at a python.exe in Matlab with the commands pyenv (or previously pyversion): https://www.mathworks.com/help/matlab/ref/pyenv.html

I don’t know if there is any introspective function in Python that would let you know at runtime what the real python executable is beyond the matlab.exe. If this is an info you need and can’t find it anywhere on the Python side, I could try to ask the Mathworks developers.