Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Study parallel jobs should use multiprocessing

See original GitHub issue

When you do parallel execution in study.optimize (when n_jobs=-1 for example). You say with Parallel(n_jobs=n_jobs, prefer="threads") as parallel:. See here:

https://github.com/optuna/optuna/blob/61c6a0acb22338789a83a02bb147326159f41d1e/optuna/study.py#L317

IMO this just opens (threading) which suffers from the Python Global Interpreter Lock. See here: https://joblib.readthedocs.io/en/latest/generated/joblib.Parallel.html

This would not lead to real parallel execution and is useless with CPU bound problems. I suggest to use this: with Parallel(n_jobs=n_jobs, backend="multiprocessing") as parallel:

If you want to I can provide a PR.

Issue Analytics

State:
Created 3 years ago
Reactions:16
Comments:7 (3 by maintainers)

Top GitHub Comments

1reaction

profPlumcommented, Feb 23, 2022

@sile I see thanks

1reaction

silecommented, Feb 23, 2022

FYI. Optuna contributors are currently developing the v3 version, and there are storage/multi-process related topics (e.g., Consider storage that can be easily used with multi-process). Thus, your feedback or opinion will be welcomed I think.