current status of n_jobs in v2.10?

Question

I find a merge Jan 29 using concurrent.futures https://github.com/optuna/optuna/pull/2269, then I find the njobs parameter was deprecated here https://github.com/optuna/optuna/pull/2560 on Apr 1. And now there is discussion about reintroducing it here https://github.com/optuna/optuna/issues/2937 on Sept 18. I also found an older work-arounds, using joblib https://gist.github.com/sile/2c329d0cb38f35e25a8c7bab44edbb63.

I am wondering what the current recommended procedure for multi-core environments using v2.10.0 optuna. What does the implementation currently do with n_jobs specified? It seems to produce odd, non-montonic best values (see example below).

Environment

>>> print(op.__version__) 
2.10.0
>>> print(platform.python_version()) 
3.8.5
>>> print(platform.platform()) 
macOS-10.15.7-x86_64-i386-64bit

Error messages, stack traces, or logs

Trial 101 finished with best value: 3.051867903393622 and parameters: {'R0': 1.0712727856991768, 'alpha': 0.9121462925283631, 'gamma': 1.1823702204733835, 'mu': 0.036417631166185516}. 
Trial 188 finished with best value: 2.2365535937364998 and parameters: {'R0': 1.0694600337354254, 'alpha': 0.8999322001243446, 'gamma': 1.3864523757139084, 'mu': 0.02695870506276768}. 
Trial 277 finished with best value: 2.088390682306482 and parameters: {'R0': 1.054260021970593, 'alpha': 1.5693093881559335, 'gamma': 1.469040063506123, 'mu': 0.03048413180433311}. 
Trial 276 finished with best value: 15.34937870021244 and parameters: {'R0': 1.1377582001694242, 'alpha': 0.6932468664055964, 'gamma': 1.236927077386912, 'mu': 0.04018656386490023}. 
Trial 292 finished with best value: 1.3179174767632489 and parameters: {'R0': 1.0631375246172654, 'alpha': 1.5265556047812037, 'gamma': 1.3094584471794986, 'mu': 0.025065419768277475}. 
Trial 293 finished with best value: 1.6933340465788846 and parameters: {'R0': 1.0665697529499913, 'alpha': 1.5104281490808449, 'gamma': 1.4077384827255122, 'mu': 0.02765503785586052}. 

Steps to reproduce

N/A

Reproducible examples

N/A

[himkt]

Thank you @rbelew for the explanation. I reproduced the results by the following code (@hvy wrote it):

import optuna
import random
import time


# Turn off optuna log notes.
optuna.logging.set_verbosity(optuna.logging.WARN)


def objective(trial):
    x = trial.suggest_float("x", 0, 1)
    return x ** 2


def logging_callback(study, frozen_trial):
    previous_best_value = study.user_attrs.get("previous_best_value", None)

    if previous_best_value != study.best_value:
        study.set_user_attr("previous_best_value", study.best_value)

        print(
            "Trial {} finished with best value: {} and parameters: {}. ".format(
            frozen_trial.number,
            frozen_trial.value,
            frozen_trial.params,
            )
        )


study = optuna.create_study()
study.optimize(objective, n_trials=100, callbacks=[logging_callback], n_jobs=10)

And I found this code doesn’t work as expected in multi-{thread,process} situations.

• In multi-thread situation, some threads may update a ~system~ user attribute simultaneously (race-condition)
• In multi-process situation, I’m not sure but I think each process doesn’t share the study (which causes the discrepancy and odd behavior)
  • https://gist.github.com/sile/2c329d0cb38f35e25a8c7bab44edbb63#file-multi_process_storage-py-L59-L62

In summary, reporting best value occurance is difficult with multi-{thread,process} optimization. We have to update our documentation for letting users know this limitation.

@hvy Please add a comment if I misunderstand something or you have supplement for this topic.

[github-actions[bot]]

This issue was closed automatically because it had not seen any recent activity. If you want to discuss it, you can reopen it freely.