parallelisation is not working in optuna

How can we execute the code in parallel in optuna. The documentation is so naive about that. Suppose I have a test.py file which includes the following code:

 study = optuna.create_study(
        direction="maximize",
        study_name="ali", #for parallel execution you need these couple of lines 
        storage="sqlite:///mydb.db", 
        load_if_exists= True, #to skip creating a new study if it already exists. 
        pruner = SuccessiveHalvingPruner(),
        #pruner=HyperbandPruner(min_resource=1, max_resource=100, reduction_factor=3),
    )

and I have another test2.py file which calls the test.py file like that:

import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor
import os
from multiprocessing import Pool

def run_process (process):
    os.system('python {}'.format(process))

if __name__ == "__main__":
    processes = ['TestDriver.py']
    pool = Pool()
    pool.map(run_process, processes)
    pool.close    

But the code:

1. is not executed in parallel. I cannot see for example in the console: Trial 0 finsihed … Trial 1…
2. The code creates two experiments in neptune.

Note to the questioner

If you are more comfortable with Stack Overflow, you may consider posting your question with the “optuna” tag there instead. Alternatively, for issues that would benefit from more of an interactive session with the developers, you may refer to the optuna/optuna chat on Gitter.

[koh-joshua]

I think what ali3assi may be asking for is the good old n_jobs=-1: study.optimize(objective, n_trials=n_trials, n_jobs=-1)

This will utilize all the cores on your computer for the single study. However, depending on the complexity of the objective and number of trials, you will typically not see all cores utilized or 100% CPU usage. Also, since Optuna > 2.0, the n_jobs=-1 don’t really seem to be efficient in using multiple cores. Up until version 2.0, I can set n_jobs=-1 and launch multiple instances of Optuna and see my CPU usage hit 100% for 32 cores sustained. After 2.0, it seems that each study is locked to a process or there’s some limitations and this prevents my CPU usage scaling to 100%. The speed penalty is extreme.

For now, version 2.0 is gold to me.

What about distributed parallelism in Optuna using RDB? That is ok if your trial run is up to 100,000 or less, split or distributed between multiple instances of Optuna for the same study. It gets extremely slow after that. But due to the random seed initialization in the optimizers, you’d be better running multiple instances of Optuna to get a better tuned model or objective. At least in my own experience, I recover the best result from 5 instances/5 studies of 20,00 trials each versus one instance/study at 100,000 trials.

Hope this helps.

[ali3assi]

@HideakiImamura Thanks for your reply. I am on windows. But I need to parallelize the code on a pc of 48 cpus, I need to open 48 terminales?