not using all CPU

Hi there,

I’m running a ML tuning with optuna on an instance with 48 cores and has set n_jobs=-1:

study.optimize(OptunaObjective(clf=self.clf,
                                       X=X,
                                       y=y,
                                       sample_weight=sample_weight,
                                       scoring=self.scoring,
                                       cv=self.cv,
                                       decay_f=decay_f),
                       self.n_trials, n_jobs=-1)

I noticed that cpu utilization is around 20% and ram is around 4%.

I’m wondering if i set it up correctly or is there is another way to increase performance. Regarding the “Distributed Optimization tutorial”, I do not know if i can set it up an sql database as i’m using instances on my university cluster.

If you need more code, let me know. Thank you for any help.

Best regards

[lschneidpro]

I could try.

In the mean time, I started renting a Postgre SQL server online (30 first days free)

[lschneidpro]

Thanks again for spoting such a mistake. It seems that the author of your example is running 10**2 trials then

Yes, it’s a typical way to run optimization in parallel. If you don’t want to use shell, you can spawn multiple optimization processes within Python script by using a process management library such as joblib (please see quadratic_joblib_simple.py for an example).