test on optuna btb and smac

test description

find the minimum value of (x1-1)**2 + (x2-2)**2

x1 is a float in [-10 ,10] x2 is a float in [-10 ,10]

test 100 studies and each study has 100 trials

test code

optuna

import optuna
import time
import numpy as np
optuna.logging.set_verbosity(optuna.logging.ERROR)
start = time.time()
def objective(trial):
    x1 = trial.suggest_uniform('x1', -10, 10)
    x2 = trial.suggest_uniform('x2', -10, 10)
    return  (x1 - 1) ** 2 + (x2 - 2) ** 2

best_score=[]
for i in range(100):
    study = optuna.create_study()
    study.optimize(objective, n_trials=100)
    best_score.append(study.best_value)

end = time.time()
print(np.mean(best_score))
print(end-start)

time 647s average of 100 results 0.27

btb

from btb import HyperParameter, ParamTypes
from btb.tuning import GP
import time
import numpy as np
import warnings
warnings.filterwarnings('ignore')

best_score = []

start = time.time()

for i in range(100):
    tunables = [
        ('x1', HyperParameter(ParamTypes.FLOAT, 
                                        [-10, 10])),
        ('x2', HyperParameter(ParamTypes.FLOAT, 
                                        [-10, 10]))
    ]
    tuner = GP(tunables)

    for i in range(100):
        params = tuner.propose()
        #  print('params',params)
        score = -(params['x1'] - 1) ** 2 -  (params['x2'] - 2) ** 2
        tuner.add(params, score)
    best_score.append(tuner._best_score)
    
end = time.time()
print(np.mean(best_score))
print(end-start)

time 264s average of 100 results 0.002

[nomuramasahir0]

Regarding the difference between the results of Optuna and btb, as mentioned above, I think that it is due to the difference of surrogate models (Optuna uses TPE and btb uses Gaussian Process). Eggensperger et al. confirmed that Gaussian process based Bayesian optimization performs better than TPE and SMAC for low-dimensional functions [Eggensperger 13].

And for the results of SMAC, it looks like the initial value of SMAC is given a very good value (the initial value of SMAC is (0, 0) that is near the optimal value (1, 2)). If you experiment with the different setting, the result may change.

[Eggensperger 13] Eggensperger, K., Feurer, M., Hutter, F., Bergstra, J., Snoek, J., Hoos, H., and Leyton-Brown, K.: Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters, in NeurIPS workshop on Bayesian Optimization in Theory and Practice (2013).

[luxu1220]

I am a newcomer to this aspect. Thank you for your patient. You are right, I realized that the time is indeed not a problem. I also made a test on SMAC, this is just one study. You can see that it’s much slower than optuna and btb. 10 trials cost about 55s. But the result is pretty good.

After all these tests, I found that optuna is much easier for beginners to use. And it also supports saving/resuming studies. I hope optuna can support more algorithms. Thank you.