trial.suggest_int uses RandomSampler instead of CmaEsSampler

All my parameters are using trial.suggest_int. Tried optuna.samplers.CmaEsSampler but performance really bad. Many parameters used RandomSampler and objective score, R2 was very low and seemed static, R2 = 0.30 Tried optuna.integration.CmaEsSampler and performance was slightly better, R2 = 0.58 - 0.60 but also many parameters was sampled using RandomSampler. The same study using TPESampler gives R2 = 0.749, no problems.

Some representative codes:

def objective (trial): params_list =[] n_params = trial.suggest_int(‘n_params’, 3, 5) param1 = trial.suggest_int(‘param1’, 0, 1200) param2 = trial.suggest_int(‘param2’, 0, 1199) params_list.append(param1) params_list.append(param2) if n_params >=3: param3 = trial.suggest_int(‘param3’, 0, 1198) params_list.append(param3) if n_params >=4: param4 = trial.suggest_int(‘param4’, 0, 1197) params_list.append(param4) if n_params >=5: param5 = trial.suggest_int(‘param5’, 0, 1196) params_list.append(param5) model = model(params_list) score = model.evaluate(x_test, y_test) # R2 score return score

study = optuna.create_study(direction=‘maximize’, sampler=optuna.samplers.CmaEsSampler) study.optimize(objective, n_trials=10000, n_jobs=-1)

I have noticed that RandomSampler is always used on params 3, 4, 5 i.e. those that were in the conditional if statements.

I expect performance to be comparable to TPESampler and sampling is done using the CmaEsSampler.

Environment

• Optuna version: 2.0
• Python version: 3.7
• OS: win10 64-bit
• (Optional) Other libraries and their versions:

Error messages, stack traces, or logs

# error messages, stack traces, or logs
Example error messages:
[W 2020-07-29 20:58:44,714] The parameter 'param3' in trial#9993 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-29 20:58:44,714] The parameter 'param4' in trial#9994 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.
[W 2020-07-29 20:58:44,714] The parameter 'param5' in trial#9996 is sampled independently by using `RandomSampler` instead of `CmaEsSampler` (optimization performance may be degraded). You can suppress this warning by setting `warn_independent_sampling` to `False` in the constructor of `CmaEsSampler`, if this independent sampling is intended behavior.

[koh-joshua]

Thank you. I am trying PyCmaSampler…it is slow…very slow…but seems to be working…will update later. I wanted to compare a Bayesian based sampler (TPE) to evolution based sampler (CMA-ES, PSO or GA). I think your cma-es implementation is the best, because it is fast. Fingers crossed the bug will be flushed out. Thanks!

[koh-joshua]

Thank you for your super quick response and explanation. You guys are totally amazing! Perhaps I can shout the Optuna team some beer and sake in the future when I drop by Tokyo. Thank!