Conditional CategoricalDistribution

Dear all,

I am a big fan of Optuna. Thank you for this great system!

However, during my first tries, I run into the following problem:

import optuna
from optuna.samplers.random import RandomSampler

def objective(trial):

    dataset = trial.suggest_categorical('dataset', ['sparse_data', 'dense_data'])

    classifiers = ['classifier that can handle sparse and dense']
    if dataset == 'dense_data':
        classifiers.append('classifier that can only handle dense')

    classifier = trial.suggest_categorical('dataset', classifiers)

    return 0

study = optuna.create_study(direction='maximize', sampler=RandomSampler(seed=42))
study.optimize(objective, n_trials=4)

Of course, it is well-reported that this will lead to an exception because a dynamic value space for categories is not supported:

Traceback (most recent call last):
  File "/home/felix/FastFeatures/new_project/fastsklearnfeature/test/optuna/dynamic_categories.py", line 18, in <module>
    study.optimize(objective, n_trials=4)
  File "/home/felix/anaconda3/envs/new_project/lib/python3.7/site-packages/optuna/study.py", line 339, in optimize
    func, n_trials, timeout, catch, callbacks, gc_after_trial, None
  File "/home/felix/anaconda3/envs/new_project/lib/python3.7/site-packages/optuna/study.py", line 682, in _optimize_sequential
    self._run_trial_and_callbacks(func, catch, callbacks, gc_after_trial)
  File "/home/felix/anaconda3/envs/new_project/lib/python3.7/site-packages/optuna/study.py", line 713, in _run_trial_and_callbacks
    trial = self._run_trial(func, catch, gc_after_trial)
  File "/home/felix/anaconda3/envs/new_project/lib/python3.7/site-packages/optuna/study.py", line 734, in _run_trial
    result = func(trial)
  File "/home/felix/FastFeatures/new_project/fastsklearnfeature/test/optuna/dynamic_categories.py", line 12, in objective
    classifier = trial.suggest_categorical('dataset', classifiers)
  File "/home/felix/anaconda3/envs/new_project/lib/python3.7/site-packages/optuna/trial/_trial.py", line 468, in suggest_categorical
    return self._suggest(name, CategoricalDistribution(choices=choices))
  File "/home/felix/anaconda3/envs/new_project/lib/python3.7/site-packages/optuna/trial/_trial.py", line 652, in _suggest
    return self._set_new_param_or_get_existing(name, param_value, distribution)
  File "/home/felix/anaconda3/envs/new_project/lib/python3.7/site-packages/optuna/trial/_trial.py", line 659, in _set_new_param_or_get_existing
    self._trial_id, name, param_value_in_internal_repr, distribution
  File "/home/felix/anaconda3/envs/new_project/lib/python3.7/site-packages/optuna/storages/in_memory.py", line 257, in set_trial_param
    self._studies[study_id].param_distribution[param_name], distribution
  File "/home/felix/anaconda3/envs/new_project/lib/python3.7/site-packages/optuna/distributions.py", line 512, in check_distribution_compatibility
    CategoricalDistribution.__name__ + " does not support dynamic value space."
ValueError: CategoricalDistribution does not support dynamic value space.

Now, my question: Is there any way around it to make it work or any hack to accomplish the same?

Best regards, Felix

[FelixNeutatz]

Dear @toshihikoyanase,

thank you, for your answer. Yes, this simple trick should work 😃

Best regards, Felix

[Lawrence-Krukrubo]

Was having the very same problem. Thanks for the solution @toshihikoyanase it’s helped me too.