Hierarchical search spaces

Is there a way to define a hierarchy of parameters? for example a parameter that chooses architecture, and each architecture has its own parameters.

example (pseudo code):

architecture = choise(["NeuralNetwork","xgdboost"])

if architecture=="NeuralNetwork":
     n_layers = choise(range(1,10,1))
     #more architecture releted params here.

else if  architecture=="xgdboost":
    max_depth =  choise(range(1,5,1))
     #more architecture releted params here.

[lena-kashtelyan]

@yonatanMedan, @LyzhinIvan, @Tandon-A, @riyadparvez, BayesOpt mode is supported in alpha-mode now and currently works through search space flattening (so the Gaussian Process model is not aware of the hierarchical structure of the search space under the hood). cc @dme65 to say more about when BayesOpt over flattened search spaces is effective

If you try it, please let us know how it goes for you (ideally in this issue)! Updated version of my example above that should le you run BayesOpt:

from ax.service.ax_client import AxClient, ObjectiveProperties
from ax.service.utils.report_utils import exp_to_df
from ax.utils.measurement.synthetic_functions import branin

ax_client = AxClient()
ax_client.create_experiment(
    parameters=[
        {
            "name": "model",
            "type": "choice",
            "values": ["Linear", "XGBoost"],
            "dependents": {
                "Linear": ["learning_rate", "l2_reg_weight"],
                "XGBoost": ["num_boost_rounds"],
            },
        },
        {
            "name": "learning_rate",
            "type": "range",
            "bounds": [0.001, 0.1],
            "log_scale": True,
        },
        {
            "name": "l2_reg_weight",
            "type": "range",
            "bounds": [0.00001, 0.001],
        },
        {
            "name": "num_boost_rounds",
            "type": "range",
            "bounds": [0, 15],
        },
    ],
    objectives={"objective": ObjectiveProperties(minimize=True)},
    # To force "Sobol" if BayesOpt does not work well (please post a repro into 
    # a GitHub issue to let us know, it will be great help in debugging this faster!)
    # choose_generation_strategy_kwargs={"no_bayesian_optimization": True},
)

[riyadparvez]

Yes, this would be a great addition! I have a similar usecase - after hyperparameter optimization choose the right threshold for classification.