How to include extended features (e.g. state variables like temperature or pressure) in a restricted search space: "Fixed parameter ... with out of design value"

The model requires certain features in order to obtain a good fit, but these parameters are not fixed, nor are they necessarily something that makes sense to be optimized for (especially in the existing data #743).

An example would be Vickers hardness measurements scraped from scientific literature. The Vickers hardness depends on the applied load of the Vickers tip, which varies from publication to publication (in this case, from experience - removing the applied load results in almost no predictive performance). However, if provided to the algorithm as a search parameter, it will naturally suggest minimizing the applied load, because Vickers hardness is (almost?) always higher for smaller loads. This same thing appears in many materials science & chemistry optimization problems, including several of the projects I’m working on.

Phrased another way, the model benefits from additional features (appended, not replacing), but only optimizes over the original features. The FixedParameter class comes to mind, but I don’t think the intention was for a FixedParameter to vary for every datapoint. The discussion in https://github.com/facebook/Ax/issues/773 related to sequential search spaces also comes to mind, but after attempting to use this, I get:

Fixed parameter size with out of design value: 5.0 passed to `RemoveFixed`.
  File "[C:\Users\sterg\miniconda3\envs\axforchemistry\Lib\site-packages\ax\modelbridge\transforms\remove_fixed.py]()", line 51, in transform_observation_features
    raise ValueError(
  File "[C:\Users\sterg\miniconda3\envs\axforchemistry\Lib\site-packages\ax\modelbridge\base.py]()", line 217, in _transform_data
    obs_feats = t_instance.transform_observation_features(obs_feats)
  File "[C:\Users\sterg\miniconda3\envs\axforchemistry\Lib\site-packages\ax\modelbridge\base.py]()", line 170, in __init__
    obs_feats, obs_data, search_space = self._transform_data(
  File "[C:\Users\sterg\miniconda3\envs\axforchemistry\Lib\site-packages\ax\modelbridge\torch.py]()", line 99, in __init__
    super().__init__(
  File "[C:\Users\sterg\miniconda3\envs\axforchemistry\Lib\site-packages\ax\modelbridge\registry.py]()", line 342, in __call__
    model_bridge = bridge_class(
  File "[C:\Users\sterg\Documents\GitHub\sparks-baird\AxForChemistry\axforchemistry\utils\experiment.py]()", line 61, in generate_sobol_experiments
    model = BayesModel(
  File "[C:\Users\sterg\Documents\GitHub\sparks-baird\AxForChemistry\tutorials\multiple_constraints.py]()", line 183, in <module>
    y_preds, y_stds, next_experiments, trial_indices, model = generate_sobol_experiments(
  File "[C:\Users\sterg\miniconda3\envs\axforchemistry\Lib\runpy.py]()", line 87, in _run_code
    exec(code, run_globals)
  File "[C:\Users\sterg\miniconda3\envs\axforchemistry\Lib\runpy.py]()", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "[C:\Users\sterg\miniconda3\envs\axforchemistry\Lib\runpy.py]()", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "[C:\Users\sterg\miniconda3\envs\axforchemistry\Lib\runpy.py]()", line 87, in _run_code
    exec(code, run_globals)
  File "[C:\Users\sterg\miniconda3\envs\axforchemistry\Lib\runpy.py]()", line 197, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,

(call stack)

transform_observation_features (c:\Users\sterg\miniconda3\envs\axforchemistry\Lib\site-packages\ax\modelbridge\transforms\remove_fixed.py:51)
_transform_data (c:\Users\sterg\miniconda3\envs\axforchemistry\Lib\site-packages\ax\modelbridge\base.py:217)
__init__ (c:\Users\sterg\miniconda3\envs\axforchemistry\Lib\site-packages\ax\modelbridge\base.py:170)
__init__ (c:\Users\sterg\miniconda3\envs\axforchemistry\Lib\site-packages\ax\modelbridge\torch.py:99)
__call__ (c:\Users\sterg\miniconda3\envs\axforchemistry\Lib\site-packages\ax\modelbridge\registry.py:342)
generate_sobol_experiments (c:\Users\sterg\Documents\GitHub\sparks-baird\AxForChemistry\axforchemistry\utils\experiment.py:61)
<module> (c:\Users\sterg\Documents\GitHub\sparks-baird\AxForChemistry\tutorials\multiple_constraints.py:183)
_run_code (c:\Users\sterg\miniconda3\envs\axforchemistry\Lib\runpy.py:87)
_run_module_code (c:\Users\sterg\miniconda3\envs\axforchemistry\Lib\runpy.py:97)
run_path (c:\Users\sterg\miniconda3\envs\axforchemistry\Lib\runpy.py:268)
_run_code (c:\Users\sterg\miniconda3\envs\axforchemistry\Lib\runpy.py:87)
_run_module_as_main (Current frame) (c:\Users\sterg\miniconda3\envs\axforchemistry\Lib\runpy.py:197)

So I guess the idea of using FixedParameter doesn’t pan out because of the use of RemoveFixed (https://ax.dev/docs/models.html#transforms). Maybe a way to get around this is to set narrow bounds in a new search space for a RangeParameter or set a ChoiceParameter with only one choice (not sure if the latter would throw an error).

[saitcakmak]

Hi @adam-bhaiji. Your setup reminds me of what we refer to as “environmental variables” in robust optimization. These are the variables that the decision maker does not control (e.g., the wind speed) but they have an effect on the metrics. In robust optimization, we try to find designs that are robust to these variables, given their distribution. This setup makes most sense when the final value of the environmental variable isn’t observed until after the decision is made, which is not exactly your setup. You seem to be making the final decision once you observe the wind speed. Anyway, if the robust optimization setup is of interest to you, I have an internal implementation of this that I’d be happy to clean up.

I don’t know how much of this is exposed in Ax, but @qingfeng10 has done a bunch of work on contextual BO that is implemented in BoTorch. Flagging that since it might be useful here.

I haven’t settled on an idea for how to deal with a context that is only known after starting the experiment.

Let’s call x_d the decision variables and x_c the context variables that are observed after starting the experiment. Since you can’t control x_c, you can’t optimize it while generating candidates. But since you get it along with the metric observations, you can include it in the surrogate model. So, your model training data would be (x_d, x_c) and the corresponding metrics. This is similar to the setup used in robust BO with environmental variables, though there we typically assume x_c can be controlled in the simulator (which is not strictly necessary). The crucial point here is that you need to be using an acquisition function that knows of this setup, so a bit of customization may be required (the robust BO implementation I referred to above may be helpful).

I think the 2nd one is new

Pretty sure I implemented them together 😃. The implementation I have exposes that formulation in Ax modular BoTorch model setup, building on the RobustSearchSpace and all the relevant bits that were introduced in past few months.

[sgbaird]

So, I think the tire type would fit into the first category I mentioned, whereas the wind speed is something that fits into the second category. AFAIK, the first category can be dealt with by using the workaround I mentioned in https://github.com/facebook/Ax/issues/905#issuecomment-1096097249. The second category probably requires something more custom, but maybe @Balandat or someone else can comment here. I haven’t settled on an idea for how to deal with a context that is only known after starting the experiment. I wonder if this would involve making a list of predefined candidates and treating the stochastic wind speed as a feature in a feature vector https://github.com/facebook/Ax/issues/771, and add that information before filtering and suggesting your next candidate. Maybe there’s a way to do this without the somewhat hacky/non-robust workaround of using a predefined list. I’d also be worried about using predefined candidates if your task is high-dimensional. I don’t see anything blocking this from a theoretical standpoint - your model has a variable for wind speed, the wind speeds in the past are fixed and known, and on the “day of the race”, you can measure the wind speed and make decisions about the best car design based on that information.

With your task, is it the case that you could effectively “design a new car on the day of the race” based on the wind conditions for that day? Or is it just that you could choose a particular set of tires to put on, but everything else about the car design is fixed? (asking since you mentioned this is a toy problem and to make sure I understand the non-toy problem nuances).

I wonder also if it would make sense to try use a risk-averse BO method. Maybe one of these two would be useful:

• https://botorch.org/tutorials/risk_averse_bo_with_input_perturbations
• https://botorch.org/tutorials/risk_averse_bo_with_environmental_variables

I think the 2nd one is new (just came across it) and it seems very relevant to the toy problem you described.