`KeyError` when using multiple contraints

I’m running an experiment with a number of linear constraints. However, when I add more than two, I start getting some weird KeyErrors.

If I move the contraints inside my function, and just returns a large loss whenever they are not satisfied, things seem to work. But I feel like this may be a bug?

$ python text_emb.py --fast_dev_run 100
/fsx/users/ahle/conda/lib/python3.9/site-packages/torchvision/io/image.py:11: UserWarning: Failed to load image Python extension: libtorch_cuda_cu.so: cannot open shared object file:
No such file or directory
  warn(f"Failed to load image Python extension: {e}")
[INFO 02-24 05:50:48] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in th
e logs are rounded to 6 decimal points.
[INFO 02-24 05:50:48] ax.service.utils.instantiation: Inferred value type of ParameterType.INT for parameter window_size. If that is not the expected value type, you can explicity spe
cify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 02-24 05:50:48] ax.service.utils.instantiation: Inferred value type of ParameterType.INT for parameter embedding_dim_log. If that is not the expected value type, you can explici
ty specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 02-24 05:50:48] ax.service.utils.instantiation: Inferred value type of ParameterType.INT for parameter hidden_size_log. If that is not the expected value type, you can explicity
 specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 02-24 05:50:48] ax.service.utils.instantiation: Inferred value type of ParameterType.INT for parameter splits_log. If that is not the expected value type, you can explicity spec
ify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 02-24 05:50:48] ax.service.utils.instantiation: Inferred value type of ParameterType.INT for parameter table_size_log. If that is not the expected value type, you can explicity
specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 02-24 05:50:48] ax.service.utils.instantiation: Inferred value type of ParameterType.INT for parameter lstm_layers. If that is not the expected value type, you can explicity spe
cify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 02-24 05:50:48] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter lstm_dropout. If that is not the expected value type, you can explicity
specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 02-24 05:50:48] ax.service.utils.instantiation: Inferred value type of ParameterType.STRING for parameter pool. If that is not the expected value type, you can explicity specify
 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/core/parameter.py:467: UserWarning:

`is_ordered` is not specified for `ChoiceParameter` "pool". Defaulting to `False` for parameters of `ParameterType` STRING. To override this behavior (or avoid this warning), specify
`is_ordered` during `ChoiceParameter` construction.

/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/core/parameter.py:467: UserWarning:

`sort_values` is not specified for `ChoiceParameter` "pool". Defaulting to `False` for parameters of `ParameterType` STRING. To override this behavior (or avoid this warning), specify
 `sort_values` during `ChoiceParameter` construction.

[INFO 02-24 05:50:48] ax.service.utils.instantiation: Inferred value type of ParameterType.BOOL for parameter norm. If that is not the expected value type, you can explicity specify $
value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/core/parameter.py:467: UserWarning:

`is_ordered` is not specified for `ChoiceParameter` "norm". Defaulting to `True` for parameters of `ParameterType` BOOL. To override this behavior (or avoid this warning), specify `is
_ordered` during `ChoiceParameter` construction.

/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/core/parameter.py:467: UserWarning:

`sort_values` is not specified for `ChoiceParameter` "norm". Defaulting to `True` for parameters of `ParameterType` BOOL. To override this behavior (or avoid this warning), specify `s
ort_values` during `ChoiceParameter` construction.

[INFO 02-24 05:50:48] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='window_size', parameter_type=INT, range=[2, 30]), RangeParamet
er(name='embedding_dim_log', parameter_type=INT, range=[5, 10]), RangeParameter(name='hidden_size_log', parameter_type=INT, range=[1, 13]), RangeParameter(name='splits_log', parameter
_type=INT, range=[0, 5]), RangeParameter(name='table_size_log', parameter_type=INT, range=[3, 20]), RangeParameter(name='lstm_layers', parameter_type=INT, range=[1, 5]), RangeParamete
r(name='lstm_dropout', parameter_type=FLOAT, range=[0.0, 1.0]), ChoiceParameter(name='pool', parameter_type=STRING, values=['mean', 'mult', 'max', 'tensor-sketch', 'concat', 'att'], i
s_ordered=False, sort_values=False), ChoiceParameter(name='norm', parameter_type=BOOL, values=[False, True], is_ordered=True, sort_values=True)], parameter_constraints=[OrderConstrain
t(embedding_dim_log <= hidden_size_log), OrderConstraint(splits_log <= table_size_log), ParameterConstraint(1.0*hidden_size_log + -1.0*table_size_log <= -1.0)]).
[INFO 02-24 05:50:48] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parame
ters.
[INFO 02-24 05:50:48] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 18 trials, GPEI for subseq
uent trials]). Iterations after 18 will take longer to generate due to  model-fitting.
[WARNING 02-24 05:50:48] ax.modelbridge.transforms.int_to_float: Unable to round {'window_size': 8, 'embedding_dim_log': 11, 'hidden_size_log': 6, 'splits_log': 0, 'table_size_log': 9
, 'lstm_layers': 5, 'lstm_dropout': 0.8974349498748779, 'pool_OH_PARAM__0': 0.09609586745500565, 'pool_OH_PARAM__1': 0.05041775479912758, 'pool_OH_PARAM__2': 0.5942715406417847, 'pool
_OH_PARAM__3': 0.042279504239559174, 'pool_OH_PARAM__4': 0.8558655381202698, 'pool_OH_PARAM__5': 0.06802913546562195, 'norm': -1}to meet constraints of SearchSpace(parameters=[RangePa
rameter(name='window_size', parameter_type=INT, range=[2, 30]), RangeParameter(name='embedding_dim_log', parameter_type=INT, range=[5, 10]), RangeParameter(name='hidden_size_log', par
ameter_type=INT, range=[1, 13]), RangeParameter(name='splits_log', parameter_type=INT, range=[0, 5]), RangeParameter(name='table_size_log', parameter_type=INT, range=[3, 20]), RangePa
rameter(name='lstm_layers', parameter_type=INT, range=[1, 5]), RangeParameter(name='lstm_dropout', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='pool_OH_PARAM__0', par
ameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='pool_OH_PARAM__1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='pool_OH_PARAM__2', parameter_type=FLOAT, ra
nge=[0.0, 1.0]), RangeParameter(name='pool_OH_PARAM__3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='pool_OH_PARAM__4', parameter_type=FLOAT, range=[0.0, 1.0]), Rang
eParameter(name='pool_OH_PARAM__5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='norm', parameter_type=INT, range=[0, 1])], parameter_constraints=[OrderConstraint(emb
edding_dim_log <= hidden_size_log), OrderConstraint(splits_log <= table_size_log), ParameterConstraint(1.0*hidden_size_log + -1.0*table_size_log <= -1.0)])
Traceback (most recent call last):
  File "/data/home/ahle/kaggle2/text_emb.py", line 464, in <module>
    main()
  File "/data/home/ahle/kaggle2/text_emb.py", line 443, in main
    parameters, trial_index = ax_client.get_next_trial()
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/utils/common/executils.py", line 147, in actual_wrapper
    return func(*args, **kwargs)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/service/ax_client.py", line 355, in get_next_trial
    generator_run=self._gen_new_generator_run(), ttl_seconds=ttl_seconds
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/service/ax_client.py", line 1344, in _gen_new_generator_run
    return not_none(self.generation_strategy).gen(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/modelbridge/generation_strategy.py", line 330, in gen
    return self._gen_multiple(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/modelbridge/generation_strategy.py", line 471, in _gen_multiple
    generator_run = _gen_from_generation_step(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/modelbridge/generation_strategy.py", line 833, in _gen_from_generation_step
    generator_run = generation_step.gen(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/modelbridge/generation_node.py", line 111, in gen
    return model_spec.gen(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/modelbridge/model_spec.py", line 170, in gen
    return fitted_model.gen(**model_gen_kwargs)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/modelbridge/base.py", line 674, in gen
    observation_features, weights, best_obsf, gen_metadata = self._gen(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/modelbridge/random.py", line 86, in _gen
    X, w = self.model.gen(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/models/random/sobol.py", line 114, in gen
    points, weights = super().gen(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/models/random/base.py", line 121, in gen
    points, attempted_draws = rejection_sample(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/models/model_utils.py", line 84, in rejection_sample
    point = rounding_func(point)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/modelbridge/modelbridge_utils.py", line 417, in _roundtrip_transform
    observation_features = t.untransform_observation_features(
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/ax/modelbridge/transforms/choice_encode.py", line 112, in untransform_observation_features
    obsf.parameters[p_name] = reverse_transform[pval]
KeyError: -1

[Balandat]

The inferred types are all correct. Would it be better practice if I stated them explicitly?

Probably better practice to be explicit, but if they are inferred correctly that shouldn’t be an issue.

I updated the code a bit since yesterday, but I just got the error again with the following setup:

It’s pretty hard to debug this issue without a fully reproducible example. If you can’t share the full code/setup, maybe you can provide some of the data under which you see these issues? Or do the occur if you just feed back some synthetic data to the AxClient?

[lena-kashtelyan]

This issue hasn’t been updated in two weeks, so closing. @thomasahle, please reopen it if/when you have a follow up! If you post a more detailed repro of your issue, we’ll be happy to help!