Rounding issues when generating with constraints

Problem

Hi, I’ve been trying to generate trials for an experiment with constraints, and I keep getting an error because the proposed trials don’t satisfy the constraints by small differences, probably due to some rounding error somewhere. I did submit a similar issue once (#439) that seemed to be resolved, but may be related.

How To Replicate Error

from ax.service.ax_client import AxClient
from ax.modelbridge import get_sobol
from ax.modelbridge.registry import Models

client = AxClient()
client.create_experiment(
    name="experiment",
    parameters=[
        {"name": "x", "type": "range", "bounds": [0, 10], "value_type": "float"},
        {"name": "y", "type": "range", "bounds": [0, 10], "value_type": "float"},
    ],
    parameter_constraints=[
        "x + y <= 15",
        "x + y >= 10"
        ],
)

# Generate and attach sobol points
search_space = client.experiment.search_space
sobol = get_sobol(search_space, deduplicate=True, seed=0)
sobol_points = sobol.gen(5)
for i, arm in enumerate(sobol_points.arms):
    params = arm.parameters
    _, idx = client.attach_trial(params)
    client.complete_trial(idx, params["x"])

# Generate and attach from GP
bopt = Models.BOTORCH(
    experiment=client.experiment,
    data=client.experiment.fetch_data(),
)
samples = bopt.gen(5).param_df.reset_index(drop=True).to_dict("records")
for sample in samples:
    _, idx = client.attach_trial(sample)
    client.complete_trial(idx, 0.5)

Stack Trace

[INFO 11-29 15:11:39] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 2 decimal points.
[INFO 11-29 15:11:39] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 11-29 15:11:39] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 5 trials, GPEI for subsequent trials]). Iterations after 5 will take longer to generate due to  model-fitting.
[INFO 11-29 15:11:39] ax.service.ax_client: Attached custom parameterization {'x': 4.75, 'y': 5.93} as trial 0.
[INFO 11-29 15:11:39] ax.service.ax_client: Completed trial 0 with data: {'objective': (4.75, None)}.
[INFO 11-29 15:11:39] ax.service.ax_client: Attached custom parameterization {'x': 2.15, 'y': 9.74} as trial 1.
[INFO 11-29 15:11:39] ax.service.ax_client: Completed trial 1 with data: {'objective': (2.15, None)}.
[INFO 11-29 15:11:39] ax.service.ax_client: Attached custom parameterization {'x': 8.54, 'y': 4.03} as trial 2.
[INFO 11-29 15:11:39] ax.service.ax_client: Completed trial 2 with data: {'objective': (8.54, None)}.
[INFO 11-29 15:11:39] ax.service.ax_client: Attached custom parameterization {'x': 7.42, 'y': 6.95} as trial 3.
[INFO 11-29 15:11:39] ax.service.ax_client: Completed trial 3 with data: {'objective': (7.42, None)}.
[INFO 11-29 15:11:39] ax.service.ax_client: Attached custom parameterization {'x': 2.58, 'y': 7.87} as trial 4.
[INFO 11-29 15:11:39] ax.service.ax_client: Completed trial 4 with data: {'objective': (2.58, None)}.
{'x': 9.999999999999755, 'y': 5.000000000001556}
Traceback (most recent call last):
  File "error.py", line 35, in <module>
    _, idx = client.attach_trial(sample)
  File "/Users/yoavnavon/Documents/NotCo/Bridge/toolbox-backend-2/.venv/lib/python3.8/site-packages/ax/service/ax_client.py", line 620, in attach_trial
    self._validate_search_space_membership(parameters=parameters)
  File "/Users/yoavnavon/Documents/NotCo/Bridge/toolbox-backend-2/.venv/lib/python3.8/site-packages/ax/service/ax_client.py", line 1375, in _validate_search_space_membership
    self.experiment.search_space.check_membership(
  File "/Users/yoavnavon/Documents/NotCo/Bridge/toolbox-backend-2/.venv/lib/python3.8/site-packages/ax/core/search_space.py", line 181, in check_membership
    raise ValueError(f"Parameter constraint {constraint} is violated.")
ValueError: Parameter constraint ParameterConstraint(1.0*x + 1.0*y <= 15.0) is violated.

[Balandat]

It looks like here the constraint not being satisfied is just an issue of numerical inaccuracy. I wonder if we should just change https://github.com/facebook/Ax/blob/4929a2e93e92d31c61c29974610d363df361bafb/ax/core/parameter_constraint.py#L78 to sth like weighted_sum <= self._bound + eps where ideally eps is dynamically looked up as the machine epsilon (e.g. via numpy). I can’t really see much downsides to be just a little bit forgiving here

[lena-kashtelyan]

The fix is now part of the latest stable version: 0.2.3.