Optimize doesn't work with evaluation function that returns value of some NumPy types

I noticed that when the evaluation_function returns value of some NumPy type(I was able to reproduce with np.int64, np.int32, np.float32), optimize method crashes with the following exception:

File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/service/managed_loop.py”, line 206, in optimize loop.full_run() File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/service/managed_loop.py”, line 150, in full_run self.run_trial() File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/service/managed_loop.py”, line 141, in run_trial trial.fetch_data() File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/core/base_trial.py”, line 259, in fetch_data trial_index=self.index, metrics=metrics, **kwargs File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/core/simple_experiment.py”, line 205, in _fetch_trial_data return self.eval_trial(self.trials[trial_index]) File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/core/simple_experiment.py”, line 119, in eval_trial not_none(trial.arm).parameters, None File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ax/core/simple_experiment.py”, line 192, in evaluation_function_outer "Evaluation function returned an invalid type. The function must " Exception: Evaluation function returned an invalid type. The function must either return a dictionary of metric names to mean, sem tuples or a single mean, sem tuple, or a single mean.

My environment:

numpy==1.16.1
ax-platform==0.1.2

Script to reproduce:

import numpy as np
from ax import optimize
from ax.utils.measurement.synthetic_functions import branin

best_parameters, values, experiment, model = optimize(
    parameters=[
        {
            "name": "x1",
            "type": "range",
            "bounds": [-5.0, 10.0],
        },
        {
            "name": "x2",
            "type": "range",
            "bounds": [0.0, 10.0],
        },
    ],
    evaluation_function=lambda p: np.float32(branin(p["x1"], p["x2"])),
    minimize=True,
)

[Irynei]

@lena-kashtelyan, thanks for your explanation. I was expecting the evaluation function to work with numpy types, so for me, that was a surprising behavior. No, It’s not hard to convert them for now, but thanks for the suggestion.

[lena-kashtelyan]

@Irynei, this is intentional behavior –– the evaluation function is strictly typed, and it’s return type is a union of a float, a tuple of two floats, and a dictionary of a string to such tuples, as documented in SimpleExperiment. We use python builtins for primitive types and convert from numpy primitives where necessary. However, you’re right that for the evaluation function, we should allow numpy types –– will add that support shortly.

Is it much trouble to convert back from numpy to python primitive in the evaluation function, for now? You could use our utility ax.utils.common.typeutils.numpy_type_to_python_type, if you’d like.