Noisy objective function not taken into account in `SimpleExperiment` when suggesting best parameters

I’ve been doing ax hyperparameter optimisation for a DNN doing regression on images like this:

    exp = SimpleExperiment(
        name=EXPERIMENT_NAME,
        search_space=dnn_search_space,
        evaluation_function=train_cross,
        objective_name="regression_error",
        minimize=True,
    )

    # sobol sample search space
    for i in range(20):
        exp.new_trial(generator_run=sobol.gen(1))

    # converge on best hyperparams
    best_arm = None
    for i in range(50):
        gpei = Models.GPEI(experiment=exp, data=exp.eval())
        generator_run = gpei.gen(1)
        best_arm, _ = generator_run.best_arm_predictions
        exp.new_trial(generator_run=generator_run)
        best_parameters = best_arm.parameters
        print(str(i) + " best params " + str(best_parameters))

and I have found that the “best parameters” recommended by ax tend to not change very much. This suggests that ax is giving me the hyperparameters that were evaluated and found to give the best result.

The problem with this is that the best results tend to be flukes - the training process is of course noisy and non-determinate and things that make the process more stochastic, such as very high learning rates and small batch sizes, tend to give more varied results. The more varied results will happen to include the best and worst results and on average, be worse than smoother more stable parameter sets. But Ax seems to just take the best result it finds and recommend this.

Is there some way of using Ax in which it will assume a noisy underlying object function and recommend the best hyperparameters based on an interpolation which uses all of the information available to it. rather than just, which trial scored best, one time?

[stevemandala]

Hey @LukeAI, thanks for raising this. I believe this was caused by a bug in simple experiment assuming 0.0 SEM. We recently pushed a fix on master, which should ensure we don’t default to noise-less modeling when SEM isn’t provided: https://github.com/facebook/Ax/commit/f6ccdd7be3cfff835fe2b75feaad6a800a1378ea

[lena-kashtelyan]

This should now be fixed in latest stable release, 0.1.20.