Repeated trials in experiment (and numerical errors they sometimes cause: `RuntimeError: cholesky_cpu: U(63,63) is zero, singular U.`)

I have a question regarding repeating trials. To my code I am not adding any new option compared to the Service API Example on Hartmann6 tutorial. However, I am changing the objective function and the parameters. Checking the results, I keep noticing that some runs produce lot of repeating trials. Am I missing something? Did it converge? If yes, how do I stop the repetition? By breaking the loop when the trial is identical to the previous one? Thanks.

for i in range(20):
    parameters, trial_index = ax.get_next_trial()
    ax.complete_trial(trial_index=trial_index, raw_data=evaluate(parameters))

Results: ax.get_trials_data_frame().sort_values('trial_index')

arm_name	MAE	trial_index	x1	x2	x3	x4
0	0_0	0.354344	0	3	29	56	a
3	1_0	0.392026	1	21	26	34	b
12	2_0	0.366922	2	15	88	67	a
13	3_0	0.395405	3	40	83	24	b
14	4_0	0.360699	4	7	8	60	a
15	5_0	0.36654		5	1	27	66	b
16	6_0	0.360878	6	1	35	47	b
17	7_0	0.354756	7	4	21	54	b
18	8_0	0.352988	8	4	29	55	b
19	9_0	0.355494	9	7	37	56	b
1	10_0	0.35465		10	5	31	54	b
2	11_0	0.366325	11	6	38	66	b
4	12_0	0.359888	12	5	27	57	b
5	13_0	0.351413	13	1	43	55	b
6	13_0	0.351413	14	1	43	55	b
7	13_0	0.351413	15	1	43	55	b
8	13_0	0.351413	16	1	43	55	b
9	13_0	0.351413	17	1	43	55	b
10	13_0	0.351413	18	1	43	55	b
11	13_0	0.351413	19	1	43	55	b

Note: x4 is not used in the function.

[sdaulton]

We have some improved methods in the works for better treatment of integer parameters (in the next couple weeks), which should resolve this issue. cc: @Balandat

[lena-kashtelyan]

Update: we had a team discussion around this, and here are the outcomes:

1. We are working on making our modeling layer more robust to the numerical errors caused by data logged for many repeated trials and will update this issue when the fix for that is in.
2. We are also considering adding an option to specify experiment-level trial deduplication (currently we only offer deduplication for Sobol, but not for BayesOpt).
3. In the meantime, when encountering identical or very similar trials, you can skip over repeated trials and eventually use them as a stopping criterion, as described in my commend above:

A better alternative might be not to stop when you get a repeated trial, but continue getting new trials (without completing them) until you get a new one (with some limit, of course, at which you can just stop the whole optimization).

1. If standard errors on observations are known (or is the observations are known to be noiseless, in which case standard errors are 0.0), it’s better to actually pass those values to Ax, since in that case Ax will not need to infer noise level and numerical errors will be less likely to crop up.

cc @Balandat, @eytan, @bletham, @ldworkin