Selective noise inference for some observations in field experiment

Hi! We are a group of social scientists trying to use Bayesian optimizations in our experiment. We are running optimization in a complete field experiment, which means that there are some cases where we are only able to attain a very small number of observations for certain parameters. This means that we cannot get a good estimate of SEM (bootstrapping will give us SEM=0, naturally). Therefore, the data we input to the Ax experiment might look something like this (all arbitrary numbers):

	arm_name	metric_name	mean	sem	trial_index	n
0	0_0	score	6.51	0.94	0	4
1	0_1	score	7.33	0.55	0	4
2	0_2	score	6.94	0.53	0	3
3	0_3	score	9.42	0.91	0	2
4	0_4	score	3.91	0	0	1
5	0_5	score	2.50	0	0	1

We were wondering if Ax takes account of the fact that SEM being 0 when n=1 does not mean that we are fully confident that we have the right value. If it does not, what is the best way to proceed? More generally, what can we do when we are relatively less confident about the value of some values for some parameters?

We know that we can incorporate unknown variance by putting in np.nan, but it looks like we can’t put it selectively for certain parameters that we are not confident about - when we tried, it raises this error:

ValueError: Mix of known and unknown variances indicates valuation function errors. Variances should all be specified, or none should be.

Let us know if anything is unclear, and thank you so much in advance!

[Balandat]

@nwrim That indeed makes sense and inasmuch as your setup does this it should do the right thing already.

I don’t have much additional wisdom to dispense, only that if you’re trying to estimate a SEM from a very small number of observations then your error will likely not be Gaussian. So technically you’re going to be violating some of the modeling assumptions, but from a practical perspective you’re probably going to be fine (at least from an optimization perspective, but maybe be careful not to trust the model too much).

[nwrim]

Thank you so much again @Balandat!