Calculating for high variance when the mean of cv_scores is zero

Right now, our calculation for high variance is:

        cv_scores_std = cv_scores.std()
        cv_scores_mean = cv_scores.mean()
        if cv_scores_std != 0 and cv_scores_mean != 0:
            high_variance_cv = bool(abs(cv_scores_std / cv_scores_mean) > threshold)```

This was put in place to prevent a divide-by-zero warning / error. However, this impl is not ideal: if the mean is close to zero (but not zero), high variance will likely be true (dividing by small number).

In addition, it is possible for there to be high variance if the mean is zero but the standard deviation is very high. Right now, we default to False but we may want to look for a impl that is smarter in this situation.

[freddyaboulton]

@angela97lin Thanks for explaining! The problem of dividing by zero only happens with the objectives that can take on negative values. For those objectives, I wonder if our use of the coefficient of variation is even valid?

I looked at the wiki, and the definition section mentions that it should only be calculated for data on a ratio scale, i.e. data measured on a scale with a “meaningful zero”. I take that to mean that the data can’t be negative. This stats stackexchange post seems to support that interpretation.

With that in mind, I guess I see four options:

1. If the mean is <=0, don’t calculate the coefficient of variation. We would never raise the warning in this situation.
2. Add an epsilon to the denominator. I guess it depends on the value of epsilon but basically we would always raise the warning if the mean is zero.
3. Don’t change anything. We would never raise the warning when the mean == 0.
4. Find a different measure of overfitting that accounts for objectives that can take on negative values.

What do you think?

[angela97lin]

@freddyaboulton Ah sorry, the order of the PRs is a bit confusing. You’re right, #2024 handled the case where mean is zero so that we don’t trigger the the divide by zero issue. What I wanted this PR issue to track was a better way to calculate / handle when the mean is zero, since right now we’re defaulting to False. Will update the description :d