"auto" `{min,max}_resource` heuristic could yield different results on different processes.

Expected behavior

Estimated min_resource of the SuccessiveHalvingPruner and max_resource of HyperbandPruner using the "auto" argument, on different processes used under distributed optimization is expected to behave identically. As it’s currently implemented, although it in practice might be rare (see reproduction steps), could lead to unexpected behaviors. See https://github.com/optuna/optuna/pull/1171#issuecomment-625604963.

Steps to reproduce

1. Run distributed optimization with multiple processes with either of the two pruners given above using the "auto" resource argument. Make sure trials don’t always run for the same number of steps (e.g. the number of epochs could be a hyperparameter).
2. Note that the pruners in different processes could end up with different resources, as those are computed based on the number of steps of the completed trial(s).

[hvy]

Thanks for #1252, I took a look and had it merged.

I think that we should raise a warning when max_resource = “auto” and the step of the last intermediate value changes with each trial. What do you think?

That might be an option. At first, I was a bit concerned since having various number of steps between trials a valid use case. For instance, sampling learning rates is quite common and it’s reasonable to vary the number of steps in the same way. However, with HB and “auto” combined it might be a viable option, if we can do it without any significant overhead. With a naive approach, we probably iterate over all trials (similar to _try_initialization) in each call to prune, and although it won’t add to the time complexity as we’re doing the same thing in ASHA, I’m not sure if it’s worth just to raise a warning. I haven’t done any benchmarks though.

[hvy]

Thanks for you input and example scenario. Yes actually just documenting it might be a reasonable decision, as doing a proper fix isn’t trivial.