Parameter searches: folds used in evaluate_candidates aren't consistent across calls

If I use a custom CV iterator with suffle=True, random_state=None, then different folds will be generated for each call to evaluate_candidates().

This isn’t an issue for GridSearchCV or RandomizedSearchCV since these only call evaluate_candidates() once. But this is fundamentally wrong for e.g. Successive Halving #13900, which repeatedly call evaluate_candidates but assumes the folds to be always the same (if resource != n_samples). This is even more of an issue if we implement warm start for SH: estimators would not be warmstarted on the same data.

ping in particular @jnothman , I think this is something you noted before (but can’t remember where, sorry)?

[thomasjpfan]

I would prefer storing the seed. The RandomState.get_state() tuple is quite big (compared to a seed). numpy reference

[GaelVaroquaux]

That would work too, though I don’t think we should prevent users from using a RandomState instance or None. It’s convenient to just define an instance at the top of your file and pass it down to each call, e.g. random_state=rng.

I would not mind erroring on this with a good error message.