IncrementalSearchCV runs an adaptive algorithm by default

There are two dimensions in which model selection searches can vary:

Does the search…

1. use partial_fit or fit?
2. use previous evaluations to choose which models to evaluate?

(adapted from https://github.com/dask/dask-ml/pull/370#discussion_r221819839, which is a response to https://github.com/dask/dask-ml/pull/370#discussion_r221573023)

I think (1) should be named “incremental” and (2) should be named “adaptive”.

Currently, IncrementalSearchCV implements

• a complex adaptive algorithm
• stopping on plateau
• an incremental search that’s used with _incremental.fit.

I’d like to clean the naming of IncrementalSearchCV.

[TomAugspurger]

I don’t think this should block the release (which I’m going to do today). We can easily change the name (and provide a shim for backwards compat) if we want to in the future.

[stsievert]

I’ll have difficulty recommending when people should use it vs. RandomizedSearchCV(Incremental(Est), params).

I think the hyper-parameter search documentation can re-structured to make it clear what Dask-ML offers for hyper-parameter optimization. I’ve implemented this restructure in #221.

I’ve changed the default decay_rate to 0 in d13384fc63d894a2ba080c291e69d3c6a740e955 as part of #221. This will help IncrementalSearchCV serve hyper-parameter searches that memory constrained but not compute constrained.

Functionally, this is the same as GridSearchCV(Incremental(model), params) but there are apparently some issues with that: https://github.com/dask/dask-ml/issues/421#issuecomment-435212089.