Capture parameters from scikit-optimize for hyperparameter optimization

The Scikit-optimize BayesSearchCV class likely has internal functions that help it determine new parameters given scores on a set of older parameters. It would be useful to either introduce Dask to scikit-optimize or else reuse this logic and implement our own BayesSearchCV. This could be used with standard fit style algorithms or with Incremental if we have a nice early-stopping criterion.

This came out of conversation with @ogrisel

[mrocklin]

So I tried things today with joblib and the following example:

from dask.distributed import Client, progress
import distributed.joblib  # needed until sklearn 0.20
client = Client(processes=False)

from skopt import BayesSearchCV
# parameter ranges are specified by one of below
from skopt.space import Real, Categorical, Integer

from sklearn.datasets import load_iris
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.externals.joblib import parallel_backend

X, y = load_iris(True)
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.75,
                                                    random_state=0)


# log-uniform: understand as search over p = exp(x) by varying x
opt = BayesSearchCV(
    SVC(),
    {
        'C': Real(1e-6, 1e+6, prior='log-uniform'),
        'gamma': Real(1e-6, 1e+1, prior='log-uniform'),
        'degree': Integer(1,8),
        'kernel': Categorical(['linear', 'poly', 'rbf']),
    },
    n_iter=32,
    n_jobs=1,
)


with parallel_backend('dask'):
    # executes bayesian optimization
    opt.fit(X_train, y_train)

and the dashboard shows tasks like the following:

Each green rectangle is a call to fmin_l_bfgs_batch, which takes around 50ms. We’re seeing a lot of dead space in between tasks, likely meaning that skopt is doing some non-trivial work in between sending out batches.

After running this experiment I have some questions:

1. Is scikit-optimize simple enough that just using joblib is the right way to go? Or are there things that we would want to do with a more expressive interface

2. In particular, I might suggest trying to solve this problem with the concurrent.futures interface. You would submit lots of tasks (presumably a bit more than you had cores), then as they completed you would update your next best guess and submit that one out to run with the others. This is maybe a similar problem to what is running today, you just have more tasks in flight.

3. Is there a problem with a larger cost so that we can see how much the overhead hurts in a realistic setting?

4. What is the cause of the overhead here?

   Actually I profiled a bit. It’s spending a bit of time in scipy.optimize, coming from the skopt/optimizer/optimizer.py::Optimizer._tell method.

[iaroslav-ai]

So to elaborate on your comment, the following is happening:

• Generate n_jobs points to evaluate using a “constant liar” approach
• Evaluate these points in parallel
• Update the model of the costly objective
• Go to 1