Hyperparameter optimization benchmarking
See original GitHub issueIt’d be nice to have some benchmarks for how our different hyperparameter optimizers perform. There are a few comparisons that would be useful
- dask_ml’s drop-in replacements for GridSearchCV & RandomizedSearchCV. We’re able to deconstruct Pipeline objects to avoid redundant fit calls. This benchmark would compare a
GridSearchCV(Pipeline(...))
fordask_ml.model_selection.GridSearchCV
andsklearn.model_selection.GridSearchCV
. We’d expect Dask-ML’s to perform better the more CV splits there are and the more parameters that are explored early on in the pipeline (https://github.com/dask/dask-ml/issues/141 has some discussion). - Scaling of Dask’s joblib backend for large problems. Internally, scikit-learn uses joblib for parallel for loops. With
with joblib.parallel_backend("dask"):
...
The items in the for loop are executed on the Dask Cluster. There are some issues with the backend (https://github.com/joblib/joblib/issues/1020, https://github.com/joblib/joblib/issues/1025). Fixing those aren’t in scope for this work, but we’d like to have benchmarks to understand the current performance and measure the speedup from fixing those. 3. General performance on large datasets with Incremental, Hyperband, etc. We can’t really compare to scikit-learn here, since it doesn’t handle larger-than-memory datasets. @stsievert may have some thoughts / benchmarks to share here.
cc @dankerrigan. This is more than enough work I think. If you’re able to make progress on any of these (or other things you think are important) it’d be great.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:15 (11 by maintainers)
Top GitHub Comments
I think that I’m the western-most person who would be interested in this. My day starts around 2:30 UTC (7:30 US Pacific, 10:30 US Eastern). I suggest that if people are interested they click the Heart icon on this comment. I’ll then send out an e-mail with some scheduling options.
FYI I’m working a lot on improving the
joblib/dask
integration these days. Among other things, I’m building a benchmark suite for thejoblib
using thedask
backend for a variety of workloads and use-cases, including things likescikit-learn
cross validations, GridSearch etc. So I’m very interested by this.