Example using dask.array.learn on non-trivial data

We built dask.array.learn a long while ago https://github.com/blaze/dask/pull/138 .

It’s a fairly trivial pairing of sklearn and dask.array (and honestly something that sklearn does just fine with a for loop). Still, it’s something that more established libraries show off, so we might as well have a clear user example on how to combine a large HDF5 file with an sklearn classifier using dask.array, probably something like the following

from sklearn import MyFavoriteEstimator
e = MyFavoriteEstimator()

import h5py
X = h5py.File(...)['/X']
y = h5py.File(...)['/X']

import dask.array as da
X = da.from_array(X, chunks=(...))
y = da.from_array(y, chunks=(...))

da.learn.fit(e, X, y)
e.predict(new_X)

The hard thing here is to find a large and meaningful dataset and the right estimator within sklearn that supports partial fitting. Then this work should be publicized in some way, perhaps both in documentation and in some sort of blogpost / notebook.

[shoyer]

Dictionary learning with a large stack of images would be an interesting application.

[mrocklin]

This is ill-formed and stale. Closing.

Also: http://matthewrocklin.com/blog/work/2016/04/20/dask-distributed-part-5