Add dask array support

Stages:

1. Basic support for dask.array
2. Stream dask.array
3. Use dask.delayed or dask.futures to help downsample images.

CC: @mrocklin @dani-lbnl xref #43 https://github.com/scisprints/2018_05_sklearn_skimage_dask/issues/12

[mrocklin]

The coarsen function might be relevant here

http://dask.pydata.org/en/latest/array-api.html#dask.array.coarsen

On Fri, May 25, 2018, 4:45 PM Matt McCormick [e-mail] wrote:

Stages:

1. Basic support for dask.array
2. Stream dask.array
3. Use dask.delayed or dask.futures to help downsample images.

CC: @mrocklin https://github.com/mrocklin @dani-lbnl https://github.com/dani-lbnl xref #43 https://github.com/InsightSoftwareConsortium/itk-jupyter-widgets/issues/43 scisprints/2018_05_sklearn_skimage_dask#12 https://github.com/scisprints/2018_05_sklearn_skimage_dask/issues/12

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/InsightSoftwareConsortium/itk-jupyter-widgets/issues/44, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszI3FbtJ66NHWn1cuorIXbBs5G3Hlks5t2JeugaJpZM4UOsT5 .

[thewtex]

you are downsampling 2048, 2048, 1600 by 10, 10, 10 which causes the array to get padded. The input array should be contiguous and its shape should be divisible by the shrink factors.

For this use case, we have to accept all data as it comes. All data is welcome 😃.

PS, that you can probably speedup your large array creating by following some of the not so much hacks suggested here: numpy/numpy#11919 though these shouldn’t be so necessary in numpy 1.16.

Nicely done!

2\. use `persist`. It is unlikely that you care about creating a final contiguous matrix (unless you do).

3\. Remove the call to `astype`. This is pretty contentious in scikit-image. I've tried to see if using smaller dtypes would help, it just depends on how the compiler can optimize them. It isn't obvious that Cython is the right tool to ensure that fast operations occur on uint8.

Finally, if you really want to hack, provide a dtype to coarsen and to local_means and feed in float32.

Good ideas, but these were unfortunately constraints from the use case.

Thanks for the reviews @hmaarrfk @mrocklin !