NEP-18: mean_chunk() object __array__ method not producing an array

Several Dask operations that utilize mean_chunk() from dask/array/reductions.py fail for Dask arrays created from non-NumPy (e.g., CuPy, sparse) arrays.

Some of the operations confirmed to fail are (including other non-core Dask projects):

• dask.mean()
• dask.glm.algorithms.*

A sample to reproduce the issue is given below, followed by its traceback.

import cupy
import dask.array as da

x = cupy.random.random((5000, 1000))

d = da.from_array(x, chunks=(1000, 1000), asarray=False)

d.mean().compute()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/base.py", line 156, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/base.py", line 398, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/threaded.py", line 76, in get
    pack_exception=pack_exception, **kwargs)
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/local.py", line 459, in get_async
    raise_exception(exc, tb)
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/compatibility.py", line 112, in reraise
    raise exc
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/local.py", line 230, in execute_task
    result = _execute_task(task, data)
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/core.py", line 119, in _execute_task
    return func(*args2)
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/optimization.py", line 942, in __call__
    dict(zip(self.inkeys, args)))
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/core.py", line 119, in _execute_task
    return func(*args2)
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/compatibility.py", line 93, in apply
    return func(*args, **kwargs)
  File "/home/nfs/pentschev/.local/lib/python3.5/site-packages/dask/array/reductions.py", line 336, in mean_chunk
    result['n'] = n
ValueError: object __array__ method not producing an array

[mrocklin]

So the following change is probably needed and welcome on the dask side

-   result = empty(n.shape, dtype=[('total', total.dtype), ('n', n.dtype)])
+   result = empty(n.shape, dtype=np.dtype([('total', total.dtype), ('n', n.dtype)]))

[pentschev]

I had a quick look here, but it really seems that it’s a more complicated issue than it seemed at first. I will try to investigate a little further.

By the way, I would say this is probably the highest priority bug on the Dask side regarding the CuPy integration. I’ve seen many functions breaking due to mean_chunk(), and this is also the breaking point for Dask GLM algorithms as well. Most likely we’ll uncover other bugs after fixing this as well, but we should really fix this soon.