Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

da.concatenate raises an error on contatenation of custom arrays

See original GitHub issue

When I create dask Arrays from dictionaries, I get problems when concatenating them later on. I can’t see what I’m doing wrong, so I would believe this is a bug…

Here comes a minimal test script and the corresponding traceback. I’m running on python 2.7 and dask 0.15.3.

import dask.array as da
import numpy as np


def chunks_from_blocksize(shape, chunks):
    shape = tuple(map(int, shape))
    chunks = tuple(map(int, chunks))
    return tuple(((bd,) * (d // bd) + ((d % bd,) if d % bd else ())
                 if d else (0,))
                 for d, bd in zip(shape, chunks))

def onez(shape, blocksize):
    name = 'onez1563'  # unique identifier
    vchunks, hchunks = chunks_from_blocksize((shape, (blocksize, blocksize))

    dsk = {(name, i, j): (np.ones, (vcs, hcs))
             for i, vcs in enumerate(vchunks)
             for j, hcs in enumerate(hchunks)}

    dtype = np.ones(0).dtype  # take dtype default from numpy

    return da.Array(dsk, name, shape=shape, chunks=(blocksize, blocksize), dtype=dtype)


arr1 = onez((60, 70), 50)
arr2 = onez((60, 80), 50)

da.concatenate((arr1, arr2), axis=1)

traceback:

Traceback (most recent call last):
  File "test_dask.py", line 28, in <module>
    da.concatenate((arr1, arr2), axis=1)
  File "/home/a001673/.local/lib/python2.7/site-packages/dask/array/core.py", line 2346, in concatenate
    _, seq = unify_chunks(*uc_args, warn=False)
  File "/home/a001673/.local/lib/python2.7/site-packages/dask/array/core.py", line 2115, in unify_chunks
    arrays.append(a.rechunk(chunks))
  File "/home/a001673/.local/lib/python2.7/site-packages/dask/array/core.py", line 1695, in rechunk
    return rechunk(self, chunks, threshold, block_size_limit)
  File "/home/a001673/.local/lib/python2.7/site-packages/dask/array/rechunk.py", line 261, in rechunk
    raise ValueError("Provided chunks are not consistent with shape")
ValueError: Provided chunks are not consistent with shape

Thanks for any help in that matter !

Issue Analytics

State:
Created 6 years ago
Comments:9 (5 by maintainers)

Top GitHub Comments

1reaction

mrocklincommented, Oct 4, 2017

If they are in any way different in value, shape, chunks, or dtype.

0reactions

mraspaudcommented, Oct 4, 2017

Thanks. It makes sense. I my real life case, I’m using the differing attributes of the class it belongs to, and it seems to work now. Sorry again for the noise, I hadn’t fully understood the importance of the name.

Top Results From Across the Web

Concatenating two one-dimensional NumPy arrays

Use: np.concatenate([a, b]). The arrays you want to concatenate need to be passed in as a sequence, not as separate arguments. From the...

Why do I get the error “Dimensions of arrays being ...

This error is encountered when you try to vertically concatenate arrays that do not have compatible sizes. For example, to vertically concatenate two ......

A Solver for Arrays with Concatenation

Although the array property fragment without concatenation is decidable, the fragment with concate- nation is undecidable in general (e.g., when the base theory ......

Stack, Concatenate, and Block - Dask documentation

We stack many existing Dask arrays into a new array, creating a new dimension as we go. >>> import dask.array as da >>>...

Incorrect behavior when concatenating multiple ... - GitHub

Iterate through the dtypes, calling ExtensionDtype.get_concat_dtype(dtypes) . As soon as an array type says "I know how to handle all these ...