Supporting structured array records with non-trivial shape

With a structured array in NumPy, it is possible to create a record that has a non-trivial shape of its own in addition to the shape of the recorded array. An example use case would be having a record that is a centroid of some object. The centroid will have dimensions of whatever it had come from in addition to there being one centroid per record for how many ever records there are. It appears that Zarr does not support this use case ATM. Not sure if that is a bug/unsupported feature or an intentional restriction.

Example of the error:

In [1]: import numpy

In [2]: import zarr

In [3]: a = numpy.array([(2, [4.0, 6.0]), (3, [5.0, 7.0])], dtype=[("a", numpy.uint64), ("b", numpy.float, (2,))])

In [4]: z = zarr.array(a)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/opt/conda/envs/test/lib/python3.5/site-packages/zarr/meta.py in decode_array_metadata(s)
     23     try:
---> 24         dtype = decode_dtype(meta['dtype'])
     25         fill_value = decode_fill_value(meta['fill_value'], dtype)

/opt/conda/envs/test/lib/python3.5/site-packages/zarr/meta.py in decode_dtype(d)
     78 def decode_dtype(d):
---> 79     d = _decode_dtype_descr(d)
     80     return np.dtype(d)

/opt/conda/envs/test/lib/python3.5/site-packages/zarr/meta.py in _decode_dtype_descr(d)
     73         else:
---> 74             d = [(f, _decode_dtype_descr(v)) for f, v in d]
     75     return d

/opt/conda/envs/test/lib/python3.5/site-packages/zarr/meta.py in <listcomp>(.0)
     73         else:
---> 74             d = [(f, _decode_dtype_descr(v)) for f, v in d]
     75     return d

ValueError: too many values to unpack (expected 2)

During handling of the above exception, another exception occurred:

MetadataError                             Traceback (most recent call last)
<ipython-input-4-3d14a571b8d9> in <module>()
----> 1 z = zarr.array(a)

/opt/conda/envs/test/lib/python3.5/site-packages/zarr/creation.py in array(data, **kwargs)
    306 
    307     # instantiate array
--> 308     z = create(**kwargs)
    309 
    310     # fill with data

/opt/conda/envs/test/lib/python3.5/site-packages/zarr/creation.py in create(shape, chunks, dtype, compressor, fill_value, order, store, synchronizer, overwrite, path, chunk_store, filters, cache_metadata, **kwargs)
     88     # instantiate array
     89     z = Array(store, path=path, chunk_store=chunk_store,
---> 90               synchronizer=synchronizer, cache_metadata=cache_metadata)
     91 
     92     return z

/opt/conda/envs/test/lib/python3.5/site-packages/zarr/core.py in __init__(self, store, path, read_only, chunk_store, synchronizer, cache_metadata)
     98 
     99         # initialize metadata
--> 100         self._load_metadata()
    101 
    102         # initialize attributes

/opt/conda/envs/test/lib/python3.5/site-packages/zarr/core.py in _load_metadata(self)
    108         """(Re)load metadata from store."""
    109         if self._synchronizer is None:
--> 110             self._load_metadata_nosync()
    111         else:
    112             mkey = self._key_prefix + array_meta_key

/opt/conda/envs/test/lib/python3.5/site-packages/zarr/core.py in _load_metadata_nosync(self)
    123 
    124             # decode and store metadata
--> 125             meta = decode_array_metadata(meta_bytes)
    126             self._meta = meta
    127             self._shape = meta['shape']

/opt/conda/envs/test/lib/python3.5/site-packages/zarr/meta.py in decode_array_metadata(s)
     35         )
     36     except Exception as e:
---> 37         raise MetadataError('error decoding metadata: %s' % e)
     38     else:
     39         return meta

MetadataError: error decoding metadata: too many values to unpack (expected 2)

Conda environment:

name: test
channels: !!python/tuple
- conda-forge
- defaults
dependencies:
- conda-forge::blas=1.1=openblas
- conda-forge::ca-certificates=2017.1.23=0
- conda-forge::certifi=2017.1.23=py35_0
- conda-forge::decorator=4.0.11=py35_0
- conda-forge::fasteners=0.14.1=py35_2
- conda-forge::ipython=5.2.2=py35_0
- conda-forge::ipython_genutils=0.1.0=py35_0
- conda-forge::monotonic=1.2=py35_1
- conda-forge::ncurses=5.9=10
- conda-forge::numpy=1.12.0=py35_blas_openblas_200
- conda-forge::openblas=0.2.19=0
- conda-forge::openssl=1.0.2h=3
- conda-forge::pexpect=4.2.1=py35_0
- conda-forge::pickleshare=0.7.3=py35_0
- conda-forge::pip=9.0.1=py35_0
- conda-forge::prompt_toolkit=1.0.13=py35_0
- conda-forge::ptyprocess=0.5.1=py35_0
- conda-forge::pygments=2.2.0=py35_0
- conda-forge::python=3.5.3=1
- conda-forge::readline=6.2=0
- conda-forge::setuptools=33.1.0=py35_0
- conda-forge::simplegeneric=0.8.1=py35_0
- conda-forge::six=1.10.0=py35_1
- conda-forge::sqlite=3.13.0=1
- conda-forge::tk=8.5.19=1
- conda-forge::traitlets=4.3.0=py35_0
- conda-forge::wcwidth=0.1.7=py35_0
- conda-forge::wheel=0.29.0=py35_0
- conda-forge::xz=5.2.2=0
- conda-forge::zarr=2.1.4=py35_0
- conda-forge::zlib=1.2.11=0
- libgfortran=3.0.0=1
- pip:
  - ipython-genutils==0.1.0
  - prompt-toolkit==1.0.13
prefix: /opt/conda/envs/test

[alimanfoo]

What is the ETA for the 2.3 release? I am hitting this issue and see that it has been resolved in 2.3

FWIW I’d like to get 2.3 out before the end of January, although I haven’t said that out loud yet.

[vellamike]

@alimanfoo that would be great.