zarr.open_array does not properly recognize NestedDirectoryStore

import numpy
import zarr

a = zarr.create((10, 100, 100), chunks = (1, 100, 100), dtype = 'f4',
                store = zarr.NestedDirectoryStore('foo.zarr'), overwrite = True)

a[...] = 1.0

print('shape =', a.shape)
print('a has non-zero values =', numpy.any(a))

del a

a = zarr.open_array('foo.zarr', mode = 'r')

print('shape =', a.shape)
print('a has non-zero values =', numpy.any(a))

Problem description

A zarr ‘file’ created using NestedDirectoryStore will not load correctly when later opened using the path alone. The shape (and other metadata) is correct, but the values are not loaded. In the code sample above, the shape of the original and reloaded array is the same, but the contents don’t match. The second array contains only zeroes.

Version and installation information

• Value of zarr.__version__ - 2.3.2
• Value of numcodecs.__version__ - 0.6.4
• Version of Python interpreter - 3.6.7 (anaconda)
• Operating system (Linux/Windows/Mac) - Mac OS Mojave (10.14.6)
• How Zarr was installed - using conda

[stuarteberg]

+1 for this issue!

Instead of guessing the store from the file extension, can the store class name simply be added to the metadata in .zarray?

[alimanfoo]

FWIW I agree that ideally the application should not need to know ahead of time which store path separator has been used for chunks, but should be able to discover that from the array metadata. This is something I have proposed to introduce in the v3 spec, although that is still just a draft.

For the current implementation based on the v2 spec we could either leave it as is, and try to be clear in documentation that if you use nested store you need to communicate this somehow to users of the data so they know how to open correctly. Alternatively we could add some metadata field in the .zarray to communicate which chunk path separator is used.

Given this will be fixed in the v3 spec, I’d be inclined to wait until then. Happy to discuss though.

On Fri, 28 Feb 2020, 18:31 Stuart Berg, [e-mail] wrote:

+1 for this issue!

Instead of guessing the store from the file extension, can the store class name simply be added to the metadata in .zarray?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/zarr-developers/zarr-python/issues/530?email_source=notifications&email_token=AAFLYQSAXJBOBXBYOIA6LZDRFFJ6TA5CNFSM4J4M52QKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENJVNVQ#issuecomment-592664278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFLYQR7UJYHOXA4MPRODZDRFFJ6TANCNFSM4J4M52QA .