Transform Directory storage to Zip storage.

store = zarr.ZipStore("/mnt/test.zip", "r")

Problem description

Hi, sry for bothering, I found this statement inside Zarr official documentation about ZipStorage: Alternatively, use a DirectoryStore when writing the data, then manually Zip the directory and use the Zip file for subsequent reads. I am trying to transform a DirectoryStorage format Zarr dataset to a ZipStorage. I use zip operation provided in Linux. zip -r test.zip test.zarr here test.zarr is a directory storage dataset including three groups. However, when I try to use the codes above to open it, get the error as below:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/eddie/miniconda3/envs/train/lib/python3.8/site-packages/zarr/storage.py", line 1445, in __init__
    self.zf = zipfile.ZipFile(path, mode=mode, compression=compression,
  File "/home/eddie/miniconda3/envs/train/lib/python3.8/zipfile.py", line 1190, in __init__
    _check_compression(compression)
  File "/home/eddie/miniconda3/envs/train/lib/python3.8/zipfile.py", line 686, in _check_compression
    raise NotImplementedError("That compression method is not supported")
NotImplementedError: That compression method is not supported

I wonder if my compression method is wrong, and if there some workarounds to transform directory storage to zip storage or some other DB format, cause when the groups rise, the previous storage has so many nodes and not so convenient to transport. Thanks in advance.

Version and installation information

• Value of zarr.__version__: 2.8.1
• Value of numcodecs.__version__: 0.7.3
• Version of Python interpreter: 3.8.0
• Operating system (Linux/Windows/Mac): linux ubuntu 18.04
• How Zarr was installed: pip

[eddiecong]

@pmav99 Thanks for the summary. Finally, we decide to use the LMDB storage format, which supports both reads and writes in multiprocessing, by doing so, we did not have to run the additional cmd to zip the directory.

[pmav99]

@joshmoore thank you. I confirm that your proposal does indeed work.

To make this more clear. If the zip archive contains the outer directory, then the ZipStore throws an exception. If the outer directory is omitted then it works just fine.

So this fails:

$ zip -r0 foo.zarr.zip foo.zarr   
$ unzip -l foo.zarr.zip | head 
Archive:  foo.zarr.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-06-01 15:39   foo.zarr/
        0  2021-06-01 15:39   foo.zarr/lat/
      316  2021-06-01 15:39   foo.zarr/lat/.zarray
      345  2021-06-01 15:39   foo.zarr/lat/0
       50  2021-06-01 15:39   foo.zarr/lat/.zattrs
       24  2021-06-01 15:39   foo.zarr/.zgroup
        0  2021-06-01 15:39   foo.zarr/aaa/

while this works:

$ cd foo.zarr
$ zip -r0 ../foo.zarr.zip ./
$ cd ../
$ unzip -l foo.zarr.zip | head
Archive:  foo.zarr.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-06-01 15:39   lat/
      316  2021-06-01 15:39   lat/.zarray
      345  2021-06-01 15:39   lat/0
       50  2021-06-01 15:39   lat/.zattrs
       24  2021-06-01 15:39   .zgroup
        0  2021-06-01 15:39   aaa/
   184600  2021-06-01 15:39   aaa/0.3.2

# ...

AFAIK there is no way to create a suitable zip archive using zip unless you do this trick with the change of the CWD. Nevertheless, according to this SO answer it is possible to avoid cd-ing into the zarr archive by using 7z instead of zip:

$ 7z a -tzip foo.zarr.zip foo.zarr/.
$ unzip -l foo.zarr.zip | head -n20
Archive:  foo.zarr.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        2  2021-06-01 15:39   .zattrs
       24  2021-06-01 15:39   .zgroup
        0  2021-06-01 15:39   aaa/
      365  2021-06-01 15:39   aaa/.zarray
       81  2021-06-01 15:39   aaa/.zattrs
   196189  2021-06-01 15:39   aaa/0.0.0
   196189  2021-06-01 15:39   aaa/0.0.1