Creating a virtual zarr datacube from COG s3 objects

@cgohlke, as mentioned at https://github.com/fsspec/kerchunk/issues/78#issuecomment-1088696226, I’d like to try reading a time stack of geotiffs via the Zarr library.

As as test, I’ve got 2 COGs on S3-API-accessible object storage that I’d like to create a virtual Zarr dataset for using the tifffile library.

I can get it going if the COGS are local, but I’m a little unsure how to handle the s3 objects.

I’m trying to do:

from tifffile import TiffSequence
import zarr
import fsspec

fs = fsspec.filesystem('s3', anon=True, client_kwargs={'endpoint_url': 'https://mghp.osn.xsede.org'})

flist = fs.ls('/rsignellbucket1/lcmap/cog')

s3_flist = [f's3://{f}' for f in flist]

# try with s3 urls:
image_sequence = TiffSequence(s3_flist, pattern=r'.*_(\d+)_V12_LCPRI.tif')
with image_sequence.aszarr() as store:
    dz = zarr.open(store, mode='r')

# try with filelike objects:
file_like_object_list = [fs.open(f) for f in flist]
image_sequence = TiffSequence(file_like_object_list, pattern=r'.*_(\d+)_V12_LCPRI.tif')

but neither one works. These buckets are public access (no egress!) so this code should be reproducible.

Here’s the Jupyter Notebook demonstrating the local success (except I’d like to use the native 2048x2048 chunking in the COG instead of the entire file!) and remote s3 failures.

What is the best way to approach this?

[martindurant]

Unless you have also updated numcodecs, you need to do

import imagecodecs.numcodecs
imagecodecs.numcodecs.register_codecs()

to make sure zarr knows about it.

[cgohlke]

am I behind?

https://github.com/cgohlke/tifffile/blob/375d97f62df6482142b51f1b38a49bdd24d18a60/examples/issue125.py#L16