module 'numcodecs' has no attribute 'Shuffle'

I’m a newbie here, so apologies if I’m missing something obvious! I’m trying to read a NetCDF4 file over http(s) and am getting the error message “module 'numcodecs' has no attribute 'Shuffle'”. This has happened with a range of files.

I’ve done quite a lot of googling about this error but haven’t found a solution (or at least, not one that I understood!) Perhaps this is the closest discussion: https://github.com/zarr-developers/numcodecs/issues/260.

I’m not sure if this is an issue in kerchunk.hdf or whether it’s deeper in numcodecs. In case it helps, I’m on an M1 Mac - sometimes this causes issues because there’s a missing binary (ARM) dependency somewhere in the tree.

Here’s some code that reproduces the error on my system. I’m using one of the free NetCDF4 files provided by Unidata as the example:

import json
import fsspec
import kerchunk.hdf

URL = "https://www.unidata.ucar.edu/software/netcdf/examples/test_echam_spectral-deflated.nc"

with fsspec.open(URL, "rb") as f:
    h5chunks = kerchunk.hdf.SingleHdf5ToZarr(f, URL, inline_threshold=100)
    print(json.dumps(h5chunks.translate()))

Traceback:

Traceback (most recent call last):
  File "/Users/me/Code/kerchunk-test/test2.py", line 12, in <module>
    print(json.dumps(h5chunks.translate()))
  File "/Users/me/opt/miniconda3/envs/kerchunk-test/lib/python3.10/site-packages/kerchunk/hdf.py", line 73, in translate
    self._h5f.visititems(self._translator)
  File "/Users/me/opt/miniconda3/envs/kerchunk-test/lib/python3.10/site-packages/h5py/_hl/group.py", line 613, in visititems
    return h5o.visit(self.id, proxy)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 355, in h5py.h5o.visit
  File "h5py/h5o.pyx", line 302, in h5py.h5o.cb_obj_simple
  File "/Users/me/opt/miniconda3/envs/kerchunk-test/lib/python3.10/site-packages/h5py/_hl/group.py", line 612, in proxy
    return func(name, self[name])
  File "/Users/me/opt/miniconda3/envs/kerchunk-test/lib/python3.10/site-packages/kerchunk/hdf.py", line 182, in _translator
    filters.append(numcodecs.Shuffle(elementsize=h5obj.dtype.itemsize))
AttributeError: module 'numcodecs' has no attribute 'Shuffle'

[martindurant]

Sounds like we should make a kerchunk conda-forge release 😃

[peterm790]

This runs with no error on my intel laptop or on a Qhub/aws instance, so I suspect you are right that it’s a numcodecs issue specific to M1.