Older versions of anndata throw unintuitive errors when trying to read newer formats

Hi,

I wanted to get help on an error reading h5ads created by the 0.8.0rc version of anndata. In my experience, h5ads that are created using 0.8.0rc1 cannot be opened using older anndata versions.

How to reproduce

1. In an environment with 0.8.0rc1 installed:

   import scanpy as sc
   adata = sc.datasets.pbmc3k()
   adata.write_h5ad("adata.0.8.h5ad")
   

2. In an environment with 0.7.* installed (tested with 0.7.6 and 0.7.8)

   from anndata import read_h5ad
   ad = read_h5ad("adata.0.8.h5ad")
   

   You get the following error:

   ---------------------------------------------------------------------------
   KeyError                                  Traceback (most recent call last)
   /opt/conda/envs/py37good/lib/python3.7/site-packages/anndata/_io/utils.py in func_wrapper(elem, *args, **kwargs)
       176         try:
   --> 177             return func(elem, *args, **kwargs)
       178         except Exception as e:
   
   /opt/conda/envs/py37good/lib/python3.7/site-packages/anndata/_io/h5ad.py in read_group(group)
       526     if encoding_type:
   --> 527         EncodingVersions[encoding_type].check(
       528             group.name, group.attrs["encoding-version"]
   
   /opt/conda/envs/py37good/lib/python3.7/enum.py in __getitem__(cls, name)
       356     def __getitem__(cls, name):
   --> 357         return cls._member_map_[name]
       358 
   
   KeyError: 'dict'
   
   During handling of the above exception, another exception occurred:
   
   AnnDataReadError                          Traceback (most recent call last)
   ~/tmp/ipykernel_17002/906833588.py in <module>
   ----> 1 ad = read_h5ad("adata.0.8.h5ad")
   
   /opt/conda/envs/py37good/lib/python3.7/site-packages/anndata/_io/h5ad.py in read_h5ad(filename, backed, as_sparse, as_sparse_fmt, chunk_size)
       419                 d[k] = read_dataframe(f[k])
       420             else:  # Base case
   --> 421                 d[k] = read_attribute(f[k])
       422 
       423         d["raw"] = _read_raw(f, as_sparse, rdasp)
   
   /opt/conda/envs/py37good/lib/python3.7/functools.py in wrapper(*args, **kw)
       838                             '1 positional argument')
       839 
   --> 840         return dispatch(args[0].__class__)(*args, **kw)
       841 
       842     funcname = getattr(func, '__name__', 'singledispatch function')
   
   /opt/conda/envs/py37good/lib/python3.7/site-packages/anndata/_io/utils.py in func_wrapper(elem, *args, **kwargs)
       182                 parent = _get_parent(elem)
       183                 raise AnnDataReadError(
   --> 184                     f"Above error raised while reading key {elem.name!r} of "
       185                     f"type {type(elem)} from {parent}."
       186                 )
   
   AnnDataReadError: Above error raised while reading key '/layers' of type <class 'h5py._hl.group.Group'> from /.
   

I’m using h5py==3.6.0. Let me know if you need me to list anything else about my environment.

Thanks!

[jeffquinn-msk]

Hey there, I recently noticed this issue. It’s unfortunate and I think will stand in the way of wider AnnData adoption if not properly addressed, which would be a shame. I really like the AnnData abstraction and I’d like to see it stick around. Let me know if there’s anything I can do, I’m a software engineer and I have some bandwidth to help contribute

[ivirshup]

Hey, this is expected. What you’re looking for would be forward compatibility.

Sometime we update the format of an AnnData objects stored on disk. We can’t really make older versions of the library know how to deal with this. We’ve actually added some internal features in the new version which should make having some form of forward compatibility easier in the future (even if it’s just writing older versions of the schema).

Is there a reason you’d need to keep using older versions of the library once this is released?

Worst case we could make another release in the 0.7.x series with smaller forward compatible changes, but I’d need to know it’s needed first.