Alignment with xarray

I’m opening this issue to track and discuss how our data structure differs from xarray. Ideally I would close it when AnnData could easily be implemented via xarray.

Some previous discussion: #308

The idea

I often think of AnnData as a kind of “special case” of xarray Datasets. We just improve convenience by specializing on the 2d case, plus a few other features. It would be nice if I didn’t just think of it that way, and we could actually just use their code here.

sgkit basically accomplishes this. It basically uses a very “anndata shaped”[^1] xarray Dataset[^2] for representing genomics data. These data structures and our goals with them are so similar that searching for open issues by the sgkit devs on the xarray repository is a great way to find compatibility issues for anndata.

Additionally, zarr and OME-zarr are quite aligned with xarray. (TODO: expand with context here)

What’s missing

Some things we need, which xarray does not currently provide:

• We support repeated dimensions (e.g. obsp, varp) https://github.com/pydata/xarray/issues/3731
• We have support for the fast sparse array library (ideally we can get pydata/sparse to become fast)
• We have a nested structure (though it’s on the roadmap with datatree being implemented)
• We support categorical variables
• We are actively working on support for awkward arrays (https://github.com/theislab/anndata/pull/647 https://github.com/pydata/xarray/issues/4285, https://github.com/pystatgen/sgkit/issues/643)

[^1]: Since we’re in the same language, working with biological data, and using many of the same technologies it would make a lot of sense for us to have greater alignment with sgkit. [^2]: More context: https://github.com/single-cell-data/matrix-api/issues/11#issuecomment-1072533371

[jpivarski]

Bit of a tangent, but it might be worthwhile to write up a Data Array API issue about the Awkward Array use case.

We already talked about it here: https://github.com/data-apis/consortium-feedback/discussions/6. It sounded pretty clear that Awkward (and by extension, Arrow) are out of scope for Data Array API, and it’s understandable that the scope would have to cut off somewhere.

[SimonHeybrock]

@ivirshup The two things you point out (holding other Python arrays, dask support) are indeed somewhat sore points for us. We would like to do both, but currently have no funding to do so.

We have serialization compatible with dask, so a number of the dask multi-processing APIs can be used, but we do not have an implementation of the dask collections interface, i.e., we currently do not support chunking and operations in the style of xarray’s dask support.