Writing to H5AD fails if AnnData contains datetime64 objects

Writing an AnnData object to disk fails if it contains numpy datetime64 objects. Example:

import anndata
import numpy as np

ad = anndata.AnnData()

ad.uns['now'] = np.datetime64('now')

ad.write("test_h5py.h5ad")

ERROR MESSAGE

Traceback (most recent call last):
  File "/Users/luke.zappia/miniconda3/envs/test-h5py/lib/python3.9/site-packages/anndata/_io/utils.py", line 188, in func_wrapper
    return func(elem, key, val, *args, **kwargs)
  File "/Users/luke.zappia/miniconda3/envs/test-h5py/lib/python3.9/site-packages/anndata/_io/h5ad.py", line 144, in write_not_implemented
    raise NotImplementedError(
NotImplementedError: Failed to write value for uns/now, since a writer for type <class 'numpy.datetime64'> has not been implemented yet.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/luke.zappia/Desktop/test_h5py.py", line 8, in <module>
    ad.write("test_h5py.h5ad")
  File "/Users/luke.zappia/miniconda3/envs/test-h5py/lib/python3.9/site-packages/anndata/_core/anndata.py", line 1846, in write_h5ad
    _write_h5ad(
  File "/Users/luke.zappia/miniconda3/envs/test-h5py/lib/python3.9/site-packages/anndata/_io/h5ad.py", line 112, in write_h5ad
    write_attribute(f, "uns", adata.uns, dataset_kwargs=dataset_kwargs)
  File "/Users/luke.zappia/miniconda3/envs/test-h5py/lib/python3.9/functools.py", line 877, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/Users/luke.zappia/miniconda3/envs/test-h5py/lib/python3.9/site-packages/anndata/_io/h5ad.py", line 126, in write_attribute_h5ad
    _write_method(type(value))(f, key, value, *args, **kwargs)
  File "/Users/luke.zappia/miniconda3/envs/test-h5py/lib/python3.9/site-packages/anndata/_io/h5ad.py", line 286, in write_mapping
    write_attribute(f, f"{key}/{sub_key}", sub_value, dataset_kwargs=dataset_kwargs)
  File "/Users/luke.zappia/miniconda3/envs/test-h5py/lib/python3.9/functools.py", line 877, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/Users/luke.zappia/miniconda3/envs/test-h5py/lib/python3.9/site-packages/anndata/_io/h5ad.py", line 126, in write_attribute_h5ad
    _write_method(type(value))(f, key, value, *args, **kwargs)
  File "/Users/luke.zappia/miniconda3/envs/test-h5py/lib/python3.9/site-packages/anndata/_io/utils.py", line 191, in func_wrapper
    raise type(e)(
NotImplementedError: Failed to write value for uns/now, since a writer for type <class 'numpy.datetime64'> has not been implemented yet.

Above error raised while writing key 'uns/now' of <class 'h5py._hl.files.File'> from /.

This behaviour is from h5py rather than anndata. From v3.0.0 they suggest storing these values using opaque dtypes https://docs.h5py.org/en/stable/special.html#opaque-dtypes.

Not sure how big a problem this is but it came up here https://github.com/theislab/zellkonverter/issues/24 so I just wanted to raise it. Would be nice if anndata could handle this (at least with a nicer error) but maybe not a priority.

[ivirshup]

We are aiming to support more types, as well as an interface for registering extensions, for reading and writing. Date times seem like a good case for first party support, depending on how complicated this gets. I’m worried this is going to be complicated.

Some points on this:

• pandas uses different a different datetime type than numpy, to accommodate for timezones.
• Could we consolidate to a single type?
• The arrow representation of times could be a useful reference
  • arrow timestamps in the context of python

[ivirshup]

So, it turns out datetimes are super complicated and contentious. To some extent we are beholden to whatever pandas does, but it looks like what pandas does is under active development (https://github.com/pandas-dev/pandas/issues/40932).

Some notes on the state of time types in the our dependencies:

• Currently, pandas only really supports nanosecond resolution. This is one of the main things they want to change.
• Pandas also has “date ranges” with frequencies that are encoded to the dtype.
• Numpy supports many time resolutions
• Numpy also supports time resolutions with steps (e.g. the unit for the dtype is 5 seconds), unclear how common this is. There is no documentation for this in the “Datetimes and Timedeltas” doc page

Other libraries:

• apache arrow, has a limited set of units, no steps.