padding datetime arrays with missing values

Describe the issue:

Trying to pad with NaT (the nan-equivalent for datetime dtypes) results in a TypeError:

TypeError: `pad_value` must be composed of integral typed values.

This is the case for np.datetime64("NaT"), np.array("NaT", dtype="datetime64") and np.array(["NaT"], dtype="datetime64")

I think the reason for this is that dask.array.creation.expand_pad_value checks if the value is an instance of Number or Sequence, but that evaluates to False for dtype instances and 0d arrays.

Minimal Complete Verifiable Example

In [1]: import dask.array as da
   ...: import pandas as pd
   ...: import numpy as np

In [2]: a = da.from_array(
   ...:     pd.date_range("2022-04-01", freq="s", periods=15).to_numpy(), chunks=(5,)
   ...: )
   ...: a
Out[2]: dask.array<array, shape=(15,), dtype=datetime64[ns], chunksize=(5,), chunktype=numpy.ndarray>

In [3]: np.pad(a, (1, 1), mode="constant", constant_values=np.datetime64("NaT"))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In [3], line 1
----> 1 np.pad(a, (1, 1), mode="constant", constant_values=np.datetime64("NaT"))

File <__array_function__ internals>:180, in pad(*args, **kwargs)

File .../dask/array/core.py:1760, in Array.__array_function__(self, func, types, args, kwargs)
   1757 if has_keyword(da_func, "like"):
   1758     kwargs["like"] = self
-> 1760 return da_func(*args, **kwargs)

File .../dask/array/creation.py:1231, in pad(array, pad_width, mode, **kwargs)
   1229 elif mode == "constant":
   1230     kwargs.setdefault("constant_values", 0)
-> 1231     return pad_edge(array, pad_width, mode, **kwargs)
   1232 elif mode == "linear_ramp":
   1233     kwargs.setdefault("end_values", 0)

File .../dask/array/creation.py:966, in pad_edge(array, pad_width, mode, **kwargs)
    959 def pad_edge(array, pad_width, mode, **kwargs):
    960     """
    961     Helper function for padding edges.
    962 
    963     Handles the cases where the only the values on the edge are needed.
    964     """
--> 966     kwargs = {k: expand_pad_value(array, v) for k, v in kwargs.items()}
    968     result = array
    969     for d in range(array.ndim):

File .../dask/array/creation.py:966, in <dictcomp>(.0)
    959 def pad_edge(array, pad_width, mode, **kwargs):
    960     """
    961     Helper function for padding edges.
    962 
    963     Handles the cases where the only the values on the edge are needed.
    964     """
--> 966     kwargs = {k: expand_pad_value(array, v) for k, v in kwargs.items()}
    968     result = array
    969     for d in range(array.ndim):

File .../dask/array/creation.py:912, in expand_pad_value(array, pad_value)
    910     pad_value = array.ndim * (tuple(pad_value[0]),)
    911 else:
--> 912     raise TypeError("`pad_value` must be composed of integral typed values.")
    914 return pad_value

TypeError: `pad_value` must be composed of integral typed values.

Anything else we need to know?:

I wonder if 0d arrays could be considered scalars in general? That would also help fixing xarray-contrib/pint-xarray#186

Environment:

• Dask version: 2022.10.2+11.g9d624c6df
• Python version: 3.10.6
• Operating System: ubuntu 22.04
• Install method (conda, pip, source): pip install -e <path to git checkout>

[keewis]

I’m curious what np.isscalar(pad_value) gets us beyond what isinstance(pad_value, Number) already provides

I was not sure whether numpy scalars define ndim, so my idea was to check if “pad_value is a number, a numpy scalar, or a 0d array”. However, since numpy scalars do define ndim == 0 I agree that it doesn’t add anything and can be removed.

If you’ve got bandwidth for that a PR would be welcome

Sure, I’ll try to take a stab at this on Monday.

[phobson]

Thanks for the report @keewis. Agree that it might be related to #4841. I also notice that this fails with any datetime value, not just missing ones.

I get the same error with:

np.pad(a, (1, 1), mode="constant", constant_values=np.datetime64("1970-01-01T00:00:00"))

On Dask Version: 2022.10.2+4.gee8812b8 & 2022.10.2+11.g9d624c6d