Proposal: a new nullable Timestamp data type with support for non-ns resolutions

Motivation for this proposal: for full dtype support with nullable dtypes, we also need a nullable version of the datetime-like dtypes. For backwards compatibility, we need a new dtype (like we did for the other nullable dtypes), and that’s what the proposal below is describing. And when creating a new dtype, I think it is the perfect opportunity to have a different default for the resolution (eg microsecond unit instead of nanosecond).

Summary: This proposal puts forward a new TimestampDtype, a nullable extension dtype to hold timestamp data:

• A new timestamp data type that follows the pattern of the nullable dtypes (e.g. integer, boolean) with consistent missing value behaviour.
• A parameterized data type with support for multiple resolutions (from seconds through nanoseconds) and optionally time zones (unifying the tz-naive and tz-aware dtypes into a single ExtensionDtype).
• The new data type can have a better default resolution (e.g. microseconds instead of nanoseconds).
• I suggest using “timestamp” for the dtype name, because 1) we need a different name to differentiate from “datetime64” anyway and 2) this is then internally consistent with our Timestamp scalar. But an alternative could also be “Datetime64” (capitalized).

Full version at https://docs.google.com/document/d/1uCdxjlYAafdHD7f57kpkPsJV2Q9Oaxkg1a8V5steMBM/edit?usp=sharing

This would address #7307

Small illustrative code snippet:

>>> s
0   2020-01-01 00:00:00
1                  <NA>
2   2020-01-01 02:00:00
dtype: timestamp[us]

>>> s_tz
0   2020-01-01 00:00:00+01:00
1   2020-01-01 01:00:00+01:00
2   2020-01-01 02:00:00+01:00
dtype: timestamp[ns, tz=Europe/Brussels]

Looking forward to your thoughts / comments.

cc @pandas-dev/pandas-core @pandas-dev/pandas-triage

[jbrockmendel]

side-note: a PR that changed Period.__richcmp__ checking for freq equality from self.freq != other.freq to self.freq._period_dtype_code != other.freq._period_dtype_code would likely improve the performance there quite a bit

[jbrockmendel]

What’s the current status on this?

I’m planning to implement this once we are able to use cython 3