BUG: raised error when converting (`.dt.tz_convert`) a NaT Series

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> df1 = pd.DataFrame({'id': [1,2,3,4,5], 'resolved': [np.nan,np.nan,np.nan,'2022-07-02T09:45:33Z',np.nan]})
>>> df2 = pd.DataFrame({'id': [1,2,3,4,5], 'resolved': [np.nan,np.nan,np.nan,np.nan,np.nan]})
>>> pd.to_datetime(df1['resolved'], format="%Y-%m-%dT%H:%M:%S", errors="coerce").dt.tz_convert("Europe/Madrid").dt.tz_localize(None)
0                   NaT
1                   NaT
2                   NaT
3   2022-07-02 11:45:33
4                   NaT
Name: resolved, dtype: datetime64[ns]
>>> pd.to_datetime(df2['resolved'], format="%Y-%m-%dT%H:%M:%S", errors="coerce").dt.tz_convert("Europe/Madrid").dt.tz_localize(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/x/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/accessor.py", line 94, in f
    return self._delegate_method(name, *args, **kwargs)
  File "/home/x/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/indexes/accessors.py", line 123, in _delegate_method
    result = method(*args, **kwargs)
  File "/home/x/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/indexes/datetimes.py", line 268, in tz_convert
    arr = self._data.tz_convert(tz)
  File "/home/x/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/arrays/datetimes.py", line 874, in tz_convert
    raise TypeError(
TypeError: Cannot convert tz-naive timestamps, use tz_localize to localize

Issue Description

.dt.tz_convert is able to transform a Series with mixed tz-aware datetimes and NaTs, but it raises an error when trying to convert a Series with only NaTs.

I understand that this error should be the expected when having tz-naive datetimes (for example “2022-07-02 09:45:33” instead of “2022-07-02T09:45:33Z”), but when trying to convert a Series of NaTs I think that everyone would expect a Series of NaTs as well.

Expected Behavior

Expected behaviour:

• Return a Series of NaT when converting a Series of NaT

>>> pd.to_datetime(df2['resolved'], format="%Y-%m-%dT%H:%M:%S", errors="coerce").dt.tz_convert("Europe/Madrid")
0   NaT
1   NaT
2   NaT
3   NaT
4   NaT

Keep the behaviour of:

• Return a Series of NaT and converted tz datetimes when converting a Series of NaT and tz-aware datetimes.
• Raise Cannot convert tz-naive timestamps, use tz_localize to localize error when converting a Series with tz-naive datetimes.

Installed Versions

commit : e8093ba372f9adfe79439d90fe74b0b5b6dea9d6 python : 3.9.7.final.0 python-bits : 64 OS : Linux OS-release : 5.10.102.1-microsoft-standard-WSL2 Version : #1 SMP Wed Mar 2 00:30:59 UTC 2022 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : C.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.4.3 numpy : 1.22.2 pytz : 2021.3 dateutil : 2.8.2 setuptools : 58.0.4 pip : 21.2.4 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : None IPython : 8.2.0 pandas_datareader: None bs4 : None bottleneck : None brotli : None fastparquet : None fsspec : None gcsfs : None markupsafe : None matplotlib : None numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : None snappy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None zstandard : None

[ProgrammingDao]

I have managed to reproduce the issue on the main branch of pandas. It seems that the difference stems from pd.to_datetime(<timeseries>, format="%Y-%m-%dT%H:%M:%S", errors="coerce") which sets dt.tz to UTC in the 1st case and doesn’t in the second, i.e. dt.tz equals to None. Question is: Should the function pd.to_datetime sets a default value different from None for dt.tz or should the None value be handled differently down the line ?

[simonjayhawkins]

Thanks @igonro for the comments.

Maybe it’s fine as it is. In this case, I think the docs should be updated to point out that sometimes NaN values are considered as tz-naive and sometimes tz-aware.

The examples in the docs are fairly extensive but because the examples are using different data to your comments (and don’t include missing values), probably difficult for the casual reader to grasp. Is it possible, here in the issue, to suggest the doc changes and additions you would like to see.

If NaNs are interpreted as tz-naive NaTs, then the 1st case… shouldn’t return an object Series?

NaT is a python object that is used to represent a scalar missing value (e.g. Indexing) in datetimelike arrays where the missing values are stored internally as sentinel values or using a mask.

NaT is neither tz-naive or tz-aware. These are properties of the array. So I am not sure what the question is. NaNs are treated as missing values in the array, and the result array type, tz-aware or tz-naive is determined by the other values in the data. (which cannot be determined if there are no other values for all NaN data)

I hope this code sample helps

s = pd.Series(pd.date_range("2022", periods=7))
s[3] = np.nan
s[4] = None
s[5] = pd.NaT
s[6] = ""
print(s)
# 0   2022-01-01
# 1   2022-01-02
# 2   2022-01-03
# 3          NaT
# 4          NaT
# 5          NaT
# 6          NaT
# dtype: datetime64[ns]

vals = s.astype(int)
print(vals)
# 0    1640995200000000000
# 1    1641081600000000000
# 2    1641168000000000000
# 3   -9223372036854775808
# 4   -9223372036854775808
# 5   -9223372036854775808
# 6   -9223372036854775808
# dtype: int64

np.iinfo(vals.dtype).min
# -9223372036854775808

• NaNs are kept as NaNs because we have a mixed Series with tz-naive and tz-aware datetimes.

Yes, the mixed case which is clearly ambiguous, casts to object. adding utc=True clears up the ambiguity and limitations of mixed cases is called out in the docs. Probably best not to add more complexity to this discussion.

Currently both utc=True and utc=False return a Series of dtype datetime64[ns, UTC]

That’s because “If False (default), inputs will not be coerced to UTC. Timezone-naive inputs will remain naive, while timezone-aware ones will keep their time offsets. Limitations exist for mixed offsets (typically, daylight savings), see Examples section for details.”

so the values have timezone data and since “timezone-aware ones will keep their time offsets” and the NaNs are treated as missing values, a Series of dtype datetime64[ns, UTC] appears correct to me with utc=False and I don’t think the behavior should be changed on the basis of consistency on the mixed case.