question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

BUG: raised error when converting (`.dt.tz_convert`) a NaT Series

See original GitHub issue

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> df1 = pd.DataFrame({'id': [1,2,3,4,5], 'resolved': [np.nan,np.nan,np.nan,'2022-07-02T09:45:33Z',np.nan]})
>>> df2 = pd.DataFrame({'id': [1,2,3,4,5], 'resolved': [np.nan,np.nan,np.nan,np.nan,np.nan]})
>>> pd.to_datetime(df1['resolved'], format="%Y-%m-%dT%H:%M:%S", errors="coerce").dt.tz_convert("Europe/Madrid").dt.tz_localize(None)
0                   NaT
1                   NaT
2                   NaT
3   2022-07-02 11:45:33
4                   NaT
Name: resolved, dtype: datetime64[ns]
>>> pd.to_datetime(df2['resolved'], format="%Y-%m-%dT%H:%M:%S", errors="coerce").dt.tz_convert("Europe/Madrid").dt.tz_localize(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/x/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/accessor.py", line 94, in f
    return self._delegate_method(name, *args, **kwargs)
  File "/home/x/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/indexes/accessors.py", line 123, in _delegate_method
    result = method(*args, **kwargs)
  File "/home/x/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/indexes/datetimes.py", line 268, in tz_convert
    arr = self._data.tz_convert(tz)
  File "/home/x/miniconda3/envs/myenv/lib/python3.9/site-packages/pandas/core/arrays/datetimes.py", line 874, in tz_convert
    raise TypeError(
TypeError: Cannot convert tz-naive timestamps, use tz_localize to localize

Issue Description

.dt.tz_convert is able to transform a Series with mixed tz-aware datetimes and NaTs, but it raises an error when trying to convert a Series with only NaTs.

I understand that this error should be the expected when having tz-naive datetimes (for example “2022-07-02 09:45:33” instead of “2022-07-02T09:45:33Z”), but when trying to convert a Series of NaTs I think that everyone would expect a Series of NaTs as well.

Expected Behavior

Expected behaviour:

  • Return a Series of NaT when converting a Series of NaT
>>> pd.to_datetime(df2['resolved'], format="%Y-%m-%dT%H:%M:%S", errors="coerce").dt.tz_convert("Europe/Madrid")
0   NaT
1   NaT
2   NaT
3   NaT
4   NaT

Keep the behaviour of:

  • Return a Series of NaT and converted tz datetimes when converting a Series of NaT and tz-aware datetimes.
  • Raise Cannot convert tz-naive timestamps, use tz_localize to localize error when converting a Series with tz-naive datetimes.

Installed Versions

commit : e8093ba372f9adfe79439d90fe74b0b5b6dea9d6 python : 3.9.7.final.0 python-bits : 64 OS : Linux OS-release : 5.10.102.1-microsoft-standard-WSL2 Version : #1 SMP Wed Mar 2 00:30:59 UTC 2022 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : C.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.4.3 numpy : 1.22.2 pytz : 2021.3 dateutil : 2.8.2 setuptools : 58.0.4 pip : 21.2.4 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : None IPython : 8.2.0 pandas_datareader: None bs4 : None bottleneck : None brotli : None fastparquet : None fsspec : None gcsfs : None markupsafe : None matplotlib : None numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : None snappy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None zstandard : None

Issue Analytics

  • State:open
  • Created a year ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
ProgrammingDaocommented, Jul 6, 2022

I have managed to reproduce the issue on the main branch of pandas. It seems that the difference stems from pd.to_datetime(<timeseries>, format="%Y-%m-%dT%H:%M:%S", errors="coerce") which sets dt.tz to UTC in the 1st case and doesn’t in the second, i.e. dt.tz equals to None. Question is: Should the function pd.to_datetime sets a default value different from None for dt.tz or should the None value be handled differently down the line ?

0reactions
simonjayhawkinscommented, Jul 27, 2022

Thanks @igonro for the comments.

Maybe it’s fine as it is. In this case, I think the docs should be updated to point out that sometimes NaN values are considered as tz-naive and sometimes tz-aware.

The examples in the docs are fairly extensive but because the examples are using different data to your comments (and don’t include missing values), probably difficult for the casual reader to grasp. Is it possible, here in the issue, to suggest the doc changes and additions you would like to see.

If NaNs are interpreted as tz-naive NaTs, then the 1st case… shouldn’t return an object Series?

NaT is a python object that is used to represent a scalar missing value (e.g. Indexing) in datetimelike arrays where the missing values are stored internally as sentinel values or using a mask.

NaT is neither tz-naive or tz-aware. These are properties of the array. So I am not sure what the question is. NaNs are treated as missing values in the array, and the result array type, tz-aware or tz-naive is determined by the other values in the data. (which cannot be determined if there are no other values for all NaN data)

I hope this code sample helps

s = pd.Series(pd.date_range("2022", periods=7))
s[3] = np.nan
s[4] = None
s[5] = pd.NaT
s[6] = ""
print(s)
# 0   2022-01-01
# 1   2022-01-02
# 2   2022-01-03
# 3          NaT
# 4          NaT
# 5          NaT
# 6          NaT
# dtype: datetime64[ns]

vals = s.astype(int)
print(vals)
# 0    1640995200000000000
# 1    1641081600000000000
# 2    1641168000000000000
# 3   -9223372036854775808
# 4   -9223372036854775808
# 5   -9223372036854775808
# 6   -9223372036854775808
# dtype: int64

np.iinfo(vals.dtype).min
# -9223372036854775808
  • NaNs are kept as NaNs because we have a mixed Series with tz-naive and tz-aware datetimes.

Yes, the mixed case which is clearly ambiguous, casts to object. adding utc=True clears up the ambiguity and limitations of mixed cases is called out in the docs. Probably best not to add more complexity to this discussion.

Currently both utc=True and utc=False return a Series of dtype datetime64[ns, UTC]

That’s because “If False (default), inputs will not be coerced to UTC. Timezone-naive inputs will remain naive, while timezone-aware ones will keep their time offsets. Limitations exist for mixed offsets (typically, daylight savings), see Examples section for details.”

so the values have timezone data and since “timezone-aware ones will keep their time offsets” and the NaNs are treated as missing values, a Series of dtype datetime64[ns, UTC] appears correct to me with utc=False and I don’t think the behavior should be changed on the basis of consistency on the mixed case.

Read more comments on GitHub >

github_iconTop Results From Across the Web

pandas.Series.dt.tz_convert — pandas 1.5.2 documentation
A tz of None will convert to UTC and remove the timezone information. Returns. Array or Index. Raises. TypeError. If Datetime Array/Index is...
Read more >
How to convert date format when there are missing/NaT ...
I have created a sample data set and intentionally kept one missing value as ' ' . In that case also it shows...
Read more >
14 Time Zone Handling — Pandas Doc - GitHub Pages
Both of these Series can be manipulated via the .dt accessor, see here. For example, to localize and convert a naive stamp to...
Read more >
Pandas Series: dt.tz_convert() function - w3resource
The tz_convert() function is used to convert tz-aware Datetime Array/Index from one time zone to another. Syntax: Series.dt.tz_convert(self, ...
Read more >
Pandas to_datetime() Usage Explained [Practical Examples]
Learn about pandas to_datetime using multiple examples to convert String, Series, DataFrame into DateTime Index. Modify the output format of the to_datetime ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found