question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

BUG: tz-aware datetime with column-wise comparisions failing with np.minmum/maximum

See original GitHub issue

Code Sample, a copy-pastable example if possible

>>> import numpy as np
>>> import pandas as pd

>>> naive = pd.to_datetime(['now'])
>>> utc = naive.tz_localize('UTC')

>>> np.minimum(naive, naive)
>>> np.minimum(utc, utc)
>>> np.minimum(pd.Series(naive), pd.Series(naive))
>>> np.minimum(pd.Series(utc), pd.Series(utc))

  File "bug.py", line 10, in <module>
    np.minimum(pd.Series(utc), pd.Series(utc))
  File "~/anaconda3/lib/python3.5/site-packages/pandas/core/series.py", line 498, in __array_prepare__
    op=context[0].__name__))
TypeError: Series with dtype datetime64[ns, UTC] cannot perform the numpy op minimum

Problem description

When a tz-aware datetime is placed in a Series, the numpy operations fmin/fmax/minimum/maximum throw an error, even though these operations work fine on a tz-aware DatetimeIndex.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None python: 3.5.2.final.0 python-bits: 64 OS: Linux OS-release: 4.9.8-100.fc24.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: C LANG: C LOCALE: None.None

pandas: 0.19.2 nose: 1.3.7 pip: 9.0.1 setuptools: 27.2.0 Cython: 0.25.2 numpy: 1.11.3 scipy: 0.18.1 statsmodels: 0.8.0 xarray: 0.9.1 IPython: 4.2.0 sphinx: 1.5.1 patsy: 0.4.1 dateutil: 2.6.0 pytz: 2016.10 blosc: None bottleneck: 1.2.0 tables: 3.3.0 numexpr: 2.6.2 matplotlib: 2.0.0 openpyxl: 2.4.1 xlrd: 1.0.0 xlwt: 1.2.0 xlsxwriter: 0.9.6 lxml: 3.7.2 bs4: 4.5.3 html5lib: 0.999 httplib2: None apiclient: None sqlalchemy: 1.1.5 pymysql: None psycopg2: None jinja2: 2.9.4 boto: 2.45.0 pandas_datareader: None

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
jrebackcommented, Mar 2, 2017

and @adbull

When a tz-aware datetime is placed in a Series, the numpy operations fmin/fmax/minimum/maximum throw an error, even though these operations work fine on a tz-aware DatetimeIndex.

this is not true at all, they naively look like they are working, but because of the same issue above (numpy has no clue about timezones, and forget about missing values), these are completely wrong (they are tz shifted incorrectly)

In [78]: i = pd.DatetimeIndex(df.A)
Out[78]: 
0   2013-01-01 00:00:00-05:00
1   2013-01-02 00:00:00-05:00
2   2013-01-03 00:00:00-05:00
3   2013-01-04 00:00:00-05:00
Name: A, dtype: datetime64[ns, US/Eastern]

In [79]: np.maximum(i, i)
Out[79]: DatetimeIndex(['2013-01-01 05:00:00-05:00', '2013-01-02 05:00:00-05:00', '2013-01-03 05:00:00-05:00', '2013-01-04 05:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', name='A', freq='D')
0reactions
mroeschkecommented, Jul 12, 2019

This looks fixed on master again:

In [24]: np.minimum(pd.Series(utc), pd.Series(utc))
Out[24]:
0   2019-07-12 05:14:37.896979+00:00
dtype: datetime64[ns, UTC]

In [25]: pd.__version__
Out[25]: '0.25.0rc0+50.g5a7a8e1de'
Read more comments on GitHub >

github_iconTop Results From Across the Web

Can't compare naive and aware datetime.now() <= challenge ...
By default, the datetime object is naive in Python, so you need to make both of them either naive or aware datetime objects....
Read more >
What's new in 1.5.0 (September 19, 2022) - Pandas
Currently timezones in datetime columns are not preserved when a dataframe is converted into ORC files. df = pd.DataFrame(data ...
Read more >
pandas: doc/source/whatsnew/v0.24.0.rst - Fossies
PandasArray , which is a thin (no-copy) wrapper around a numpy.ndarray . ... The conversion from a Series or Index with timezone-aware datetime...
Read more >
Release Notes — Airflow Documentation
In order to make airflow dags test more useful as a testing and debugging tool, we no longer run a backfill job and...
Read more >
time (Transact-SQL) - SQL Server - Microsoft Learn
Time zone offset aware and preservation, No ... If the conversion is to date, the conversion fails, and error message 206 is raised: ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found