question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Methods min and max give NaN in time-aware rolling window even if min_periods=1

See original GitHub issue

Code Sample, a copy-pastable example if possible

df = pd.DataFrame({'a': [None, 2, 3]}, index=pd.to_datetime(['20170403', '20170404', '20170405']))

df.rolling('3d', min_periods=1)['a'].sum()

df.rolling('3d', min_periods=1)['a'].min()
df.rolling('3d', min_periods=1)['a'].max()

Problem description

Even if we set min_periods=1, the functions min and max give NaN if there is one NaN value inside the time-aware rolling window.

However, there is no bug when the window width is fixed (not a time period):

In [397]: df.rolling(3, min_periods=1)['a'].min()
Out[397]:
2017-04-03    NaN
2017-04-04    2.0
2017-04-05    2.0
Name: a, dtype: float64

Expected Output

The expected output, analogously to the one given by the function sum, should be a non-NaN value if at least there is a non-NaN value inside the rolling window.

In [397]: df.rolling('3d', min_periods=1)['a'].min()
Out[397]:
2017-04-03    NaN
2017-04-04    2.0
2017-04-05    2.0
Name: a, dtype: float64

In [397]: df.rolling('3d', min_periods=1)['a'].min()
Out[397]:
2017-04-03    NaN
2017-04-04    2.0
2017-04-05    3.0
Name: a, dtype: float64

Output of pd.show_versions()

commit: None python: 3.4.5.final.0 python-bits: 64 OS: Linux OS-release: 2.6.32-431.29.2.el6.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: C LOCALE: None.None

pandas: 0.19.2 nose: 1.3.7 pip: 9.0.1 setuptools: 27.2.0 Cython: 0.25.2 numpy: 1.11.3 scipy: 0.18.1 statsmodels: 0.6.1 xarray: None IPython: 5.1.0 sphinx: 1.5.1 patsy: 0.4.1 dateutil: 2.6.0 pytz: 2016.10 blosc: None bottleneck: 1.2.0 tables: 3.3.0 numexpr: 2.6.1 matplotlib: 1.5.1 openpyxl: 2.4.0 xlrd: 1.0.0 xlwt: 1.1.2 xlsxwriter: 0.9.6 lxml: 3.7.2 bs4: 4.5.3 html5lib: None httplib2: 0.9.2 apiclient: None sqlalchemy: 1.1.4 pymysql: None psycopg2: None jinja2: 2.8.1 boto: 2.45.0 pandas_datareader: None

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:16 (15 by maintainers)

github_iconTop GitHub Comments

1reaction
jrebackcommented, Jun 22, 2019

thanks for checking @ihsansecer

1reaction
jrebackcommented, Apr 6, 2017

ok if someone wants to take a crack at this, have at it.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Pandas rolling gives NaN - python - Stack Overflow
The first thing to notice is that by default rolling looks for n-1 prior rows of data to aggregate, where n is the...
Read more >
Max and min keeping NaNs - File Exchange - MATLAB Central
Most arithmetic operations propagate NaN values, but max and min ignore them. For example 3+NaN and sum([3 NaN]) return NaN, but max(3, NaN)...
Read more >
Moving Average (Rolling Average) in Pandas and Python
Learn how to create a simple moving average ( rolling average) in Pandas with Python! You'll learn how to change your window size, ......
Read more >
Moving Averages and Rolling Window Statistics for Stock Prices
pandas #python #rollingPlease SUBSCRIBE:https://www.youtube.com/subscription_center?add_user=mjmacartyTry my Hands-on Python for Finance ...
Read more >
bottleneck.move module - Read the Docs
Moving window index of maximum along the specified axis, optionally ignoring NaNs. ... If the number of non-NaN values in a window is...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found