Rolling_median fails with large number of elements
See original GitHub issueThe new version of rolling_median from 0.17.1 fails with:
pandas\algos.pyx in pandas.algos.roll_median_c (pandas\algos.c:36408)()
MemoryError: skiplist_insert failed
when called with series containing large number of values. In comparison, the previous version of 0.17.0 was robust and never showed this error.
On my win-64 computer, I can routinely trigger the error by the following command:
import numpy as np
import pandas as pd
n=50000
pd.rolling_median(pd.Series(np.random.randn(n)), window=2, center=False)
while
import numpy as np
import pandas as pd
n=5000
pd.rolling_median(pd.Series(np.random.randn(n)), window=2, center=False)
works fine.
The actual point at which the error begins appears to be non-deterministic but is somewhere around n=35,000.
Versions are as follows:
INSTALLED VERSIONS
commit: None python: 2.7.10.final.0 python-bits: 64 OS: Windows OS-release: 7 machine: AMD64 processor: Intel64 Family 6 Model 58 Stepping 9, GenuineIntel byteorder: little LC_ALL: None LANG: None
pandas: 0.17.1 nose: 1.3.7 pip: 7.1.2 setuptools: 18.5 Cython: 0.23.4 numpy: 1.10.1 scipy: 0.16.0 statsmodels: 0.6.1 IPython: 4.0.0 sphinx: 1.3.1 patsy: 0.4.0 dateutil: 2.4.2 pytz: 2015.7 blosc: None bottleneck: 1.0.0 tables: 3.2.2 numexpr: 2.4.4 matplotlib: 1.5.0 openpyxl: 2.2.6 xlrd: 0.9.4 xlwt: 1.0.0 xlsxwriter: 0.7.7 lxml: 3.4.4 bs4: 4.4.1 html5lib: None httplib2: None apiclient: None sqlalchemy: 1.0.9 pymysql: None psycopg2: None Jinja2: None
Issue Analytics
- State:
- Created 8 years ago
- Comments:7 (4 by maintainers)
0.18.0 has been out for 2 months
I see the same issue. I am running 0.17.1 from spyder? A reset seems to help. I have seen this occur with 2000 element series.
File “pandas\algos.pyx”, line 1586, in pandas.algos.roll_median_c (pandas\algos.c:36408)
MemoryError: skiplist_insert failed
When is 0.18.0 going to be released? Can you really calculate a median with only 2 numbers? I am using 3 or 5, so I can reference off the center