question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Rolling_median fails with large number of elements

See original GitHub issue

The new version of rolling_median from 0.17.1 fails with:

pandas\algos.pyx in pandas.algos.roll_median_c (pandas\algos.c:36408)()

MemoryError: skiplist_insert failed 

when called with series containing large number of values. In comparison, the previous version of 0.17.0 was robust and never showed this error.

On my win-64 computer, I can routinely trigger the error by the following command:

import numpy as np
import pandas as pd
n=50000
pd.rolling_median(pd.Series(np.random.randn(n)), window=2, center=False)

while

import numpy as np
import pandas as pd
n=5000
pd.rolling_median(pd.Series(np.random.randn(n)), window=2, center=False)

works fine.

The actual point at which the error begins appears to be non-deterministic but is somewhere around n=35,000.

Versions are as follows:

INSTALLED VERSIONS

commit: None python: 2.7.10.final.0 python-bits: 64 OS: Windows OS-release: 7 machine: AMD64 processor: Intel64 Family 6 Model 58 Stepping 9, GenuineIntel byteorder: little LC_ALL: None LANG: None

pandas: 0.17.1 nose: 1.3.7 pip: 7.1.2 setuptools: 18.5 Cython: 0.23.4 numpy: 1.10.1 scipy: 0.16.0 statsmodels: 0.6.1 IPython: 4.0.0 sphinx: 1.3.1 patsy: 0.4.0 dateutil: 2.4.2 pytz: 2015.7 blosc: None bottleneck: 1.0.0 tables: 3.2.2 numexpr: 2.4.4 matplotlib: 1.5.0 openpyxl: 2.2.6 xlrd: 0.9.4 xlwt: 1.0.0 xlsxwriter: 0.7.7 lxml: 3.4.4 bs4: 4.4.1 html5lib: None httplib2: None apiclient: None sqlalchemy: 1.0.9 pymysql: None psycopg2: None Jinja2: None

Issue Analytics

  • State:closed
  • Created 8 years ago
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
jrebackcommented, Apr 21, 2016

0.18.0 has been out for 2 months

0reactions
hal98xcommented, Apr 21, 2016

I see the same issue. I am running 0.17.1 from spyder? A reset seems to help. I have seen this occur with 2000 element series.

File “pandas\algos.pyx”, line 1586, in pandas.algos.roll_median_c (pandas\algos.c:36408)

MemoryError: skiplist_insert failed

When is 0.18.0 going to be released? Can you really calculate a median with only 2 numbers? I am using 3 or 5, so I can reference off the center

Read more comments on GitHub >

github_iconTop Results From Across the Web

Finding median of large set of numbers too big to fit into memory
There's a few potential solutions: External merge sort - O(n log n) You basically sort the numbers on the first pass, then find...
Read more >
Algorithms to compute the running median? - Cross Validated
The basic idea is to maintain a sample of the data of a certain size (roughly 1/error term) and compute the median on...
Read more >
Median in a stream of integers (running integers)
Given that integers are read from a data stream. Find median of elements read so for in an efficient way. For simplicity assume,...
Read more >
Sliding Window Median - LeetCode
There is a sliding window of size k which is moving from the very left of the array to the very right. You...
Read more >
Data Structures: Solve 'Find the Running Median' Using Heaps
Learn how to solve 'Finding the Running Median ' using heaps. This video is a part of HackerRank's Cracking The Coding Interview Tutorial ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found