question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

histogram with Freedman Diaconis estimator crashes on small numbers

See original GitHub issue

np.histogram crashes with MemoryError when the input contains very small numbers.

In [13]:  print(sys.version)
2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2]

In [14]: print(np.__version__)
1.11.2

In [15]: x = np.array([ 2, 2, 2 - 1e-15, 2 - 1e-15, 1], dtype=np.float64)

In [16]: np.histogram(x, bins='fd')
---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-16-4ee7362b8a29> in <module>()
----> 1 np.histogram(x, bins='fd')

/home/artem/.local/lib/python2.7/site-packages/numpy/lib/function_base.pyc in histogram(a, bins, range, normed, weights, density)
    562 
    563         # Initialize empty histogram
--> 564         n = np.zeros(bins, ntype)
    565         # Pre-compute histogram scaling factor
    566         norm = bins / (mx - mn)

MemoryError: 

I did some debugging and figured out that in numpy.lib.function_base._hist_bin_fd returns bin width 1.29852472695e-15

def _hist_bin_fd(x):  # x: array([ 2, 2, 2 - 1e-15, 2 - 1e-15, 1])
    iqr = np.subtract(*np.percentile(x, [75, 25]))  # iqr: 1.11022302463e-15
    return 2.0 * iqr * x.size ** (-1.0 / 3.0)  # returns 1.29852472695e-15

I think that in numpy/lib/function_base.py:533 it would be good to check that we get a reasonable number of bins (may be max size of the array) or at least show a warning.

Issue Analytics

  • State:open
  • Created 7 years ago
  • Reactions:1
  • Comments:17 (11 by maintainers)

github_iconTop GitHub Comments

2reactions
madphysicistcommented, Oct 23, 2016

Not in the general case, no. Imagine a dataset with two clumps of data separated by a huge gap for an example of why not. It may, however, be worth considering merging together empty bins. To remove what I imagine to be the most common cause for the issue you are describing.

1reaction
madphysicistcommented, Oct 25, 2016

And raise a warning about the data having too fine a structure or something.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Determining Histogram Bin Width using the Freedman ...
Using the Freedman-Diaconis Rule to determine optimal histogram bin width. ... n in the empirical dataset, and returns a bin width estimate.
Read more >
Optimal number of bins in histogram by the Freedman ...
The function gives a formula for the number of bins. The relationship between number of bins and the width of bins will be...
Read more >
Relationship Between Traffic Volume and Accident Frequency ...
The ideal number of bins for accidents in low, middle- and high-volume intersections calculated using the Freedman–Diaconis rule.
Read more >
Re: root crashes when creating large histogram/histograms ...
That depends on what fraction of the bins are filled. But as you said you had a problem with integer-overflowing bin numbers anyway,...
Read more >
10. Kernel density estimators of the cubic power of a Gaussian...
Two histograms are constructed using Freedman-Diaconis'and Hall- Marron's rules ... a small or high number of samples provides a poor density estimation.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found