question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

scipy.stats.rv_continuous.fit returns wrapper instead of fit method for some distributions

See original GitHub issue

My issue is about scipy.stats.rv_continuous objects not returning the same type of object for the fit method for different distributions.

scipy.stats.genextreme.fit gives <bound method rv_continuous.fit of <scipy.stats._continuous_distns.genextreme_gen object at 0x7f66083e19a0>> scipy.stats.gumbel_r.fit gives <bound method _call_super_mom.<locals>.wrapper of <scipy.stats._continuous_distns.gumbel_r_gen object at 0x7f66083fb610>>

This has caused issues when I pass fit method of one of these distributions to a function called within multiprocessing pool. genextreme works fine, but gumbel_r (and other distributions like it returning wrapper instead of fit function) now result in infinite loop (not getting an error). This change wasn’t documented in the release notes.

Reproducing code example:

import multiprocessing
import scipy.stats

def func(fit_method):
    fit_method([1, 2, 3])

with multiprocessing.Pool(2) as pool:
    pool.map(func, [scipy.stats.genextreme.fit])

with multiprocessing.Pool(2) as pool:
    pool.map(func, [scipy.stats.gumbel_r.fit])

Error message:

Same for each process in the pool:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/local/lib/python3.9/multiprocessing/queues.py", line 368, in get
    return _ForkingPickler.loads(res)
AttributeError: 'gumbel_r_gen' object has no attribute 'wrapper'

Scipy/Numpy/Python version information:

numpy 1.21.0 scipy 1.7.0 python 3.9.6

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:7 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
rkerncommented, Jul 4, 2021

Try functools.wraps() and see if that helps.

The distribution itself is pickleable since all it needs is to record the name of the class. That’s why the (perfectly idiomatic) workaround of passing the distribution object itself to the subprocesses works. Pickling just the method itself, on the other hand, I don’t think we test.

0reactions
ev-brcommented, Jul 4, 2021

As a side note, I wonder why our test suite did not pick this. IIRC we do have tests to make sure distributions are picklable

Read more comments on GitHub >

github_iconTop Results From Across the Web

scipy.stats.rv_continuous.fit — SciPy v1.9.3 Manual
Return estimates of shape (if applicable), location, and scale parameters from data. The default estimation method is Maximum Likelihood Estimation (MLE), but ...
Read more >
How to define new distributions? · Issue #12133 · scipy/scipy
I would like to define new distributions using scipy.stats. ... through the methods of rv_continuous , for example fit would give a bit...
Read more >
How to fit a distribution defined with scipy.stats.rv_continuous?
I was able to define a new distribution using this class and to fit some artificial data, however the fit produces 2 variables...
Read more >
scipy.stats._continuous_distns — Copulas 0.6.0 documentation
This function is used in the fit method of distributions that override the default method ... *args, **kwds) return wrapper class ksone_gen(rv_continuous): ...
Read more >
fitter 1.5.1 documentation
Given a data sample, we use the `fit` method of SciPy to extract the parameters of that distribution that best fit the data....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found