scipy.stats.rv_continuous.fit returns wrapper instead of fit method for some distributions
See original GitHub issueMy issue is about scipy.stats.rv_continuous
objects not returning the same type of object for the fit
method for different distributions.
scipy.stats.genextreme.fit
gives <bound method rv_continuous.fit of <scipy.stats._continuous_distns.genextreme_gen object at 0x7f66083e19a0>>
scipy.stats.gumbel_r.fit
gives <bound method _call_super_mom.<locals>.wrapper of <scipy.stats._continuous_distns.gumbel_r_gen object at 0x7f66083fb610>>
This has caused issues when I pass fit
method of one of these distributions to a function called within multiprocessing pool. genextreme works fine, but gumbel_r (and other distributions like it returning wrapper instead of fit function) now result in infinite loop (not getting an error). This change wasn’t documented in the release notes.
Reproducing code example:
import multiprocessing
import scipy.stats
def func(fit_method):
fit_method([1, 2, 3])
with multiprocessing.Pool(2) as pool:
pool.map(func, [scipy.stats.genextreme.fit])
with multiprocessing.Pool(2) as pool:
pool.map(func, [scipy.stats.gumbel_r.fit])
Error message:
Same for each process in the pool:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 114, in worker
task = get()
File "/usr/local/lib/python3.9/multiprocessing/queues.py", line 368, in get
return _ForkingPickler.loads(res)
AttributeError: 'gumbel_r_gen' object has no attribute 'wrapper'
Scipy/Numpy/Python version information:
numpy 1.21.0 scipy 1.7.0 python 3.9.6
Issue Analytics
- State:
- Created 2 years ago
- Comments:7 (7 by maintainers)
Try
functools.wraps()
and see if that helps.The distribution itself is pickleable since all it needs is to record the name of the class. That’s why the (perfectly idiomatic) workaround of passing the distribution object itself to the subprocesses works. Pickling just the method itself, on the other hand, I don’t think we test.
As a side note, I wonder why our test suite did not pick this. IIRC we do have tests to make sure distributions are picklable