problem with mulltiprocessing

I have tried to use emcee in a class and also make multiprocessing module to work but after running such a test code:

import numpy as np
import emcee
import scipy.optimize as op
# Choose the "true" parameters.
m_true = -0.9594
b_true = 4.294
f_true = 0.534

# Generate some synthetic data from the model.
N = 50
x = np.sort(10*np.random.rand(N))
yerr = 0.1+0.5*np.random.rand(N)
y = m_true*x+b_true
y += np.abs(f_true*y) * np.random.randn(N)
y += yerr * np.random.randn(N)

class modelfit():
      def  __init__(self):
          self.x=x
          self.y=y
          self.yerr=yerr
          self.m=-0.6
          self.b=2.0
          self.f=0.9
      def get_results(self):
          def func(a):
              model=a[0]*self.x+a[1]
              inv_sigma2 = 1.0/(self.yerr**2 + model**2*np.exp(2*a[2]))
              return 0.5*(np.sum((self.y-model)**2*inv_sigma2 + np.log(inv_sigma2)))
          result = op.minimize(func, [self.m, self.b, np.log(self.f)],options={'gtol': 1e-6, 'disp': True})
          m_ml, b_ml, lnf_ml = result["x"]
          return result["x"]
      def lnprior(self,theta):
          m, b, lnf = theta
          if -5.0 < m < 0.5 and 0.0 < b < 10.0 and -10.0 < lnf < 1.0:
             return 0.0
          return -np.inf
      def lnprob(self,theta):
          lp = self.lnprior(theta)
          likelihood=self.lnlike(theta)
          if not np.isfinite(lp):
             return -np.inf
          return lp + likelihood
      def lnlike(self,theta):
          m, b, lnf = theta
          model = m * self.x + b
          inv_sigma2 = 1.0/(self.yerr**2 + model**2*np.exp(2*lnf))
          return -0.5*(np.sum((self.y-model)**2*inv_sigma2 - np.log(inv_sigma2)))
      def run_mcmc(self,nstep):
          ndim, nwalkers = 3, 100
          pos = [self.get_results() + 1e-4*np.random.randn(ndim) for i in range(nwalkers)]
          self.sampler = emcee.EnsembleSampler(nwalkers, ndim, self.lnprob,threads=10)
          self.sampler.run_mcmc(pos, nstep)
test=modelfit()
test.x=x
test.y=y
test.yerr=yerr
test.get_results()
test.run_mcmc(5000)

I got this error message: Traceback (most recent call last): File “<stdin>”, line 1, in <module> File “<stdin>”, line 27, in run_mcmc File “build/bdist.linux-x86_64/egg/emcee/sampler.py”, line 157, in run_mcmc File “build/bdist.linux-x86_64/egg/emcee/ensemble.py”, line 198, in sample File “build/bdist.linux-x86_64/egg/emcee/ensemble.py”, line 382, in _get_lnprob File “build/bdist.linux-x86_64/egg/emcee/interruptible_pool.py”, line 94, in map File “/vol/aibn84/data2/zahra/anaconda/lib/python2.7/multiprocessing/pool.py”, line 558, in get raise self._value NotImplementedError: pool objects cannot be passed between processes or pickled

or similar error raise self._value cPickle.PicklingError: Can’t pickle <type ‘instancemethod’>: attribute lookup builtin.instancemethod failed

I am wondering how I could solve this problem? One way that has been suggested in stackoverflow is to use pathos.multiprocesssing as it is mentioned here: http://stackoverflow.com/a/21345273/2379433

But I think it should be done in the emcee source code. Thanks in advance.

[mmckerns]

I’m the dill and pathos author. The pathos hack is primarily that it imports dill instead of cPickle. There’s some other stuff too, but that’s the big one line change. With dill you can serialize most things in python, so parallel computing (especially for statistics and monte carlo etc) becomes much easier to do. There are some packages (including ipython and mpi4py) starting to provide a “drop-in” mechanism that allows for replacement of the serializer or map functions used in their code. I’d be interested in seeing that this works with emcee.

[dstndstn]

Uhh, wasn’t the “same problem” above resolved as user error? Could you please post a minimal example showing the problem you are seeing.

Merging together the chains is more than a “hack”. That’s more like multiple restarts or something. It will definitely give you different results in detail. And you will have to be smart about burn-in (as always).

You may want to read the paper or page through some of DFM’s talks on emcee (all his slides are online); the walkers do interact (that’s what allows it to be affine-invariant and parallelizable in the first place).