Multiprocessing Tutorial Example Not Working

General information:

• emcee version: 3.0.2
• platform: python, jupyter notebook
• installation method (pip/conda/source/other?): pip

Problem description:

Expected behavior: That the multiprocessing example using “pool” would work in a jupyter notebook for the example shown in the multiprocessing example.

Actual behavior: Using “pool” in the sampler does not converge and continues to run indefinitely. However, the example in the tutorial shows that it should finish running and run faster than the serial code.

What have you tried so far?: Modified the code, tried different notebooks, tried to use pool.close()

Minimal example:

from multiprocessing import Pool

with Pool() as pool:
    sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, pool=pool)
    start = time.time()
    sampler.run_mcmc(initial, nsteps, progress=True)
    end = time.time()
    multi_time = end - start
    print("Multiprocessing took {0:.1f} seconds".format(multi_time))
    print("{0:.1f} times faster than serial".format(serial_time / multi_time))

[dfm]

Thanks - are you on a mac? What version of Python?

The default settings for how multiprocessing on macOS have changed in Python 3.8 so that could be the cause. You could try running

import multiprocessing as mp
mp.set_start_method('fork')

before executing the above code to see if that works.

[jorgenorena]

Hi, I was having a similar problem and this seems to have fixed the issue. I had to use the following line

mp.set_start_method('fork', force=True)

Have you considered updating the documentation?