Trying to run emcee for a fixed amount of time: MemoryError for large nsteps

General information:

• emcee version: 2.2.1
• platform: elementary OS 0.4.1 64 bit (based on Ubuntu 16.04)
• installation method (pip/conda/source/other?): conda

Problem description:

Not really a problem, perhaps a feature request? I’m attempting to set up emcee so that instead of running for nsteps, it runs for a given amount of time. This is a more reasonable approach for me when I have to run my code in a cluster and I have a maximum allocated time I want to take advantage of.

I tried setting a very large number for nsteps (so that emcee would run for and “infinite” amount of time) and then run the sampler inside a for checking if the maximum time has been elapsed. When it has, I break out.

Expected behavior:

I would expect to be able to set a value like nsteps=1000000000 to approximate an “infinite” chain, and then set it loose as described above.

Actual behavior:

I get a MemoryError for nwalkers=200 and nsteps>100000.

What have you tried so far?:

Started with a very large value for nsteps and reduced it until it worked.

Minimal example:

import emcee
import time as t

max_secs = 3600.
nsteps = 100000

nwalkers, ndim, nburn = 200, 6, 500
sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior)

# Burn-in
s = t.time()
starting_guesses = random_population()
pos, prob, state = sampler.sample(starting_guesses, iterations=nburn)
elapsed = t.time() - s

s = t.time()
for i, result in enumerate(sampler.sample(pos, lnprob0=prob, rstate0=state, iterations=nsteps)):
    elapsed += t.time() - s
    if elapsed >= max_secs:
        break
    s = t.time()

[dfm]

Your issue was that you got a memory error. If you follow the docs that I linked to, you won’t. That’s all. Inside of a sample loop, you can check the time and break out whenever you want.

On Sat, Aug 4, 2018 at 3:53 PM Gabriel Perren [e-mail] wrote:

I already save the steps manually, I’m not sure what you mean Dan. My goal was: instead of setting a nsteps parameter, set a nhours parameter which would tell emcee for how long to run.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/dfm/emcee/issues/260#issuecomment-410454976, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVYSkS5VszhWXi3VNiANtr-A4LnFC49ks5uNbVjgaJpZM4VWxlP .

– Dan Foreman-Mackey Associate Research Scientist Flatiron Institute http://dfm.io

[Gabriel-p]

I understand what you meant now, thank you. I guess I could use that to run the sampler in blocks of smaller nsteps saving and loading until the time has run out.