Best practice for running multiple (possible parallel) experiments

Take a really basic example like this

ex = Experiment('basic')
ex.observers.append(FileStorageObserver.create('my_runs'))

@ex.config
def cfg():
    x = 2

@ex.automain
def run(x):
    print (x, x**3)

What is the best way of configuring multiple experiments that could be run independently, each one producing it’s own config output, e.g.

ex = Experiment('basic')
ex.observers.append(FileStorageObserver.create('my_runs'))

@ex.config
def cfg():
    x = [2, 3, 4] # take one of these at a time and pass to run

@ex.automain
def run(x):
    print (x, x**3)

[Qwlouse]

There is no notion of providing several options inside the configuration. The code you posted will simply assign the list [2, 3, 4} to x.

For trying multiple options I often use a separate script that imports the experiment and runs it many times. So e.g. like this:

from basic import ex
x_choices = [2, 3, 4]
for x in x_choices:
    ex.run(config_updates={'x': x})

A grid search can then also easily be implemented using itertools.product.

If you are interested in parallel execution, then I agree with @trickmeyer: the simplest way is to use the commanline interface, and just call the experiment many times with different parameters.

A different strategy, that is only partially supported at this point is to use the queueing function of sacred (-q) to store the configurations that you plan on running in the database with the status QUEUED, and have a set of workers that periodically query the database for such entries and run them. The querying is fully supported, but there is currently no code for the workers. I have a prototype locally, that should be included in sacred soon. But I can also post it here if you are interested.

[Qwlouse]

Closing, since it seems resolved to me, and further job scheduling support deserves its own issue(s).