Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Mongo-Observer ID Creation not Thread/Process Safe

See original GitHub issue

I often start multiple python my_experiment.py -m my_database in parallel (using subprocess.check_output). Every now and then two or more processes get the same ID in the MongoDB.

This is the basic script (without parameter setting, etc):

from multiprocessing.pool import ThreadPool as Pool
import subprocess

def run_configuration(_):
    subprocess.check_output('python build_model.py tmp ', shell=True)

with Pool(3) as pool:
    pool.map(run_configuration, 5*[None])

and the output will be

INFO - run_title - Running command 'command'
INFO - run_title - Running command 'command'
INFO - run_title - Started run with ID "427"
INFO - run_title - Started run with ID "427"
INFO - run_title - Running command 'command'
INFO - run_title - Started run with ID "428"

and I only end up with 2 entries in the database.