Calling execute over a list of circuits duplicates the main running process x9

Information

• Qiskit Terra version: 0.16.0
• Python version: 3.8.6
• Operating system: macOS Catalina v10.15.7

What is the current behavior?

When calling execute on a list of quantum circuits of length greater than one, the __main__ running process gets duplicated x9.

Steps to reproduce the problem

Run the following script:

#!/usr/bin/env python3

from qiskit import QuantumRegister, ClassicalRegister, QuantumCircuit
from qiskit import BasicAer, execute

q = QuantumRegister(3, 'q')
c = ClassicalRegister(3, 'c')
test = QuantumCircuit(q, c, name='TEST')
test.h(q)
test.measure(q, c)
circuits = [test,test]

backend = BasicAer.get_backend('qasm_simulator')
job = execute(circuits, shots=8192, backend=backend)

print("Job successfully run")

This will print Job successfully run x9 times. If circuits = [test], the issue disappears, while increasing the length of the list produces the same x9 duplicity. This behavior is persistent on different backends, including non-simulators.

What is the expected behavior?

The script should only run once. This bug might be unnecessarily overloading the backends.

[mtreinish]

@1ucian0 fwiw it’s debatable whether this is actually fixed or not. #5383 changed the default to not run in parallel on macOS with python >= 3.8. So it works for you locally because it’s not trying to run in parallel at all. You should be able to reproduce it (for certain types of installs of python, homebrew vs conda vs upstream, but not all) by setting the QISKIT_PARALLEL=TRUE env variable to force it to run in parallel by default.

[mtreinish]

My assumption is that is another instance of Python 3.8 multiprocessing on macOS switching the default method for launching processes to spawn from fork. In general python multiprocessing with macOS >=10.14 (especially on 10.15) has proven to be quite problematic. My guess is here that your script not having if __name__ == '__main__' in it is getting rerun on each worker spawn as part of transpile() and assemble()'s inner parallel_map call since the newly spawned python process can’t differentiate between what should be run in the main processes and the sub processes. Can you try changing your code to be something like:

#!/usr/bin/env python3

from qiskit import QuantumRegister, ClassicalRegister, QuantumCircuit
from qiskit import BasicAer, execute

if __name__ == '__main__':
    q = QuantumRegister(3, 'q')
    c = ClassicalRegister(3, 'c')
    test = QuantumCircuit(q, c, name='TEST')
    test.h(q)
    test.measure(q, c)
    circuits = [test,test]

    backend = BasicAer.get_backend('qasm_simulator')
    job = execute(circuits, shots=8192, backend=backend)

    print("Job successfully run")

and see if that addresses what you were seeing.

Unfortunately there’s not much we can do around this since the issue is in python and macOS. On master we’ve changed the default on this to not use multiprocessing by default on macOS because of issues like this see: #5324.