question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Improve doc/subprocess.rst for multiprocessing module limitation

See original GitHub issue

Is your feature request related to a problem? Please describe. I am developing a program which leverage multiprocessing module to do parallel tasks. After read doc/subprocess.rst and try, I figured out that documented solution doesn’t work for multiprocessing.Process.Fisrt, both sitecustomize.py and .pth file doesn’t work in subprocesses created by multiprocessing.Process. It’s because when use multiprocessing.Process, subprocesses won’t import site module again, there is not chance to do sitecustomize.py or .pth file customization. Second, when these subprocesses exit, atexit functions get never called, coverage has no chance to write collected data.

Describe the solution you’d like Maybe we could improve document and point out this limitation.

Describe alternatives you’ve considered I came up with a workaround.

test.py

def func():
    try:
        import coverage
        coverage.process_startup()
    except:
        traceback.print_exc()
    # Emulate time-consuming work
    time.sleep(5)
    atexit._run_exitfuncs()

p = Process(target=func)
p.start()
p.join()
COVERAGE_PROCESS_START=.coveragerc coverage run test.py

This seems work. Apparently, there are many cons. 1. there may exists side-effects when other libraries depends on this atexit feature. 2. We call atexit protected method. 3. At most times, we can not (or want to avoid to) add such code into normal code.

Additional context None

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:6 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
nedbatcommented, Dec 11, 2019

Done: 5379317ad73694099b7876797695bc76bd9cdce5

0reactions
nedbatcommented, Dec 11, 2019

I should add more mention of it to the docs.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Python multiprocessing's Pool process limit - Stack Overflow
You can ask for as many processes as you like. Any limit that may exist will be imposed by your operating system, not...
Read more >
Why your multiprocessing Pool is stuck (it's full of sharks!)
On Linux, the default configuration of Python's multiprocessing library can lead to deadlocks and brokenness. Learn why, and how to fix it.
Read more >
multiprocessing — Process-based parallelism — Python 3.11 ...
multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and ......
Read more >
Can't use multiprocessing module in IPython #10894 - GitHub
I am having issues with the multiprocessing module when using IPython. It works fine running in my console.
Read more >
Things I Wish They Told Me About Multiprocessing in Python
The better option is to pass messages using multiprocessing.Queue objects. Queues should be used to pass all data between subprocesses. This ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found