Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Deadlock in _python_exit
See original GitHub issueThis can be reproduced in joblib by uncommenting a test case:
Here is the traceback we get when running ctrl-c on the frozen pytest:
$ pytest -xk test_nested_parallel_warnings
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.10.6, pytest-7.1.3, pluggy-1.0.0
rootdir: /Users/ogrisel/code/joblib, configfile: setup.cfg, testpaths: joblib
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, timeout-2.1.0
collecting ... /Users/ogrisel/code/joblib/joblib/executor.py:108: PytestCollectionWarning: cannot collect test class '_TestingMemmappingExecutor' because it has a __init__ constructor (from: joblib/test/test_memmapping.py)
class _TestingMemmappingExecutor(MemmappingExecutor):
collected 1225 items / 1219 deselected / 2 skipped / 6 selected
joblib/test/test_parallel.py ...... [100%]
======================================================= 6 passed, 2 skipped, 1219 deselected in 2.24s =======================================================
[DEBUG:MainProcess:MainThread] Interpreter shutting down. Waking up executor_manager_thread [(<_ExecutorManagerThread(ExecutorManagerThread, started 6172438528)>, (<unlocked _thread.lock object at 0x1042a82c0>, <joblib.externals.loky.process_executor._ThreadWakeup object at 0x10441c1c0>))]
[DEBUG:MainProcess:ExecutorManagerThread] closing call_queue
[DEBUG:MainProcess:ExecutorManagerThread] telling queue thread to quit
[DEBUG:MainProcess:ExecutorManagerThread] Queue.join_thread()
[DEBUG:MainProcess:ExecutorManagerThread] closing result_queue
[DEBUG:MainProcess:ExecutorManagerThread] closing thread_wakeup
[DEBUG:MainProcess:ExecutorManagerThread] joining processes
[DEBUG:MainProcess:QueueFeederThread] feeder thread got sentinel -- exiting
^CException ignored in: <module 'threading' from '/Users/ogrisel/mambaforge/envs/dev/lib/python3.10/threading.py'>
Traceback (most recent call last):
File "/Users/ogrisel/mambaforge/envs/dev/lib/python3.10/threading.py", line 1537, in _shutdown
atexit_call()
File "/Users/ogrisel/code/joblib/joblib/externals/loky/process_executor.py", line 193, in _python_exit
thread.join()
File "/Users/ogrisel/mambaforge/envs/dev/lib/python3.10/threading.py", line 1096, in join
self._wait_for_tstate_lock()
File "/Users/ogrisel/mambaforge/envs/dev/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt:
[INFO:MainProcess:MainThread] process shutting down
[DEBUG:MainProcess:MainThread] running all "atexit" finalizers with priority >= 0
[INFO:MainProcess:MainThread] calling join() for process LokyProcess-1
[INFO:MainProcess:MainThread] calling join() for process LokyProcess-2
[DEBUG:MainProcess:MainThread] running the remaining "atexit" finalizers
Issue Analytics
- State:
- Created a year ago
- Comments:14 (6 by maintainers)
Top Results From Across the Web
How to Identify a Deadlock in Python
A deadlock is a concurrency failure mode where a thread or threads wait for a condition that never occurs. The result is that...
Read more >Python | Locking without Deadlocks - GeeksforGeeks
This article focuses on dealing with how to get more than one lock at a time if a multithread program is given along...
Read more >SystemExit deadlock with multiprocessing + asyncio
The solution/workaround seems to be to make the worker a daemon process so that _exit_function() will explicitly terminate it before p.join() , ...
Read more >Debugging python deadlocks - Praveen Venkatesh
Sometimes, a python script just gives up and hangs. Even Ctrl-C does not exit the program. You do not have to have a...
Read more >Deadlock in multiprocessing.pool.Pool on terminate
Finally, processes enter into a deadlock when parent join()-s workers. This is tricky to reproduce but can definitely happen.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
My feeling is that it could well be the same issue.
Another variation I tried: On the
mainbranch, editjoblib/test_parallel.pyand uncomment the nested parallel test with loky nested within loky. I run only the loky within loky test, the test pass but I get a timeout at shutdown, something similar to https://github.com/joblib/loky/issues/363#issuecomment-1240887687.On your
debug-loky-deadlock-in-joblibbranch, the shutdown does not time out, but I get lingeringlokyprocesses for some reason. Looking at https://github.com/joblib/joblib/compare/master...ogrisel:joblib:debug-loky-deadlock-in-joblib it only looks like additional debug code, so not sure what is happening.Maybe this is OSX-specific, I can not reproduce the issue on my Linux machine: