Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
MultiprocessingIterator Deadlock
See original GitHub issueI am new to chainer and trying to do a custom loop with MultiprocessingIterator.
Reproduce on Ubuntu 16.04 and Chainer version 2.0
import chainer
import numpy as np
dataset = np.ones((10000, 1000))
loader = chainer.iterators.MultiprocessIterator(
dataset, 32, n_processes=6,
shuffle=True, repeat=True)
while loader.epoch < 1:
x = loader.next()
print('test')
loader.finalize() # hang at worker.join() (randomly, very frequently)
Issue Analytics
- State:
- Created 6 years ago
- Comments:7 (7 by maintainers)
Top Results From Across the Web
Deadlock in Python's multiprocessing upon early termination
I'm creating a multiprocessing.Queue in Python and adding multiprocessing.Process instances to this Queue . I would like to add a function call ...
Read more >Why your multiprocessing Pool is stuck (it's full of sharks!)
In this journey through space and time you will encounter: A mysterious failure wherein Python's multiprocessing.Pool deadlocks, mysteriously.
Read more >multiprocessing — Process-based parallelism — Python 3.11 ...
Source code: Lib/multiprocessing/ Availability: not Emscripten, not WASI. ... A process cannot join itself because this would cause a deadlock.
Read more >How to Use the Multiprocessing Package in Python
In multiprocessing, each process is associated with its own memory, which doesn't lead to data corruption or deadlocks.
Read more >Multiprocessing occasionally hangs (#3111) - PyPy - Heptapod
The deadlock occurs in a subprocess when unpickling tries to import the process_chunk function from __main__ . I've pushed the issue3111-no-lock ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
OK, I close this Issue.
@okuta Ah yes, this must have been solved, because #3076 includes a test case to check for this issue. (https://github.com/chainer/chainer/pull/3076/files#diff-f4b0901816a48909da3e084290dbb1fcR394)