Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
multiprocessing pool hangs on join
See original GitHub issueI’m trying to run some python code in parallel.
Once I received a message , I would use multiprocessing.Pool and ‘apply_async’ to process this message.
A moment later, I found multiprocessing pool hangs on join and no messages consumed.
With ps -ppid=${pid} command, I found this:
PID TTY TIME CMD
312 pts/42 00:00:05 ConsumeTP
313 pts/42 00:00:05 ConsumeTP
314 pts/42 00:00:07 ConsumeTP
315 pts/42 00:00:00 ConsumeTP
317 pts/42 00:00:05 ConsumeTP
44417 pts/42 00:00:01 ConsumeTP
44418 pts/42 00:00:02 ConsumeTP
44419 pts/42 00:00:00 ConsumeTP
44425 pts/42 00:00:01 ConsumeTP
44426 pts/42 00:00:00 ConsumeTP
These processes just in there without running. I don’t know if this is the reason why the process hangs and can’t consume message.
- What is your OS? Ubuntu 16.04
- What is your client version? I’m using this python client from https://github.com/apache/rocketmq-client-python/issues/77#issuecomment-563075087, because have the same import error.
- What is your RocketMQ version? Not clear
Issue Analytics
- State:
- Created 4 years ago
- Comments:7 (3 by maintainers)
Top Results From Across the Web
Python multiprocessing pool hangs at join? - Stack Overflow
My issue is that the code seems to hang and never exit. A few of the files fail with an exception, but i...
Read more >Why your multiprocessing Pool is stuck (it's full of sharks!)
On Linux, the default configuration of Python's multiprocessing library can lead to deadlocks and brokenness. Learn why, and how to fix it.
Read more >Handling Hang in Python Multiprocessing - Sefik Ilkin Serengil
I was using multiprocessing.pool module and call its starmap method. Starmap lets you to pass multiple items whereas regular map does not.
Read more >Programs using `multiprocessing` hang (deadlock) and never ...
For small amounts of data, the main process can join() all the spawned subprocesses and then pick up the data. This often works...
Read more >Multiprocessing Pool Wait For All Tasks To Finish in Python
We can then call the join() function. This will block the caller until all tasks in the process pools are completed and the...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Maybe you will share the secret?
I solved it and this issue has nothing to do with rocketmq. Close it.