Multiprocessing hangs on Mac OS X without MKL
See original GitHub issueThis bug occurs on Mac OS X 10.9 (but does not seem to occur on Linux and Windows) with both Python 2.7 and Python 3.3 at least for numpy 1.8.1 not compiled with MKL.
The following piece of code reproduces the bug:
import multiprocessing as mp
import numpy as np
def compute(n):
print('Enter')
np.dot(np.eye(n), np.eye(n))
print('Exit')
print('\nWithout multiprocessing:')
compute(1000)
print('\nWith multiprocessing:')
workers = mp.Pool(1)
results = workers.map(compute, (1000, 1000))
The above code hangs in the workers.map
call when executing np.dot(np.eye(n), np.eye(n))
.
If the sequential call to compute
is ommitted, the code runs as it should. Also, if the argument n
is reduced sufficiently (< 100 on the computer which the bug was detected on), the code runs as it should. Again, the bug is not present when numpy is compiled with MKL.
Issue Analytics
- State:
- Created 9 years ago
- Comments:7 (6 by maintainers)
Top Results From Across the Web
Multiprocessing hangs when finished on Mac but not Windows
The script runs fine on the Windows computer and will cycle through all of the pools. But on the Mac, it just hangs....
Read more >Multiprocessing with TF Metal on M1 | Apple Developer Forums
Hi,. I am running into unexpected behavior where Tensorflow Metal with python multiprocessing does not work with "forking" on M1s. It used to...
Read more >Multiprocessing.Pool() - Stuck in a Pickle
Intro. This post sheds light on a common pitfall of the Python multiprocessing module: spending too much time serializing and deserializing data ...
Read more >Running Parallel on Mac - General - RStudio Community
I've got a Mac and I'd like to be able to parallel process on it. Ideally, I'd like to use RStudio for development...
Read more >Frequently Asked Questions — scikit-learn 0.17.1 ...
Why do I sometime get a crash/freeze with n_jobs > 1 under OSX or Linux?¶ · multiprocessing that does · fork without ·...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Another option is to add the following: import os os.environ[“VECLIB_MAXIMUM_THREADS”] = “1”
I just tried the binary above - it does not hang.