question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Multiprocessing hangs on Mac OS X without MKL

See original GitHub issue

This bug occurs on Mac OS X 10.9 (but does not seem to occur on Linux and Windows) with both Python 2.7 and Python 3.3 at least for numpy 1.8.1 not compiled with MKL.

The following piece of code reproduces the bug:

import multiprocessing as mp

import numpy as np


def compute(n):
    print('Enter')
    np.dot(np.eye(n), np.eye(n))
    print('Exit')


print('\nWithout multiprocessing:')
compute(1000)

print('\nWith multiprocessing:')
workers = mp.Pool(1)
results = workers.map(compute, (1000, 1000))

The above code hangs in the workers.map call when executing np.dot(np.eye(n), np.eye(n)).

If the sequential call to compute is ommitted, the code runs as it should. Also, if the argument n is reduced sufficiently (< 100 on the computer which the bug was detected on), the code runs as it should. Again, the bug is not present when numpy is compiled with MKL.

Issue Analytics

  • State:closed
  • Created 9 years ago
  • Comments:7 (6 by maintainers)

github_iconTop GitHub Comments

3reactions
beltergdcommented, Jun 25, 2014

Another option is to add the following: import os os.environ[“VECLIB_MAXIMUM_THREADS”] = “1”

0reactions
matthew-brettcommented, Jun 13, 2014

I just tried the binary above - it does not hang.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Multiprocessing hangs when finished on Mac but not Windows
The script runs fine on the Windows computer and will cycle through all of the pools. But on the Mac, it just hangs....
Read more >
Multiprocessing with TF Metal on M1 | Apple Developer Forums
Hi,. I am running into unexpected behavior where Tensorflow Metal with python multiprocessing does not work with "forking" on M1s. It used to...
Read more >
Multiprocessing.Pool() - Stuck in a Pickle
Intro. This post sheds light on a common pitfall of the Python multiprocessing module: spending too much time serializing and deserializing data ...
Read more >
Running Parallel on Mac - General - RStudio Community
I've got a Mac and I'd like to be able to parallel process on it. Ideally, I'd like to use RStudio for development...
Read more >
Frequently Asked Questions — scikit-learn 0.17.1 ...
Why do I sometime get a crash/freeze with n_jobs > 1 under OSX or Linux?¶ · multiprocessing that does · fork without ·...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found