Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Importing sklearn messes with CPU affinity, causing n_jobs=-1 to become useless
See original GitHub issueDescription
Please see a PoC notebook.
tl:dr: import sklearn causes CPU affinity to be set to {0}. Resetting it causes RandomizedSearchCV to freeze.
Expected Results
n_jobs=-1 uses all cores and does not freeze.
Actual Results
Only one core is loaded at 100% and nothing happens for 30+ minutes. Not resetting CPU affinity or not specifying n_jobs=-1 causes the freeze to go away.
Versions
Linux-4.14.36-1-MANJARO-x86_64-with-arch-Manjaro-Linux
Python 3.6.5 (default, Apr 12 2018, 22:45:43)
[GCC 7.3.1 20180312]
NumPy 1.14.2
SciPy 1.0.1
Scikit-Learn 0.19.1
Issue Analytics
- State:
- Created 5 years ago
- Comments:16 (7 by maintainers)
Top Results From Across the Web
How to optimize for speed — scikit-learn 1.2.0 documentation
The following gives some practical guidelines to help you write efficient code for the scikit-learn project. Python, Cython or C/C++?
Read more >Managing Process Affinity in Linux - Glenn K. Lockwood
Certain types of unevenly loaded applications can experience serious performance degradation caused by the Linux scheduler treating ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@Pastafarianist any fixes?
@CentralLT um, no idea. Haven’t been using sklearn for a good while.