Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

joblib muliprocessing not working on > spyder 5?

See original GitHub issue

Issue Report Checklist

Searched the issues page for similar reports
Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
Could not reproduce inside jupyter qtconsole (if console-related)
Tried basic troubleshooting (if a bug/error)
- Restarted Spyder
- Reset preferences with spyder --reset
- Reinstalled the latest version of Anaconda
- Tried the other applicable steps from the Troubleshooting Guide
Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

I am using joblib for multiprocessing.

I get the following error with 5.1.3 and later: BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

I seem not to get the problem with 5.0.3.

What steps reproduce the problem?

run joblib multiprocessing with 5.0.3
run joblib multiprocessing with 5.1.4

What is the expected output? What do you see instead?

Both of the above should work. I obsrve today that only step #1 works.

Paste Traceback/Error Below (if applicable)


PASTE TRACEBACK HERE

Versions

Spyder version: 5.0.3 / 5.1.4
Python version: 3.8.10
Qt version:
PyQt version:
Operating System name/version: Windows 10

Dependencies

PASTE DEPENDENCIES HERE

Issue Analytics

State:
Created 2 years ago
Comments:15 (7 by maintainers)

Top GitHub Comments

1reaction

dkent287commented, Sep 16, 2021

You omitted this crucial part, which shows that this problem has nothing to do with Spyder. Please recreate your environment and install everything from conda-forge. The problem is conda-forge and defaults packages are binary incompatible, so you can’t mix them (as you’re doing right now).

That worked - works find now.

0reactions

ccordoba12commented, Sep 15, 2021

FileNotFoundError: Could not find module ‘C:\Users\dkent\Anaconda3\envs\env2\lib\site-packages\scipy.libs\libbanded5x.UGR6EUQPIWHQH7SL62IWIXB5545VDNQZ.gfortran-win_amd64.dll’ (or one of its dependencies). Try using the full path with constructor syntax.

You omitted this crucial part, which shows that this problem has nothing to do with Spyder. Please recreate your environment and install everything from conda-forge. The problem is conda-forge and defaults packages are binary incompatible, so you can’t mix them (as you’re doing right now).

Top Results From Across the Web

Cannot use imported modules with joblib #13632 - GitHub

Description. What steps will reproduce the problem? Run joblib with an imported local function. This will generate an error ...

Parallelization with multiprocessing, joblib or ... - Stack Overflow

I tried to parallelize this for-loop, but unsuccesfully. I only get 'None' as an output. How can I parallelize this for-loop in Spyder...

joblib.Parallel running through spyder hanging on Windows

I'm running Python 3.5.1 on Windows Server 2013 at work. I have some embarrassingly parallel tasks that seem to work on Python 2.7...

Embarrassingly parallel for loops - Joblib - Read the Docs

This backend creates an instance of multiprocessing. Pool that forks the Python interpreter in multiple processes to execute each of the items of...

Parallelisation in python - Kaggle

The threading module uses threads, the multiprocessing module uses processes. The difference is that threads run in the same memory space, while processes...