Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Multiprocessing issue 'Can't pickle <function ....'

See original GitHub issue

Hey, the following code is not working in ptpython on python version 3.

from multiprocessing.pool import Pool

def do(x):
    return x 

pool = Pool(processes=16,  maxtasksperchild=100)
done_x = []
for x in pool.imap_unordered(do, [1, 2, 3, 4, 5]):
    done_x.append(x)

It produces the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 695, in next
    raise value
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 385, in _handle_tasks
    put(task)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function do at 0x10388c268>: attribute lookup do on __main__ failed
Can't pickle <function do at 0x10388c268>: attribute lookup do on __main__ failed

I have tested the code on Python 3.5.2 on ubuntu and mac os. It is working correctly in default python’s REPL, in jupyter, and by running it directly.

Issue Analytics

State:
Created 6 years ago
Reactions:3
Comments:5 (1 by maintainers)

Top GitHub Comments

3reactions

ttarter3commented, Apr 9, 2018

@danai-antoniou I updated Anaconda to the latest version and that fixed the problem for me. Not sure which package was the issue. I read online it could be multiprocessing, Ipython, or pickle.

1reaction

danai-antonioucommented, Apr 9, 2018

@jonathanslenders Any update on this? Also experiencing this bug

Top Results From Across the Web

Python multiprocessing PicklingError: Can't pickle <type ...

To get a PicklingError something must be put on the Queue which is not picklable. It could be the function or its arguments....

Python “multiprocessing” “Can't pickle…” - TedChen - Medium

The 'pickle' issue in 'multiprocessing' is because of passing objects as arguments between process. There is 3 methods to start process in 'multiprocessing'....

Multiprocessing and Pickle, How to Easily fix that?

However, the multiprocess tasks can't be pickled; it would raise an error failing to pickle. That's because when dividing a single task over ......

Ways to Solve Can't Pickle local object Error - Python Pool

In this program, we are going to see how to rectify the attribute error while multiprocessing. Import multiprocessing. Create a function.

Multiprocessing Can't pickle local object - Python Forum

Incase anyone else faces the same error with the same code structure, adding global keyword word to the child function before you define...