Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Pickling error from joblib.Parallel()
See original GitHub issueWhen running a block of code from within Parallel(), it returns the error - _pickle.PicklingError: Could not pickle the task to send it to the workers.
Is anyone facing similar issues? Any help appreciated!
Description - Inside the (to be parallelized) function, I am using Spacy’s english language model to extract Named Entities from text. I encounter this error when running the script from command line. If I use interactive program like a jupyter notebook, the call to Parallel() sometimes works fine and sometimes throws the described error, basically not consistent.
The error trace looks like this -
$ python namedAliases_associatedEntities_v6.py
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=-1)]: Batch computation too fast (0.0770s.) Setting batch_size=4.
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/loky/backend/queues.py", line 151, in _feed
obj, reducers=reducers)
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/loky/backend/reduction.py", line 145, in dumps
p.dump(obj)
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 290, in __getstate__
for func, args, kwargs in self.items]
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 290, in <listcomp>
for func, args, kwargs in self.items]
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 278, in _wrap_non_picklable_objects
wrapped_obj = CloudpickledObjectWrapper(obj)
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 208, in __init__
self.pickled_obj = dumps(obj)
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 917, in dumps
cp.dump(obj)
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 268, in dump
return Pickler.dump(self, obj)
File "/usr/lib/python3.5/pickle.py", line 408, in dump
self.save(obj)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 408, in save_function
self.save_function_tuple(obj)
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 573, in save_function_tuple
save(state)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 841, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 408, in save_function
self.save_function_tuple(obj)
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 573, in save_function_tuple
save(state)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 408, in save_function
self.save_function_tuple(obj)
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/cloudpickle/cloudpickle.py", line 573, in save_function_tuple
save(state)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/ubuntu/.local/lib/python3.5/site-packages/dill/_dill.py", line 893, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 841, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 520, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.5/pickle.py", line 592, in save_reduce
"args[0] from __newobj__ args has the wrong class")
_pickle.PicklingError: args[0] from __newobj__ args has the wrong class
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "namedAliases_associatedEntities_v6.py", line 283, in <module>
list_of_named_entities = Parallel(n_jobs=-1, verbose=10, backend = 'loky')(map(delayed(processNER), all_articles))
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 996, in __call__
self.retrieve()
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/parallel.py", line 899, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/_parallel_backends.py", line 517, in wrap_future_result
return future.result(timeout=timeout)
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/loky/_base.py", line 431, in result
return self.__get_result()
File "/home/ubuntu/.local/lib/python3.5/site-packages/joblib/externals/loky/_base.py", line 382, in __get_result
raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.
Issue Analytics
- State:
- Created 5 years ago
- Comments:7 (2 by maintainers)
Top Results From Across the Web
Could not pickle the task to send it to the workers - Stack ...
Same issue. I solved by changing the backend from loky to threading in Parallel .
Read more >Serialization of un-picklable objects - Joblib
First, define functions which cannot be pickled with the standard pickle protocol. They cannot be serialized with pickle because they are defined in...
Read more >When using multiple workers, I get a pickling error. Can I still ...
sbi uses joblib to parallelize simulations, which in turn uses pickle or cloudpickle to serialize the simulator. Almost all simulators will be picklable...
Read more >PicklingError: Could not pickle the task to send it to the workers.
I am using sklearn in a databricks notebook to fit an estimator in parallel. Sklearn uses joblib with loky backend to do this....
Read more >Multiprocessing and Pickle, How to Easily fix that?
However, the multiprocess tasks can't be pickled; it would raise an error failing to pickle. That's because when dividing a single task over ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I’m getting this issue too. Worth noting that I’m in a notebook environment and using autoreload:
The error seems to be thrown up when I update one of my modules in between cell runs.
I have a very similar issue trying to run PyMC3 model training with joblib (and I also had dill in the trace). As dmparker said above, I tried using an old version of joblib (0.12.1) and it works fine.