Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Partial build fails - TypeError: cannot pickle 'module' object

See original GitHub issue

I see the following error when I do ploomber build -p train, but do not get the error when running ploomber build. My guess it that it relates to a joblib.dump. I tried to add custom un/serializers after seeing the error, but that did not fix it. Sorry, the pipeline is a bit complicated at this point and the error is hard to generate a easy to reproduce example for testing.

Traceback (most recent call last):
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/site-packages/ploomber/cli/io.py", line 34, in wrapper
    fn(**kwargs)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/site-packages/ploomber/cli/build.py", line 49, in main
    report = dag.build_partially(args.partially,
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/site-packages/ploomber/dag/dag.py", line 667, in build_partially
    dag_copy = deepcopy(self)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/hopkir/anaconda3/envs/hydrates/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'module' object

Issue Analytics

State:
Created 2 years ago
Comments:18 (10 by maintainers)

Top GitHub Comments

1reaction

edublancascommented, May 8, 2022

the conclusion from here is that it’s the user’s implementation what’s causing the non-pickable error. closing it for now, but feel free to re-open it if you need more help!

0reactions

edublancascommented, Oct 31, 2022

alright. I’ll merge this to the master branch. it’ll be part of our next release

Top Results From Across the Web

Python multiprocessing PicklingError: Can't pickle <type ...

Here is a list of what can be pickled. In particular, functions are only picklable if they are defined at the top-level of...

Cant't pickle module object · Issue #1433 · bentoml ... - GitHub

I have a model stored inside a module (not a class). ... line 496, in save rv = reduce(self.proto) TypeError: can't pickle module...

Unable to pickle certain QuerySet objects due to use of curry ...

QuerySet objects containing Model objects that have a RelatedField attribute are not pickle-able due to the use of the curry function in RelatedField....

cant pickle local object | The search engine you control.

python multiprocessing : AttributeError: Can't pickle local object ... prediction, i) error = np.mean(pool.map(functools.partial(error_fun, x, prediction), ...

dill module documentation — dill 0.3.7.dev0 documentation

If byref=True, then dill behaves a lot more like pickle as certain objects (like modules) are pickled by reference as opposed to attempting...