Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Pickling error when running Cactus on larger input

See original GitHub issue

I’m trying to run Cactus on a larger dataset on kubernetes, and this is the latest error PicklingError: Can't pickle <class 'toil.BotoCredentialAdapter'>: it's not found as toil.BotoCredentialAdapter

Is this a Toil problem?

86359de7-b94e-47bb-99e2-016fc4314469    Running command catchsegv 'cactus_blast_chunkSequences' 'INFO' '25000000' '10000' 'tsJ4ChA/tmp_DsPa0hrmcW' 'tmp4bFJbY.tmp'
86359de7-b94e-47bb-99e2-016fc4314469    Set log level to INFO
86359de7-b94e-47bb-99e2-016fc4314469    INFO:toil-rt:2019-12-05 23:45:54.502858: Successfully ran the command: "singularity --silent run -B /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpdvB9f_/9f512bca-9683-4b6d-962a-dd060f2d5983:/mnt --pwd /mnt -u /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpdvB9f_/9f512bca-9683-4b6d-962a-dd060f2d5983/tjMQD3l/cactus.img.sb cactus_blast_chunkSequences INFO 25000000 10000 tsJ4ChA/tmp_DsPa0hrmcW tmp4bFJbY.tmp" in 49.5165989399 seconds
86359de7-b94e-47bb-99e2-016fc4314469    INFO:toil.lib.bioio:Made the list of blasts
86359de7-b94e-47bb-99e2-016fc4314469    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/venv2/local/lib/python2.7/site-packages', name='cactus.blast.blast', fromVirtualEnv=True).
86359de7-b94e-47bb-99e2-016fc4314469    Traceback (most recent call last):
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 367, in workerScript
86359de7-b94e-47bb-99e2-016fc4314469        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
86359de7-b94e-47bb-99e2-016fc4314469      File "/tmp/tmpgbMfBk/3a536290e528005ce049de97a4aca1e6/cactus/shared/common.py", line 1201, in _runner
86359de7-b94e-47bb-99e2-016fc4314469        super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1399, in _runner
86359de7-b94e-47bb-99e2-016fc4314469        self._serialiseExistingJob(jobGraph, jobStore, returnValues)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1308, in _serialiseExistingJob
86359de7-b94e-47bb-99e2-016fc4314469        self._serialiseJobGraph(jobGraph, jobStore, returnValues, False)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1277, in _serialiseJobGraph
86359de7-b94e-47bb-99e2-016fc4314469        job._serialiseJob(jobStore, jobsToJobGraphs, jobGraph)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1155, in _serialiseJob
86359de7-b94e-47bb-99e2-016fc4314469        pickle.dump(self, fileHandle, pickle.HIGHEST_PROTOCOL)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 1376, in dump
86359de7-b94e-47bb-99e2-016fc4314469        Pickler(file, protocol).dump(obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 224, in dump
86359de7-b94e-47bb-99e2-016fc4314469        self.save(obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(state)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 871, in save_module_dict
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_dict(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_setitems(obj.iteritems())
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
86359de7-b94e-47bb-99e2-016fc4314469        save(v)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_appends(iter(obj))
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
86359de7-b94e-47bb-99e2-016fc4314469        save(x)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_appends(iter(obj))
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
86359de7-b94e-47bb-99e2-016fc4314469        save(x)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_appends(iter(obj))
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
86359de7-b94e-47bb-99e2-016fc4314469        save(x)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(state)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 871, in save_module_dict
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_dict(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_setitems(obj.iteritems())
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
86359de7-b94e-47bb-99e2-016fc4314469        save(v)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(state)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 871, in save_module_dict
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_dict(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_setitems(obj.iteritems())
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
86359de7-b94e-47bb-99e2-016fc4314469        save(v)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(state)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 871, in save_module_dict
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_dict(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_setitems(obj.iteritems())
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
86359de7-b94e-47bb-99e2-016fc4314469        save(v)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 425, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(state)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 871, in save_module_dict
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_dict(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 655, in save_dict
86359de7-b94e-47bb-99e2-016fc4314469        self._batch_setitems(obj.iteritems())
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 687, in _batch_setitems
86359de7-b94e-47bb-99e2-016fc4314469        save(v)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 331, in save
86359de7-b94e-47bb-99e2-016fc4314469        self.save_reduce(obj=obj, *rv)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 396, in save_reduce
86359de7-b94e-47bb-99e2-016fc4314469        save(cls)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 286, in save
86359de7-b94e-47bb-99e2-016fc4314469        f(self, obj) # Call unbound method with explicit self
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 1299, in save_type
86359de7-b94e-47bb-99e2-016fc4314469        StockPickler.save_global(pickler, obj)
86359de7-b94e-47bb-99e2-016fc4314469      File "/usr/lib/python2.7/pickle.py", line 754, in save_global
86359de7-b94e-47bb-99e2-016fc4314469        (obj, module, name))
86359de7-b94e-47bb-99e2-016fc4314469    PicklingError: Can't pickle <class 'toil.BotoCredentialAdapter'>: it's not found as toil.BotoCredentialAdapter
86359de7-b94e-47bb-99e2-016fc4314469    ERROR:toil.worker:Exiting the worker because of a failed job on host root-toil-d12ffe0b-ef0d-42cf-90f5-c2b0060670de-3188-s6m95
86359de7-b94e-47bb-99e2-016fc4314469    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'BlastSequencesAgainstEachOther' 86359de7-b94e-47bb-99e2-016fc4314469 with ID 86359de7-b94e-47bb-99e2-016fc4314469 to 2
Issued job 'BlastSequencesAgainstEachOther' 86359de7-b94e-47bb-99e2-016fc4314469 with job batch system ID: 3190 and cores: 1, disk: 85.8 G, and memory: 16.0 G
2019-12-06 00:07:33.322926: Running the command: "singularity --silent run -B /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217:/mnt --pwd /mnt -u /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217/tqINIfd/cactus.img.sb cactus_blast_chunkSequences INFO 25000000 10000 tGGRcR4/tmp_b58alVaJnN tmpdrqoB4.tmp tmpFXOVtZ.tmp"
2019-12-06 00:09:06.558278: Successfully ran the command: "singularity --silent run -B /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217:/mnt --pwd /mnt -u /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217/tqINIfd/cactus.img.sb cactus_blast_chunkSequences INFO 25000000 10000 tGGRcR4/tmp_b58alVaJnN tmpdrqoB4.tmp tmpFXOVtZ.tmp" in 92.7297759056 seconds
2019-12-06 00:09:06.559814: Running the command: "singularity --silent run -B /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217:/mnt --pwd /mnt -u /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217/tqINIfd/cactus.img.sb cactus_blast_chunkSequences INFO 25000000 10000 txNK42h/tmp_Z4yHYyAACP tmpvqeGbA.tmp"
2019-12-06 00:09:54.259745: Successfully ran the command: "singularity --silent run -B /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217:/mnt --pwd /mnt -u /var/lib/toil/toil-632aa838-c670-4e9f-97b3-b45f546568f0-dc95beedd0eeec54a13d84c25de81dc2/tmpqyLu7Z/ee90be22-885b-449b-a269-3457163d1217/tqINIfd/cactus.img.sb cactus_blast_chunkSequences INFO 25000000 10000 txNK42h/tmp_Z4yHYyAACP tmpvqeGbA.tmp" in 47.1937949657 seconds
Reissued any over long jobs

┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-466

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:18 (13 by maintainers)

github_iconTop GitHub Comments

1reaction
glennhickeycommented, Feb 7, 2020

Cactus still only runs with Python 2.7. That should be changing soon though.

0reactions
adamnovakcommented, Feb 12, 2020

Lon and I closed this because Jeffrey solved this actual error. We’ve created a new un-pickling error instead, but it sounds like we’re going to fix that on the Cactus side.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Python multiprocessing PicklingError: Can't pickle <type ...
I looked up some previous notes on this problem. They were all caused by using pool to call function defined within a class...
Read more >
Pickled Cactus Recipe with a Sweet Twist - Homesteading
Step 1: Use paddles that are bright green. The smaller you can get, the more tender, however, larger paddles may be used. ......
Read more >
Southwestern rare and endangered plants: Proceedings of the ...
address the barriers to using native plants. The greatest problem is lack of appropriate material in large enough quantities. In 1999, 2.5 million...
Read more >
Girly Pickle Cookie Cutter - Etsy Denmark
This Cookie Cutters item by SweetleighPrinted has 16 favorites from Etsy shoppers. Ships from United States. Listed on Oct 19, 2022.
Read more >
Tutorial - Sonic Pi
See if you can make it go so fast your computer will stop with an error because it can't keep up (if that...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

linkMerge of cwl has broken

Next page

toil-cwl-runner on slurm fails to run jobs on multiple nodes