Latest toil no longer works with cactus

A cactus command that works in version 3.20 now gives this error with master (I’m using a newer toil to test kubernetes for cactus). It’s strange because Job._runner() has a defer argument in the source:

INFO:toil.worker:Redirecting logging to /data1/cactus/toil-6141c5d4-202b-4550-a160-4d43177be980-565d6e0ff41b452899bd1cc744672e86/tmpj9fnaA/worker_log.txt
Job ended: 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instanceICGaXj
The job seems to have left a log file, indicating failure: 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instanceICGaXj
INFO:toil:Running Toil version 3.21.0a1-81a543d0e1a8e6c299f22bb8e862d34097e1f0bc.
WARNING:toil.resource:'JTRES_612c0b9862883c5800042ea7ac702778' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
Traceback (most recent call last):
  File "/data1/cactus/toil-venv-master/local/lib/python2.7/site-packages/toil/worker.py", line 366, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
TypeError: _runner() got an unexpected keyword argument 'defer'
ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-26-21

This can be reproduced in a minute with these commands:

deactivate
git clone https://github.com/DataBiosphere/toil.git toil.master
git clone https://github.com/ComparativeGenomicsToolkit/cactus.git --recursive
cd cactus
virtualenv toil-venv-master
. toil-venv-master/bin/activate
pip install ../toil.master[all]
pip install .
toil clean jobStore ; cactus jobStore examples/evolverMammals.txt examples/evolverMammals.hal  --binariesMode docker --batchSystem singleMachine --workDir .

Doing the same thing with the last release runs okay

# Works with toil 3.20
deactivate
git clone https://github.com/DataBiosphere/toil.git toil.320
pushd toil.320 ;  git checkout cf34ca3416697f2abc816b2538f20ee29ba16932 ; popd
git clone https://github.com/ComparativeGenomicsToolkit/cactus.git --recursive
cd cactus
virtualenv toil-venv-320
. toil-venv-320/bin/activate
pip install ../toil.320[all]
pip install .
toil clean jobStore ; cactus jobStore examples/evolverMammals.txt examples/evolverMammals.hal  --binariesMode docker --batchSystem singleMachine --workDir .

┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-451

[adamnovak]

@glennhickey I made you another Docker: quay.io/adamnovak/toil:3.22.0a1-72750c175eb84fde0e6d3302283f032edc39038b

Does that one work for you? It stops warning and just gets the size from the job store if it needs it and doesn’t have it. So you don’t need to pack/unpack anything if you don’t want to. This should let the caching file store work properly.

[Secretloong]

@glennhickey perfect thanks, Toil 3.20.0 works very well.

@adamnovak the data is the example in Cactus software, and the command:

cactus --binariesMode local --logFile test.log test_workplace/  evolverMammals.txt test.hal --disableCaching --disableHotDeployment --maxCores 4 --stats --workDir temp/

the new toil doesn’t work with cactus in AWS too, I’m not sure is this solved now. I attach the relevant issue here #https://github.com/ComparativeGenomicsToolkit/cactus/issues/100#issuecomment-554526292

Thanks for all of you.