Commandline tools do not recursively stage directory listings properly

The bug is kind of complicated so I am including an example workflow I made to replicate this issue. It seems that commandline jobs are not properly recursively staging input directories.

Here is a brief synopsis of the example workflow

Step 1 creates a nested directory structure and returns the topmost parent directory

first_dir
└── second_dir
    └── a.txt

Step 2 receives the parent directory with InitialWorkDirRequirement and tries to read a.txt

Outputs

Cwltool

cwltool --version
venv/bin//cwltool 3.0.20200324120055

cwltool workflow.cwl inputs.yaml
INFO venv/bin//cwltool 3.0.20200324120055
INFO Resolved 'workflow.cwl' to 'file:///Users/kumarn1/work/toil_fixes/workflow.cwl'
INFO [workflow ] start
INFO [workflow ] starting step create_dir
INFO [step create_dir] start
INFO [job create_dir] /private/tmp/docker_tmp32fswl1f$ create_dir_file.sh \
    another_dir
INFO [job create_dir] completed success
INFO [step create_dir] completed success
INFO [workflow ] starting step read_dir
INFO [step read_dir] start
INFO [job read_dir] /private/tmp/docker_tmpalts74hf$ read_dir.sh \
    another_dir > /private/tmp/docker_tmpalts74hf/out.txt
INFO [job read_dir] completed success
INFO [step read_dir] completed success
INFO [workflow ] completed success
{
    "output": {
        "location": "file:///Users/kumarn1/work/toil_fixes/out.txt",
        "basename": "out.txt",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "path": "/Users/kumarn1/work/toil_fixes/out.txt"
    }
}
INFO Final process status is success

toil-cwl-runner

toil-cwl-runner --version
4.1.0

cwltool workflow.cwl inputs.yaml
INFO venv/bin//cwltool 3.0.20200324120055
INFO Resolved 'workflow.cwl' to 'file:///Users/kumarn1/work/toil_fixes/workflow.cwl'
INFO [workflow ] start
	  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
	    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
	  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
	    returnValues = self._run(jobGraph, fileStore)
	  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
	    return self.run(fileStore)
	  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/cwl/cwltoil.py", line 937, in run
	    raise cwltool.errors.WorkflowException(status)
	cwltool.errors.WorkflowException: permanentFail
	ERROR:toil.worker:Exiting the worker because of a failed job on host lski2325
	WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///Users/kumarn1/work/toil_fixes/read_dir.cwl' read_dir.sh kind-file_Users_kumarn1_work_toil_fixes_read_dir.cwl/instance-daql58np with ID kind-file_Users_kumarn1_work_toil_fixes_read_dir.cwl/instance-daql58np to 0
<=========
lski2325 2020-06-19 17:49:26,953 MainThread WARNING toil.leader: Job 'file:///Users/kumarn1/work/toil_fixes/read_dir.cwl' read_dir.sh kind-file_Users_kumarn1_work_toil_fixes_read_dir.cwl/instance-daql58np with ID kind-file_Users_kumarn1_work_toil_fixes_read_dir.cwl/instance-daql58np is completely failed
lski2325 2020-06-19 17:49:34,437 MainThread INFO toil.leader: Finished toil run with 3 failed jobs.
lski2325 2020-06-19 17:49:34,437 MainThread INFO toil.leader: Failed jobs at end of the run: 'CWLWorkflow' kind-CWLWorkflow/instance-bv8j8v20 'file:///Users/kumarn1/work/toil_fixes/read_dir.cwl' read_dir.sh kind-file_Users_kumarn1_work_toil_fixes_read_dir.cwl/instance-daql58np 'file:///Users/kumarn1/work/toil_fixes/create_dir.cwl' create_dir_file.sh kind-file_Users_kumarn1_work_toil_fixes_create_dir.cwl/instance-xia3pni5

Workflow Progress 100%|██| 4/4 (0 failures) [00:09<00:00, 0.46 jobs/s]
Traceback (most recent call last):
  File "venv/bin//toil-cwl-runner", line 10, in <module>
    sys.exit(main())
  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/cwl/cwltoil.py", line 1691, in main
    outobj = toil.start(wf1)
  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/common.py", line 829, in start
    return self._runMainLoop(rootJobGraph)
  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/common.py", line 1120, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/leader.py", line 269, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/var/folders/q8/qrysvbjj45g7cnbfnltx_2twx2z3v7/T/tmp0fb7io_9' contains 3 failed jobs: 'CWLWorkflow' kind-CWLWorkflow/instance-bv8j8v20, 'file:///Users/kumarn1/work/toil_fixes/read_dir.cwl' read_dir.sh kind-file_Users_kumarn1_work_toil_fixes_read_dir.cwl/instance-daql58np, 'file:///Users/kumarn1/work/toil_fixes/create_dir.cwl' create_dir_file.sh kind-file_Users_kumarn1_work_toil_fixes_create_dir.cwl/instance-xia3pni5
Log from job 'file:///Users/kumarn1/work/toil_fixes/read_dir.cwl' read_dir.sh kind-file_Users_kumarn1_work_toil_fixes_read_dir.cwl/instance-daql58np follows:
=========>
	INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
	INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
	WARNING:toil.fileStores.nonCachingFileStore:Starting job files/for-job/kind-CWLWorkflow/instance-bv8j8v20/cleanup/file-54ammrei/stream with less than 10% of disk space remaining.
	[job read_dir.cwl] /private/var/folders/q8/qrysvbjj45g7cnbfnltx_2twx2z3v7/T/node-09a673ed-519e-4663-bd54-86e679f6b8ca-26824500935055/tmpi0fusy91/86fd0712-e57c-4880-b9a7-394919e41793/tp5rtm_hx/tmp-outk6i0iz85$ read_dir.sh \
	    another_dir > /private/var/folders/q8/qrysvbjj45g7cnbfnltx_2twx2z3v7/T/node-09a673ed-519e-4663-bd54-86e679f6b8ca-26824500935055/tmpi0fusy91/86fd0712-e57c-4880-b9a7-394919e41793/tp5rtm_hx/tmp-outk6i0iz85/out.txt
	INFO:cwltool:[job read_dir.cwl] /private/var/folders/q8/qrysvbjj45g7cnbfnltx_2twx2z3v7/T/node-09a673ed-519e-4663-bd54-86e679f6b8ca-26824500935055/tmpi0fusy91/86fd0712-e57c-4880-b9a7-394919e41793/tp5rtm_hx/tmp-outk6i0iz85$ read_dir.sh \
	    another_dir > /private/var/folders/q8/qrysvbjj45g7cnbfnltx_2twx2z3v7/T/node-09a673ed-519e-4663-bd54-86e679f6b8ca-26824500935055/tmpi0fusy91/86fd0712-e57c-4880-b9a7-394919e41793/tp5rtm_hx/tmp-outk6i0iz85/out.txt
	ls: another_dir/a.txt: No such file or directory
	[job read_dir.cwl] completed permanentFail
	WARNING:cwltool:[job read_dir.cwl] completed permanentFail
	Traceback (most recent call last):
	  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
	    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
	  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
	    returnValues = self._run(jobGraph, fileStore)
	  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
	    return self.run(fileStore)
	  File "/Users/kumarn1/work/toil_fixes/venv/lib/python3.7/site-packages/toil/cwl/cwltoil.py", line 937, in run
	    raise cwltool.errors.WorkflowException(status)
	cwltool.errors.WorkflowException: permanentFail
	ERROR:toil.worker:Exiting the worker because of a failed job on host lski2325
	WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///Users/kumarn1/work/toil_fixes/read_dir.cwl' read_dir.sh kind-file_Users_kumarn1_work_toil_fixes_read_dir.cwl/instance-daql58np with ID kind-file_Users_kumarn1_work_toil_fixes_read_dir.cwl/instance-daql58np to 0
<=========

toil fails with:

ls: another_dir/a.txt: No such file or directory

ExampleWorkflow.zip

┆Issue is synchronized with this Jira Task ┆Issue Number: TOIL-572

[nikhil]

I just tested it out. Yes! It works!!

[mr-c]

Hello @nikhil ! Perhaps https://github.com/DataBiosphere/toil/pull/3058 has solved your issue.

Do you know how to install the latest development version of toil-cwl-runner from GitHub?