Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

cwltool fails for secondary files stored on S3 jobstore

See original GitHub issue

CWL workflows that contain a secondary file that is stored on the S3 jobstore fail with the new cwltool version with the exception below:

traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/cwltool/executors.py", line 233, in run_jobs
    for job in jobiter:
  File "/usr/local/lib/python3.6/dist-packages/cwltool/command_line_tool.py", line 591, in job
    builder = self._init_job(job_order, runtimeContext)
  File "/usr/local/lib/python3.6/dist-packages/cwltool/process.py", line 893, in _init_job
    discover_secondaryFiles=getdefault(runtime_context.toplevel, False),
  File "/usr/local/lib/python3.6/dist-packages/cwltool/builder.py", line 343, in bind_input
    discover_secondaryFiles=discover_secondaryFiles,
  File "/usr/local/lib/python3.6/dist-packages/cwltool/builder.py", line 410, in bind_input
    0 : cast(str, datum["location"]).rindex("/") + 1
ValueError: substring not found

The reason is that the location for S3 Toil objects is actually like "location": "toilfs:17008:0b6b9de2-ef48-4bce-834f-fa94a1cd068e",, which doesn’t contain a /.

@mr-c this looks like a CWL bug rather than a Toil bug, but I figured to file it here first since apparently it only happens on Toil. Please file another issue on the CWL side if desired. Thanks!

P.S. This didn’t happen in previous CWL versions as that code only ran if basename didn’t exist, which in Toil, it always exists.