failure running cwl workflow referencing external file

This is the stacktrace:

Traceback (most recent call last):
  File "/home/thomas.e/.local/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/thomas.e/.local/lib/python2.7/site-packages/toil/job.py", line 1289, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/thomas.e/.local/lib/python2.7/site-packages/toil/job.py", line 1234, in _run
    return self.run(fileStore)
  File "/home/thomas.e/.local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 301, in run
    existing=existing))
  File "/home/thomas.e/.local/lib/python2.7/site-packages/cwltool/process.py", line 183, in adjustFilesWithSecondary
    adjustFilesWithSecondary(rec[d], op)
  File "/home/thomas.e/.local/lib/python2.7/site-packages/cwltool/process.py", line 183, in adjustFilesWithSecondary
    adjustFilesWithSecondary(rec[d], op)
  File "/home/thomas.e/.local/lib/python2.7/site-packages/cwltool/process.py", line 177, in adjustFilesWithSecondary
    rec["path"] = op(rec["path"], primary=primary)
  File "/home/thomas.e/.local/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 146, in getFile
    fileStoreID, fileName = fileTuple
ValueError: too many values to unpack

This is the failing workflow step:

 trim:
    run: tools/trimmomatic.cwl

    in:
      reads1:
        source: read1
        valueFrom: >
          ${
              self.format = "http://edamontology.org/format_1930";
              return self;
          }
      reads2:
        source: read2
        valueFrom: >
          ${
              self.format = "http://edamontology.org/format_1930";
              return self;
          }
      end_mode:
        default: PE
      nthreads:
        valueFrom: ${ return 2; }
      illuminaClip:
        default:
          adapters:
            class: File
            location: "/stornext/System/data/apps/trimmomatic/trimmomatic-0.36/adapters/TruSeq3-PE.fa"
            path: "/stornext/System/data/apps/trimmomatic/trimmomatic-0.36/adapters/TruSeq3-PE.fa"
          seedMismatches: 1
          palindromeClipThreshold: 20
          simpleClipThreshold: 20
          minAdapterLength: 4
          keepBothReads: "true"

    out: [output_log, reads1_trimmed, reads1_trimmed_unpaired, reads2_trimmed_paired, reads2_trimmed_unpaired]

Using print statements I know it is failing on the adapters file - there is no fileStoreId for this file because it is not in the filestore.

The workflow works with the CWL reference implementation.

[evan-wehi]

@mr-c That is a far simpler solution and possibly even consistent with my slowly developing understanding of how CWL works. It also works with both the reference implementation and toil.

Thanks!

[mr-c]

@evan-wehi I am truly sorry to hear that you’ve had such a frustrating time with CWL, the CWL reference implementation, and Toil.

The only difference in your JS is adapters": inputs.adapters, vs "adapters": inputs.illumima_adapters,, this suggests that there were other differences in your workflows.

I forget, did we try putting the JS expression in the valueFrom field? That works for me locally, I think this would be the equivalent code for you:

 trim:
    run: tools/trimmomatic.cwl

    in:
      reads1:
        source: read1
        valueFrom: >
          ${
              self.format = "http://edamontology.org/format_1930";
              return self;
          }
      reads2:
        source: read2
        valueFrom: >
          ${
              self.format = "http://edamontology.org/format_1930";
              return self;
          }
      end_mode:
        default: PE
      nthreads:
        valueFrom: ${ return 2; }
      illuminaClip:
        source: adaptersx
        valueFrom: |
          ${ return {
               "adapters": self,
               "seedMismatches": 1,
               "palindromeClipThreshold": 20,
               "simpleClipThreshold": 20,
               "minAdapterLength": 4,
               "keepBothReads": true };
           }

    out: [output_log, reads1_trimmed, reads1_trimmed_unpaired, reads2_trimmed_paired, reads2_trimmed_unpaired]

You’ll need to make sure the Workflow level requirements has something like this

requirements:
  SchemaDefRequirement:
    types:
      - $import: trimmomatic-illumina_clipping.yaml