Intermediary outputs are not deleted after workflow completion

Expected Behavior

1. Intermediary outputs should be deleted after workflow completion
2. Intermediary outputs should NOT be deleted after workflow completion if option --leave-outputs is set.

Actual Behavior

Intermediary outputs are not deleted after workflow completion

Workflow Code

cwlVersion: v1.0
$graph:
- class: Workflow
  id: main
  inputs:
    url:
      type: string
  outputs:
  - id: results
    outputSource:
    - node_2/results
    type: File
  steps:
    node_1:
      in:
        url: url
      out:
      - staged
      run: "#curl"
    node_2:
      in: 
        staged: 
          source: node_1/staged
      out:
      - results
      run: "#ls"

- class: CommandLineTool
  id: curl 
  requirements:
    DockerRequirement:
      dockerPull: curlimages/curl
  baseCommand: [curl]
  arguments: 
  - $(inputs.url)
  - -o 
  - afile.jpg
  inputs:
    url: 
      type: string
  outputs:
    staged:
      outputBinding:
        glob: "*.jpg"
      type: File

- class: CommandLineTool
  id: ls
  baseCommand: [ls]
  arguments: 
  - $(inputs.staged)
  inputs:
    staged: 
      type: File
  stdout: message
  outputs:
    results:
      outputBinding:
        glob: message
      type: File

run with:

cwltool wf.cwl --url https://en.wikipedia.org/wiki/Main_Page#/media/File:Andries-van-Eertvelt-The-battle-in-the-strait-between-Calais-and-Dover-on-3-4-October-1602-between-the-Spanish-galleys-of-Federico-Spinola-and-Dutch-and-English-warships.jpg

Full Traceback

 cwltool pippo.cwl --url https://en.wikipedia.org/wiki/Main_Page#/media/File:Andries-van-Eertvelt-The-battle-in-the-strait-between-Calais-and-Dover-on-3-4-October-1602-between-the-Spanish-galleys-of-Federico-Spinola-and-Dutch-and-English-warships.jpg
INFO /srv/conda/bin/cwltool 3.1.20210803132435
INFO Resolved 'pippo.cwl' to 'file:///home/fbrito/work/cpe-list-hasard/pippo.cwl'
INFO [workflow ] start
INFO [workflow ] starting step node_1
INFO [step node_1] start
INFO [job node_1] /tmp/_f1ozy3h$ docker \
    run \
    -i \
    --mount=type=bind,source=/tmp/_f1ozy3h,target=/imYVLS \
    --mount=type=bind,source=/tmp/9qbd_fmv,target=/tmp \
    --workdir=/imYVLS \
    --read-only=true \
    --user=1000:1000 \
    --rm \
    --cidfile=/tmp/nxgsqwpx/20211004174549-015475.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/imYVLS \
    curlimages/curl \
    curl \
    https://en.wikipedia.org/wiki/Main_Page#/media/File:Andries-van-Eertvelt-The-battle-in-the-strait-between-Calais-and-Dover-on-3-4-October-1602-between-the-Spanish-galleys-of-Federico-Spinola-and-Dutch-and-English-warships.jpg \
    -o \
    afile.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 88702  100 88702    0     0   447k      0 --:--:-- --:--:-- --:--:--  448k
INFO [job node_1] Max memory used: 0MiB
INFO [job node_1] completed success
INFO [step node_1] completed success
INFO [workflow ] starting step node_2
INFO [step node_2] start
INFO [job node_2] /tmp/78s3x6nt$ ls \
    /tmp/3ai8qw41/stged5ea0dc-4205-471b-b11e-122cbd51897e/afile.jpg > /tmp/78s3x6nt/message
INFO [job node_2] completed success
INFO [step node_2] completed success
INFO [workflow ] completed success
{
    "results": {
        "location": "file:///home/fbrito/work/cpe-list-hasard/message",
        "basename": "message",
        "class": "File",
        "checksum": "sha1$acf644d03bf54cd2297177f83c04209704eec2d8",
        "size": 64,
        "path": "/home/fbrito/work/cpe-list-hasard/message"
    }
}
INFO Final process status is success
(base) fbrito@fbrito-ThinkPad-P15-Gen-1:~/work/cpe-list-hasard$ ll /tmp/_f1ozy3h
total 352
drwx------   2 fbrito fbrito   4096 ott  4 17:45 ./
drwxrwxrwt 574 root   root   262144 ott  4 17:46 ../
-rw-r--r--   1 fbrito fbrito  88702 ott  4 17:45 afile.jpg

Your Environment

• cwltool version:

cwltool --version
/srv/conda/bin/cwltool 3.1.20210803132435

[mr-c]

cwltool 3.1.20211004060744 has been released with this fix https://github.com/common-workflow-language/cwltool/releases/tag/3.1.20211004060744 https://pypi.org/project/cwltool/3.1.20211004060744/

[mr-c]

@kinow Ah, then a new test isn’t needed; I’ll make a new release of cwltool. Thanks again!