selecting a file/dir from directory listing with symlinking

Expected Behavior

In an ExpressionTool and when selecting a file or subdirectory from an input directory, I want this file not to be copied but rather symlinked.

Is there a way to do that ?

This feature would be really useful when files are big. Especially when, in a workflow the expressionTool output is an intermediary file which is not kept in final output.

Actual Behavior

I did not find any way to implement the behavior I am looking for. The selected file is copied in my workflows. Below an example of an expressionTool to select a file from a directory.

Workflow Code

cwlVersion: v1.0
class: ExpressionTool
id: fileFromDir
requirements:
  InlineJavascriptRequirement: {}
inputs:
  dir:
    type: Directory
  prefix:
    type: string
    default: ''
  basename:
    type: string
  suffix:
    type: string
outputs:
  xfile:
    type: 'File'
expression: >
  ${
    var targetFileName=inputs.prefix+inputs.basename+inputs.suffix;
    for (var i in inputs.dir.listing){
      var itemFile=inputs.dir.listing[i];
      if (itemFile.basename == targetFileName){
        var targetFile=itemFile;
        break;
      }
    }
    return {'xfile':targetFile};
  }

Your Environment

• cwltool version: 3.0.20200709181526

[jfouret]

Thank you very much for your reactivity.

My bad, everything is fine with this implementation in cwltool.

Since the problem I had related to #888 I have temporarely replaced cwltool cmdline with toil-cwl-runner.

Using cwltool and while keeping temporary file we ccan see that everything is fine:

$cwltool --parallel --force-docker-pull --verbose --no-match-user --on-error 'continue' --tmpdir-prefix tmp --cachedir cache --leave-tmpdir --timestamps --outdir outdir file-picker-wf.cwl --my_dir test_dir

[...]

$ tree .
.
├── cache
│   ├── 0e647ac658d6b4319997e3bdbb3e5a33
│   │   └── 413bfea21ae37cc658b007d2fd2b80b4d5dd3db0
│   └── 0e647ac658d6b4319997e3bdbb3e5a33.status
├── file-picker-wf.cwl
├── outdir
│   └── 413bfea21ae37cc658b007d2fd2b80b4d5dd3db0
├── test_dir
│   ├── hello1.txt
│   └── hello2.txt
└── tmp

5 directories, 6 files

The file hello1.txt is not copied; nor in cache nor in tmp.

However the problem stands with toil-cwl-runner:

$toil-cwl-runner --jobStore file:cache/test --strict-memory-limit --force-docker-pull --logDebug --clean never --outdir outdir file-picker-wf.cwl --my_dir test_dir

[...]

$ tree .

.
├── cache
│   └── test
│       ├── files
│       │   ├── for-job
│       │   │   ├── kind-CWLWorkflow
│       │   │   │   └── instance-f0yieq5v
│       │   │   └── kind-file_home_users_jfouret_toil_test_file-picker-wf.cwl_first_20f90064-e089-49f8-b671-2b61809f0bf9
│       │   │       └── instance-wqh50svk
│       │   │           ├── file-c7rkl6um
│       │   │           │   └── hello1.txt
│       │   │           └── file-zktxmvep
│       │   │               └── 65588ba8669eb87edeb7befb2706bef493704366 -> /home/users/jfouret/toil_test/outdir/65588ba8669eb87edeb7befb2706bef493704366
│       │   ├── no-job
│       │   │   ├── file-3l6j87kh
│       │   │   ├── file-d197eyo7
│       │   │   │   └── stream
│       │   │   ├── file-qyhetvsm
│       │   │   ├── file-sc4qb0xf
│       │   │   │   └── stream
│       │   │   └── file-z9hgcs_j
│       │   └── shared
│       │       ├── config.pickle
│       │       ├── environment.pickle
│       │       ├── pid.log
│       │       ├── rootJobReturnValue
│       │       ├── rootJobStoreID
│       │       └── succeeded.log
│       ├── jobs
│       │   ├── kind-CWLWorkflow
│       │   ├── kind-ResolveIndirect
│       │   ├── kind-file_home_users_jfouret_toil_test_file-picker-wf.cwl_first_20f90064-e089-49f8-b671-2b61809f0bf9
│       │   └── kind-file_home_users_jfouret_toil_test_file-picker-wf.cwl_second_50e162d9-d940-426c-b499-3f1ab7e8218b
│       └── stats
│           ├── stats8p8nzd5o.new
│           ├── stats8pz85nbi.new
│           ├── statsacbvpvk8.new
│           ├── statsnjpn2rro
│           └── statsvrbu1oen.new
├── file-picker-wf.cwl
├── outdir
│   ├── 2093ea3651180b15e0b70374fab093583577c459
│   └── 65588ba8669eb87edeb7befb2706bef493704366
├── test_dir
│   ├── hello1.txt
│   └── hello2.txt
└── tmp

26 directories, 20 files

The problem is that the hello1.txt file is copied in the temporary directory (jobstore) cache/test/files/for-job/kind-file_home_users_jfouret_toil_test_file-picker-wf.cwl_first_20f90064-e089-49f8-b671-2b61809f0bf9/instance-wqh50svk/file-c7rkl6um/hello1.txt rather than being symlinked.

I did not see that #888 had been fixed. I will try it.

I am going to create an issue for toil and reference this post.

Thanks,

[mr-c]

Thank you @jfouret for the update and your persistence! I’ll close this issue for now. I hope the fix for #888 works for you 😃