cwltool uses both default and job_order paths

Long story short:

It appears as though cwltool validates that both the default and job_order paths exist. Ideally, it would only validate the job_order paths if provided, the default paths if not provided.

Long story:

A toy command and file tree to demonstrate this follows Required files: https://gist.github.com/denis-yuen/a1d36345ecc4bdc1580a

.
├── 8e888694-9c56-4529-a750-d6bfbd4a74e7.txt -> inputs/0bf6ab9f-83a8-4b72-ad67-460eb696bd64/ref_file_2
├── b4cdad91-676a-446c-a635-57453f17617a.txt -> inputs/d2816fb8-7773-4b2a-bd31-ee4a7efa8e04/ref_file_1
├── configs
├── foo.json
├── image-descriptor.cwl
├── inputs
│   ├── 0bf6ab9f-83a8-4b72-ad67-460eb696bd64
│   │   └── ref_file_2
│   ├── 0e129840-fa77-4cec-a885-19b1e538507a
│   │   └── hello_input
│   └── d2816fb8-7773-4b2a-bd31-ee4a7efa8e04
│       └── ref_file_1
├── logs
├── node-engine.cwl
├── outputs
│   └── hello-output.txt
└── working

When I run the following command, it executes successfully. ( cwltool --outdir /home/dyuen/consonance/consonance-arch/./datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/outputs/ image-descriptor.cwl foo.json )

However, note the symbolic links in the root. These correspond to the default paths in image-descriptor.cwl which should be overridden by the entries in foo.json.

But when I delete those two links, cwltool dies with the following validation error.

/usr/local/bin/cwltool 1.0.20151013173827
Tool definition failed validation:
While checking field `inputs`
  While checking object `file:///home/dyuen/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/image-descriptor.cwl#ref_file_1`
    While checking field `default`
      Field `path` contains undefined reference to `file:///home/dyuen/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/8e888694-9c56-4529-a750-d6bfbd4a74e7.txt`
  While checking object `file:///home/dyuen/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/image-descriptor.cwl#ref_file_2`
    While checking field `default`
      Field `path` contains undefined reference to `file:///home/dyuen/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/b4cdad91-676a-446c-a635-57453f17617a.txt`

Note that its validating the default paths, not the paths provided in foo.json. However, if I move the files referenced in the foo.json to match the defaults.

.
├── 8e888694-9c56-4529-a750-d6bfbd4a74e7.txt
├── b4cdad91-676a-446c-a635-57453f17617a.txt
├── configs
├── foo.json
├── hello-output.txt
├── image-descriptor.cwl
├── inputs
│   └── 0e129840-fa77-4cec-a885-19b1e538507a
│       └── hello_input
├── logs
├── node-engine.cwl
├── outputs
└── working

Then it dies saying it can’t find the paths in the foo.json

dyuen@odl-dyuen:~/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995$ cwltool --outdir /home/dyuen/consonance/consonance-arch/./datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/outputs/ image-descriptor.cwl foo.json
/usr/local/bin/cwltool 1.0.20151013173827
Unknown hint file:///home/dyuen/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/ResourceRequirement
Got workflow error
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cwltool/main.py", line 153, in single_job_executor
    for r in jobiter:
  File "/usr/local/lib/python2.7/dist-packages/cwltool/draft2tool.py", line 127, in job
    builder.pathmapper = self.makePathMapper(reffiles, input_basedir, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cwltool/draft2tool.py", line 62, in makePathMapper
    return DockerPathMapper(reffiles, input_basedir)
  File "/usr/local/lib/python2.7/dist-packages/cwltool/pathmapper.py", line 73, in __init__
    st = os.lstat(deref)
OSError: [Errno 2] No such file or directory: '/home/dyuen/consonance/consonance-arch/./datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/inputs/0bf6ab9f-83a8-4b72-ad67-460eb696bd64/ref_file_2'
Workflow error:
  [Errno 2] No such file or directory: '/home/dyuen/consonance/consonance-arch/./datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/inputs/0bf6ab9f-83a8-4b72-ad67-460eb696bd64/ref_file_2'

[mr-c]

Fixed, thanks all!

[kapilkd13]

Hi, Actually when i saw draft-2 I created my own code to replicate this problem, removing docker,requirements,hints to make it more simpler.😃 here I made a PR to solve this issue. https://github.com/common-workflow-language/cwltool/pull/320