NetworkAccess requirement not applicable in cwltool:overrides

Expected Behavior

Allow user to specify network access in cwltool:overrides OR explictly error out saying this is not an acceptable override.

Actual Behavior

Tell us what happens instead

We get a warning at the top of the script

overrides.json:3:5: Warning: checking item
overrides.json:4:7: Warning:   checking field `requirements`
overrides.json:5:9: Warning:     checking item
                    Warning:       Field `class` contains undefined reference to
                    `file:///home/alexiswl/170612_A00181_0011_AH2JK7DMXX_testing/NetworkAccess`

Input Json

{
  "bclconvert_run_configurations": [
    {
      "strict_mode": false,
      "output_directory": "170612_A00181_0011_AH2JK7DMXX_converted",
      "samplesheet": {
        "bclconvert_data": [
          {
            "index": "",
            "lane": 1,
            "sample_id": "PhiX"
          }
        ],
        "bclconvert_settings": {
          "override_cycles": "Y4N147;Y4N147"
        },
        "header": {
          "application": "NovaSeq FASTQ Only",
          "assay": "TruSeq HT",
          "chemistry": "Default",
          "date": "6/12/2017",
          "experiment_name": "170612",
          "file_format_version": 2,
          "iem_file_version": 4,
          "index_adapters": "Truseq",
          "instrument_type": "NovaSeq 6000",
          "workflow": "GenerateFASTQ"
        },
        "reads": {
          "read_1_cycles": 151,
          "read_2_cycles": 151
        }
      }
    }
  ],
  "bclconvert_run_input_directory": {
    "class": "Directory",
    "location": "170612_A00181_0011_AH2JK7DMXX/"
  },
  "runfolder_name": "170612_A00181_0011_AH2JK7DMXX",
  "cwltool:overrides": {
    "workflows/bclconvert/4.0.3/bclconvert__4.0.3.cwl#bcl_convert_run_step": {
      "requirements": {
        "NetworkAccess": {
          "networkAccess": true
        }
      }
    }
  }
}

Full Traceback

INFO /home/alexiswl/miniconda3/envs/cwltool-latest/bin/cwltool 3.1.20221018083734
INFO Resolved 'bclconvert-with-qc-pipeline__4.0.3/workflow.cwl' to 'file:///home/alexiswl/170612_A00181_0011_AH2JK7DMXX_testing/bclconvert-with-qc-pipeline__4.0.3/workflow.cwl'
input.json:42:5: Warning: checking item
input.json:43:7: Warning:   checking field `requirements`
input.json:44:9: Warning:     checking item
                 Warning:       Field `class` contains undefined reference to
                 `file:///home/alexiswl/170612_A00181_0011_AH2JK7DMXX_testing/NetworkAccess`
...

Your Environment

• cwltool version: Check using cwltool --version

cwltool 3.1.20221018083734

[mr-c]

Thanks @mr-c

Are there examples in the readme or other docs that need updating?

I think the documentation here does make sense for users who already understand the pattern, however it isn’t easy to take the example used and apply it to a workflow with greater complexity, such as one with subworkflows. The documentation reflects the DAG identification for the latest CWLtool version, for the 2020 version currently used in our production systems I’ve noted this discrepancy in our docs.

Personally, I used the CWL debug logs to help navigate how step IDs are set for use in overrides.

I think a --print-overrides-targets option is needed; users shouldn’t have to dig through the logs

[alexiswl]

Thanks @mr-c

Are there examples in the readme or other docs that need updating?

I think the documentation here does make sense for users who already understand the pattern, however it isn’t easy to take the example used and apply it to a workflow with greater complexity, such as one with subworkflows. The documentation reflects the DAG identification for the latest CWLtool version, for the 2020 version currently used in our production systems I’ve noted this discrepancy in our docs.

Personally, I used the CWL debug logs to help navigate how step IDs are set for use in overrides.