Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

HTTP imports - import behaviour unclear

See original GitHub issue

Expected Behavior

I’m just trying to test out HTTP(S) imports as mentioned in https://www.commonwl.org/v1.0/SchemaSalad.html#Import I’m expecting cwltool --validate to say it’s valid Here’s my process for trying it out:

FROM python:3.9.7

RUN pip3 install cwltool==3.1.20210816212154
RUN git clone https://github.com/rabix/sbpack.git
RUN cwltool --validate sbpack/tests/workflows/wf5.cwl
#RUN sed -i 's/$import: ..\/types\/recursive.yml/$import: https:\/\/raw.githubusercontent.com\/rabix\/sbpack\/master\/tests\/types\/recursive.yml/g' sbpack/tests/workflows/wf5.cwl
RUN sed -i 's/$import: ..\/types\/recursive.yml/$import: "https:\/\/raw.githubusercontent.com\/rabix\/sbpack\/master\/tests\/types\/recursive.yml"/g' sbpack/tests/workflows/wf5.cwl
RUN cwltool --validate sbpack/tests/workflows/wf5.cwl

Actual Behavior

$ docker build .
[+] Building 3.7s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                    0.1s
 => => transferring dockerfile: 628B                                                                                                                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                         0.0s
 => [internal] load metadata for docker.io/library/python:3.9.7                                                                                                                                                                                                                                                         1.0s
 => [1/6] FROM docker.io/library/python:3.9.7@sha256:46a1f3fd3d713fc9151753a6a75da327d1c872152ace16e8b3225458f8754d07                                                                                                                                                                                                   0.0s
 => CACHED [2/6] RUN pip3 install cwltool==3.1.20210816212154                                                                                                                                                                                                                                                           0.0s
 => CACHED [3/6] RUN git clone https://github.com/rabix/sbpack.git                                                                                                                                                                                                                                                      0.0s
 => CACHED [4/6] RUN cwltool --validate sbpack/tests/workflows/wf5.cwl                                                                                                                                                                                                                                                  0.0s
 => [5/6] RUN sed -i 's/$import: ..\/types\/recursive.yml/$import: "https:\/\/raw.githubusercontent.com\/rabix\/sbpack\/master\/tests\/types\/recursive.yml"/g' sbpack/tests/workflows/wf5.cwl                                                                                                                          0.5s
 => ERROR [6/6] RUN cwltool --validate sbpack/tests/workflows/wf5.cwl                                                                                                                                                                                                                                                   2.2s
------
 > [6/6] RUN cwltool --validate sbpack/tests/workflows/wf5.cwl:
#9 0.999 INFO /usr/local/bin/cwltool 3.1.20210816212154
#9 1.001 INFO Resolved 'sbpack/tests/workflows/wf5.cwl' to 'file:///sbpack/tests/workflows/wf5.cwl'
#9 2.057 ERROR Tool definition failed validation:
#9 2.057 sbpack/tests/workflows/wf5.cwl:5:1:  checking field `inputs`
#9 2.057 sbpack/tests/workflows/wf5.cwl:6:3:    checking object `sbpack/tests/workflows/wf5.cwl#in1`
#9 2.057 sbpack/tests/workflows/wf5.cwl:7:5:      Field `type` references unknown identifier
#9 2.057                                          `file:///sbpack/tests/types/recursive.yml#file_with_sample_meta`,
#9 2.057                                          tried
#9 2.057                                          file:///sbpack/tests/workflows/wf5.cwl#file:///sbpack/tests/types/recursive.yml#file_with_sample_meta
#9 2.057 sbpack/tests/workflows/wf5.cwl:12:3:   checking object `sbpack/tests/workflows/wf5.cwl#in4`
#9 2.057                                          Field `type` references unknown identifier
#9 2.057                                          `file:///sbpack/tests/types/recursive.yml#sample_meta`, tried
#9 2.057                                          file:///sbpack/tests/workflows/wf5.cwl#file:///sbpack/tests/types/recursive.yml#sample_meta
------
executor failed running [/bin/sh -c cwltool --validate sbpack/tests/workflows/wf5.cwl]: exit code: 1

Workflow Code

https://github.com/rabix/sbpack/blob/master/tests/workflows/wf5.cwl

Full Traceback

 > [6/6] RUN cwltool --validate sbpack/tests/workflows/wf5.cwl --debug:
#9 0.908 INFO /usr/local/bin/cwltool 3.1.20210816212154
#9 0.910 INFO Resolved 'sbpack/tests/workflows/wf5.cwl' to 'file:///sbpack/tests/workflows/wf5.cwl'
#9 2.090 ERROR Tool definition failed validation:
#9 2.090 sbpack/tests/workflows/wf5.cwl:5:1:  checking field `inputs`
#9 2.090 sbpack/tests/workflows/wf5.cwl:6:3:    checking object `sbpack/tests/workflows/wf5.cwl#in1`
#9 2.090 sbpack/tests/workflows/wf5.cwl:7:5:      Field `type` references unknown identifier
#9 2.090                                          `file:///sbpack/tests/types/recursive.yml#file_with_sample_meta`,
#9 2.090                                          tried
#9 2.090                                          file:///sbpack/tests/workflows/wf5.cwl#file:///sbpack/tests/types/recursive.yml#file_with_sample_meta
#9 2.090 sbpack/tests/workflows/wf5.cwl:12:3:   checking object `sbpack/tests/workflows/wf5.cwl#in4`
#9 2.090                                          Field `type` references unknown identifier
#9 2.090                                          `file:///sbpack/tests/types/recursive.yml#sample_meta`, tried
#9 2.090                                          file:///sbpack/tests/workflows/wf5.cwl#file:///sbpack/tests/types/recursive.yml#sample_meta
------
executor failed running [/bin/sh -c cwltool --validate sbpack/tests/workflows/wf5.cwl --debug]: exit code: 1

Your Environment

cwltool version: 3.1.20210816212154

Issue Analytics

State:
Created 2 years ago
Comments:13 (11 by maintainers)

Top GitHub Comments

2reactions

denis-yuencommented, Sep 27, 2021

@garyluu FYI, confirmed the dependency issue. A simpler example using https://github.com/common-workflow-language/common-workflow-language/blob/main/v1.0/examples/1st-workflow.cwl works as expected

$ git diff
diff --git a/v1.0/examples/1st-workflow.cwl b/v1.0/examples/1st-workflow.cwl
index 11c8e6f..79dc17a 100644
--- a/v1.0/examples/1st-workflow.cwl
+++ b/v1.0/examples/1st-workflow.cwl
@@ -18,7 +18,7 @@ steps:
     out: [example_out]
 
   compile:
-    run: arguments.cwl
+    run: "https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/main/v1.0/examples/arguments.cwl"
     in:
       src: untar/example_out
     out: [classfile]
$ cwltool --validate 1st-workflow.cwl 
INFO <snip>/.local/bin/cwltool 3.0.20200807132242
INFO Resolved '1st-workflow.cwl' to 'file:///<snip>/common-workflow-language/v1.0/examples/1st-workflow.cwl'
1st-workflow.cwl is valid CWL.
$ cwltool --pack 1st-workflow.cwl 
<snip>
$ echo $?
0

1reaction

mr-ccommented, Sep 27, 2021

I did cwltool --pack on the 1st-workflow.cwl with the run: https://github.com/common-workflow-language/common-workflow-language/blob/main/v1.0/examples/1st-workflow.cwl adjustment and it produces a single document with no external dependencies.

{
    "$graph": [
        {
            "class": "Workflow",
            "inputs": [
                {
                    "type": "string",
                    "id": "#main/ex"
                },
                {
                    "type": "File",
                    "id": "#main/inp"
                }
            ],
            "steps": [
                {
                    "run": "#arguments.cwl",
                    "in": [
                        {
                            "source": "#main/untar/example_out",
                            "id": "#main/compile/src"
                        }
                    ],
                    "out": [
                        "#main/compile/classfile"
                    ],
                    "id": "#main/compile"
                },
                {
                    "run": "#tar-param.cwl",
                    "in": [
                        {
                            "source": "#main/ex",
                            "id": "#main/untar/extractfile"
                        },
                        {
                            "source": "#main/inp",
                            "id": "#main/untar/tarfile"
                        }
                    ],
                    "out": [
                        "#main/untar/example_out"
                    ],
                    "id": "#main/untar"
                }
            ],
            "id": "#main",
            "outputs": [
                {
                    "type": "File",
                    "outputSource": "#main/compile/classfile",
                    "id": "#main/classout"
                }
            ]
        },
        {
            "class": "CommandLineTool",
            "baseCommand": [
                "tar",
                "xf"
            ],
            "inputs": [
                {
                    "type": "string",
                    "inputBinding": {
                        "position": 2
                    },
                    "id": "#tar-param.cwl/extractfile"
                },
                {
                    "type": "File",
                    "inputBinding": {
                        "position": 1
                    },
                    "id": "#tar-param.cwl/tarfile"
                }
            ],
            "outputs": [
                {
                    "type": "File",
                    "outputBinding": {
                        "glob": "$(inputs.extractfile)"
                    },
                    "id": "#tar-param.cwl/example_out"
                }
            ],
            "id": "#tar-param.cwl"
        },
        {
            "class": "CommandLineTool",
            "label": "Example trivial wrapper for Java 7 compiler",
            "hints": [
                {
                    "dockerPull": "java:7-jdk",
                    "class": "DockerRequirement"
                }
            ],
            "baseCommand": "javac",
            "arguments": [
                "-d",
                "$(runtime.outdir)"
            ],
            "inputs": [
                {
                    "type": "File",
                    "inputBinding": {
                        "position": 1
                    },
                    "id": "#arguments.cwl/src"
                }
            ],
            "outputs": [
                {
                    "type": "File",
                    "outputBinding": {
                        "glob": "*.class"
                    },
                    "id": "#arguments.cwl/classfile"
                }
            ],
            "id": "#arguments.cwl"
        }
    ],
    "cwlVersion": "v1.0"

Does this satisfy your needs? What could we add to the documentation to explain this to others?

Top Results From Across the Web

Including .css files with @import is non-standard behaviour ...

Use a custom importer to maintain this behaviour. Check your implementations documentation on how to create a custom importer.

Unclear (for Python people) import behavior. And how to deal ...

Then I realize, that importing import ex fixes the error. But I was surprised by this. And only then I realize, that u.uuid...

Python 3.6 project structure leads to RuntimeWarning

Hence the vague this may cause unpredictable behaviour warning - if ... If you import the package from another script .proj is imported...

Traps for the Unwary in Python's Import System

It's specific to the situation where the main module is also imported as an ... while http://bugs.python.org/issu... is a behaviour change in Python...

Supporting transitive imports from private repositories?

I think it would also need to ignore explicit headers in the case where a remote import happens to contain a fully-qualified import...