attempts to write to /var/lib/cwl with docker hint but no docker installed

On the 2 systems I have available to me (one of them Ubuntu 16), /var/lib is not user writable, so running the conformance test fails:

cwltest --tool cwltool --test=conformance_test_v1.0.yaml --junit-xml result.xml --basedir v1.0
Test [1/82] Got workflow error
Traceback (most recent call last):
  File "/software/[...]/python-2.7.12/lib/python2.7/site-packages/cwltool/main.py", line 223, in single_job_executor
    r.run(**kwargs)
  File "/software/[...]/python-2.7.12/lib/python2.7/site-packages/cwltool/job.py", line 149, in run
    stageFiles(self.pathmapper, os.symlink)
  File "/software/[...]/python-2.7.12/lib/python2.7/site-packages/cwltool/process.py", line 176, in stageFiles
    os.makedirs(os.path.dirname(p.target), 0755)
  File "/software/[...]/python-2.7.12/lib/python2.7/os.py", line 150, in makedirs
     makedirs(head, mode)
  File "/software/[...]/python-2.7.12/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/var/lib/cwl'

Is this expected, and will I hit the same problem trying to run “real” cwl scripts? That is, is sudo required to use cwltool fully?

[mr-c]

Yep, we’ve narrowed down the trigger and I can reproduce your error.

FYI: cwltool works great for folks without docker and without root; until this gets fixed you can work around the bug with --no-container or have docker installed and available to regular users.

[kapilkd13]

Hi @mr-c I tested with no-container. here is the output

/usr/bin/python2.7 /home/kapil/CWLTool/cwltool/cwltool.py --debug --no-container /home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl /home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort-job.json
/home/kapil/CWLTool/cwltool/cwltool.py 1.0.20170721221557
Resolved '/home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl' to 'file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl'
[workflow revsort.cwl] initialized from file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl
[workflow revsort.cwl] start
[workflow revsort.cwl] starting step rev
[job step rev] job input {
    "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl#rev/input": {
        "class": "File", 
        "location": "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/whale.txt", 
        "size": 1111, 
        "basename": "whale.txt", 
        "nameroot": "whale", 
        "nameext": ".txt"
    }
}
[job step rev] evaluated job input to {
    "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl#rev/input": {
        "class": "File", 
        "location": "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/whale.txt", 
        "size": 1111, 
        "basename": "whale.txt", 
        "nameroot": "whale", 
        "nameext": ".txt"
    }
}
[step rev] start
[job rev] initializing from file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revtool.cwl as part of step rev
[job rev] {
    "input": {
        "class": "File", 
        "location": "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/whale.txt", 
        "size": 1111, 
        "basename": "whale.txt", 
        "nameroot": "whale", 
        "nameext": ".txt"
    }
}
[job rev] path mappings is {
    "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/whale.txt": [
        "/home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/whale.txt", 
        "/tmp/tmpd9sNIa/stgd1a15daa-4105-4939-9556-290deb6b9dc6/whale.txt", 
        "File", 
        true
    ]
}
[job rev] command line bindings is [
    {
        "position": [
            -1000000, 
            0
        ], 
        "datum": "rev"
    }, 
    {
        "position": [
            0, 
            "input"
        ], 
        "datum": {
            "class": "File", 
            "location": "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/whale.txt", 
            "size": 1111, 
            "basename": "whale.txt", 
            "nameroot": "whale", 
            "nameext": ".txt", 
            "path": "/tmp/tmpd9sNIa/stgd1a15daa-4105-4939-9556-290deb6b9dc6/whale.txt", 
            "dirname": "/tmp/tmpd9sNIa/stgd1a15daa-4105-4939-9556-290deb6b9dc6"
        }
    }
]
[job rev] /tmp/tmpFEi1w9$ rev \
    /tmp/tmpd9sNIa/stgd1a15daa-4105-4939-9556-290deb6b9dc6/whale.txt > /tmp/tmpFEi1w9/output.txt
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
[job rev] completed success
[job rev] {
    "output": {
        "checksum": "sha1$97fe1b50b4582cebc7d853796ebd62e3e163aa3f", 
        "basename": "output.txt", 
        "nameroot": "output", 
        "nameext": ".txt", 
        "http://commonwl.org/cwltool#generation": 0, 
        "location": "file:///tmp/tmpFEi1w9/output.txt", 
        "class": "File", 
        "size": 1111
    }
}
[step rev] produced output {
    "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl#rev/output": {
        "checksum": "sha1$97fe1b50b4582cebc7d853796ebd62e3e163aa3f", 
        "basename": "output.txt", 
        "nameroot": "output", 
        "nameext": ".txt", 
        "http://commonwl.org/cwltool#generation": 0, 
        "location": "file:///tmp/tmpFEi1w9/output.txt", 
        "class": "File", 
        "size": 1111
    }
}
[step rev] completed success
[job rev] Removing input staging directory /tmp/tmpd9sNIa
[job rev] Removing temporary directory /tmp/tmpXzK3B1
[workflow revsort.cwl] starting step sorted
[job step sorted] job input {
    "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl#sorted/input": {
        "checksum": "sha1$97fe1b50b4582cebc7d853796ebd62e3e163aa3f", 
        "basename": "output.txt", 
        "nameroot": "output", 
        "nameext": ".txt", 
        "http://commonwl.org/cwltool#generation": 0, 
        "location": "file:///tmp/tmpFEi1w9/output.txt", 
        "class": "File", 
        "size": 1111
    }, 
    "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl#sorted/reverse": true
}
[job step sorted] evaluated job input to {
    "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl#sorted/input": {
        "checksum": "sha1$97fe1b50b4582cebc7d853796ebd62e3e163aa3f", 
        "basename": "output.txt", 
        "nameroot": "output", 
        "nameext": ".txt", 
        "http://commonwl.org/cwltool#generation": 0, 
        "location": "file:///tmp/tmpFEi1w9/output.txt", 
        "class": "File", 
        "size": 1111
    }, 
    "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl#sorted/reverse": true
}
[step sorted] start
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
[job sorted] initializing from file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/sorttool.cwl as part of step sorted
[job sorted] {
    "input": {
        "checksum": "sha1$97fe1b50b4582cebc7d853796ebd62e3e163aa3f", 
        "basename": "output.txt", 
        "nameroot": "output", 
        "nameext": ".txt", 
        "http://commonwl.org/cwltool#generation": 0, 
        "location": "file:///tmp/tmpFEi1w9/output.txt", 
        "class": "File", 
        "size": 1111
    }, 
    "reverse": true
}
[job sorted] path mappings is {
    "file:///tmp/tmpFEi1w9/output.txt": [
        "/tmp/tmpFEi1w9/output.txt", 
        "/tmp/tmpyED81r/stgeccbdabf-aa5e-4e3e-a4a1-c90b1ee59cab/output.txt", 
        "File", 
        true
    ]
}
[job sorted] command line bindings is [
    {
        "position": [
            -1000000, 
            0
        ], 
        "datum": "sort"
    }, 
    {
        "position": [
            1, 
            "reverse"
        ], 
        "prefix": "--reverse", 
        "datum": true
    }, 
    {
        "position": [
            2, 
            "input"
        ], 
        "datum": {
            "checksum": "sha1$97fe1b50b4582cebc7d853796ebd62e3e163aa3f", 
            "basename": "output.txt", 
            "nameroot": "output", 
            "nameext": ".txt", 
            "http://commonwl.org/cwltool#generation": 0, 
            "location": "file:///tmp/tmpFEi1w9/output.txt", 
            "path": "/tmp/tmpyED81r/stgeccbdabf-aa5e-4e3e-a4a1-c90b1ee59cab/output.txt", 
            "dirname": "/tmp/tmpyED81r/stgeccbdabf-aa5e-4e3e-a4a1-c90b1ee59cab", 
            "class": "File", 
            "size": 1111
        }
    }
]
[job sorted] /tmp/tmpqDT90M$ sort \
    --reverse \
    /tmp/tmpyED81r/stgeccbdabf-aa5e-4e3e-a4a1-c90b1ee59cab/output.txt > /tmp/tmpqDT90M/output.txt
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
[job sorted] completed success
[job sorted] {
    "output": {
        "checksum": "sha1$b9214658cc453331b62c2282b772a5c063dbd284", 
        "basename": "output.txt", 
        "nameroot": "output", 
        "nameext": ".txt", 
        "http://commonwl.org/cwltool#generation": 0, 
        "location": "file:///tmp/tmpqDT90M/output.txt", 
        "class": "File", 
        "size": 1111
    }
}
[step sorted] produced output {
    "file:///home/kapil/CWLLanguage/common-workflow-language/v1.0/v1.0/revsort.cwl#sorted/output": {
        "checksum": "sha1$b9214658cc453331b62c2282b772a5c063dbd284", 
        "basename": "output.txt", 
        "nameroot": "output", 
        "nameext": ".txt", 
        "http://commonwl.org/cwltool#generation": 0, 
        "location": "file:///tmp/tmpqDT90M/output.txt", 
        "class": "File", 
        "size": 1111
    }
}
[step sorted] completed success
[workflow revsort.cwl] completed success
[job sorted] Removing input staging directory /tmp/tmpyED81r
[job sorted] Removing temporary directory /tmp/tmpMvnHmO
Moving /tmp/tmpqDT90M/output.txt to /home/kapil/CWLTool/cwltool/output.txt
{
    "output": {
        "checksum": "sha1$b9214658cc453331b62c2282b772a5c063dbd284", 
        "basename": "output.txt", 
        "nameroot": "output", 
        "nameext": ".txt", 
        "http://commonwl.org/cwltool#generation": 0, 
        "location": "file:///home/kapil/CWLTool/cwltool/output.txt", 
        "path": "/home/kapil/CWLTool/cwltool/output.txt", 
        "class": "File", 
        "size": 1111
    }
}
Removing intermediate output directory /tmp/tmpqDT90M
Removing intermediate output directory /tmp/tmpsPt06Y
Final process status is success

Process finished with exit code 0

It worked fine on my system. I think it is correct behaviour because incase of no-container we never reach var/lib/cwl as we check for use container variable here https://github.com/common-workflow-language/cwltool/blob/master/cwltool/process.py#L573.