No longer able to parse json files within outputEval

Workflow Code

cwlVersion: v1.0
class: CommandLineTool
requirements:
  DockerRequirement: {dockerPull: "python"}
  InlineJavascriptRequirement: {}
baseCommand:
- python
- -c
- "import json; json.dump({'key1': 'foobar', 'key2': 2}, open('out.json', 'w'))"
inputs: []
outputs:
  output:
    type:
      type: record
      fields:
      - name: key1
        type: string
      - name: key2
        type: int
    outputBinding:
      loadContents: true
      glob: "out.json"
      outputEval: |
        ${
          var res = JSON.parse(self[0].contents)
          console.log(res);
          return res;
        }

Expected Behavior

Previously, I have parsed the json output of a CommandLineTool with JSON.parse(self[0].contents). The above workflow runs properly in, e.g., v1.0.20180525185854. It returns:

Javascript console output:
----------------------------------------
[log] { key1: 'foobar', key2: 2 }
----------------------------------------
[job wf.cwl] completed success
{
    "output": {
        "key2": 2,
        "key1": "foobar"
    }
}
Final process status is success

Actual Behavior

Now, I get a validation error:

Error validating output record. the `output` field is not valid because
  * the `key1` field is not valid because
      the value is not string
  * the `key2` field is not valid because
      `[]` is not int
 in {
    "output": {
        "key1": [],
        "key2": []
    }
}

Full Traceback

[job wf.cwl] /private/tmp/docker_tmp5xw1jlmb$ docker \
    run \
    -i \
    --volume=/private/tmp/docker_tmp5xw1jlmb:/uYEajn:rw \
    --volume=/private/var/folders/x0/0s7bygdj26db4gd20fx6t7k00000gn/T/tmpk1xyygwi:/tmp:rw \
    --workdir=/uYEajn \
    --read-only=true \
    --user=501:20 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/uYEajn \
    --cidfile=<REDACTED>/minimal_failing_example/20181115133418-855907.cid \
    python \
    python \
    -c \
    'import json; json.dump({'"'"'key1'"'"': '"'"'foobar'"'"', '"'"'key2'"'"': 2}, open('"'"'out.json'"'"', '"'"'w'"'"'))'
Running with support for javascript console in expressions (DO NOT USE IN PRODUCTION)
Javascript console output:
----------------------------------------
<REDACTED>/python3.7/site-packages/cwltool/sandboxjs.py:342: FutureWarning: Possible nested set at position 2
  r'^[[](?:log|err)[]].*$', stderr, flags=re.MULTILINE)))
[log] { key1: 'foobar', key2: 2 }
----------------------------------------
[job wf.cwl] Job error:
Error validating output record. the `output` field is not valid because
  * the `key1` field is not valid because
      the value is not string
  * the `key2` field is not valid because
      `[]` is not int
 in {
    "output": {
        "key1": [],
        "key2": []
    }
}
[job wf.cwl] completed permanentFail
{}
Final process status is permanentFail

Environment

• cwltool version: 1.0.20181102182747

[tetron]

This seems like it should work, so I’ll investigate when I have some time.

However you might also be interested in this feature, you can have a special file called “cwl.output.json”:

https://www.commonwl.org/v1.0/CommandLineTool.html#Output_binding

[jeremyadamsfisher]

Would writing out cwl.out.json prevent binding other outputs? Thats what the documentation seems to imply.

I’ve long since moved on from this project – but, if I recall, I had to process a fastq file and report quality metrics. It seemed to me at the time that the best way to do this was to write out a json file, put that into a record. Then, I could pass the record along to the nodes of the pipeline that needed it downstream.

Thanks for looking into this.