Dockstore CLI generates invalid JSON if Dockstore CLI wants you to update

Describe the bug Due to the nature of this bug I can’t be sure if it still exists in 1.10.0, but it does exist in 1.9.0 so it might be worth looking into. Basically, when using Dockstore’s recommended method for generating a JSON for a WDL, the output WDL will include error output about Dockstore’s version being out of date, resulting in an invalid JSON.

To Reproduce Steps to reproduce the behavior:

1. Be using Dockstore 1.9.0
2. Run dockstore workflow convert entry2json --entry github.com/manning-lab/vcfToGds:master > Dockstore.json
3. End up with a Dockstore.json that looks like this

14:23:17.596 [main] ERROR io.dockstore.client.cli.ArgumentUtility - Current version : 1.9.0 14:23:17.599 [main] ERROR io.dockstore.client.cli.ArgumentUtility - Latest version : 1.10.0 14:23:17.599 [main] ERROR io.dockstore.client.cli.ArgumentUtility - You do not have the most recent stable release of Dockstore. Please upgrade with the following command: dockstore --upgrade { “vcfToGds_wf.vcf_files”: “Array[File]”, “vcfToGds_wf.this_disk”: “Int”, “vcfToGds_wf.this_memory”: “Float” }

Expected behavior The error output should not be in the output file. The resulting JSON should be valid. While I get that we would want users to run the latest stable version of Dockstore, I don’t think we should be breaking their pipelines to do so. If we want to keep the erroring when a different version is detected, one solution might be changing how we recommended users obtain output JSONs on Dockstore rather than changing the output itself.

Screenshots

Desktop (please complete the following information):

• OS: Catalina
• Version: Dockstore CLI 1.9.0 but may persist in 1.10.0

┆Issue is synchronized with this Jira Story ┆fixVersions: Open-ended research tasks ┆friendlyId: DOCK-1713 ┆sprint: Backlog ┆taskType: Story

[amarjandu]

Related to this issue, the ds-cli is generating invalid cwl files when trying to download from quay.io.

dockstore tool cwl --entry quay.io/collaboratory/dockstore-tool-bamstats:1.25-7
Current version : 1.12.0
You have the most recent stable release.
If you wish to upgrade to the latest unstable version, please use the following command:
   dockstore --upgrade-unstable
#!/usr/bin/env cwl-runner

class: CommandLineTool
id: "BAMStats"
label: "BAMStats tool"
cwlVersion: v1.0
doc: |
    ![build_status](https://quay.io/repository/collaboratory/dockstore-tool-bamstats/status)
    A Docker container for the BAMStats command. See the [BAMStats](http://bamstats.sourceforge.net/) website for more information.

dct:creator:
  "@id": "http://orcid.org/0000-0002-7681-6415"
  foaf:name: Brian O'Connor
  foaf:mbox: "mailto:[e-mail]"

requirements:
  - class: DockerRequirement
    dockerPull: "quay.io/collaboratory/dockstore-tool-bamstats:1.25-6_1.0"

hints:
  - class: ResourceRequirement
    coresMin: 1
    ramMin: 4092  # "the process requires at least 4G of RAM"
    outdirMin: 512000

inputs:
  mem_gb:
    type: int
    default: 4
    doc: "The memory, in GB, for the reporting tool"
    inputBinding:
      position: 1

  bam_input:
    type: File
    doc: "The BAM file used as input, it must be sorted."
    format: "http://edamontology.org/format_2572"
    inputBinding:
      position: 2

outputs:
  bamstats_report:
    type: File
    format: "http://edamontology.org/format_3615"
    outputBinding:
      glob: bamstats_report.zip
    doc: "A zip file that contains the HTML report and various graphics."

baseCommand: ["bash", "/usr/local/bin/bamstats"]

$namespaces:
  dct: http://purl.org/dc/terms/
  foaf: http://xmlns.com/foaf/0.1/

In the tutorial the user is guided to create a file using the command dockstore tool cwl --entry quay.io/collaboratory/dockstore-tool-bamstats:1.25-7 > Dockstore.cwl dockstore tool convert cwl2json --cwl Dockstore.cwl > Dockstore.json

In both cases the files that are generated become malformed.

Any errors and warnings should go to stderr, and anything parseable should go to stdout.

👍🏽

[chmreid]

The behavior as described, “it was put into stdout because some users didn’t want to upgrade and also didn’t want to see it the stderr when they run their workflows,” sounds like an anti-pattern. Any errors and warnings should go to stderr, and anything parseable should go to stdout. That way, when a user runs a command like the one in the comment just above that, it still produces valid JSON.

Warning messages should not break scripts, and this approach seems like it could unexpectedly break scripts that were producing valid JSON prior to a new release. If users don’t want warning messages showing up in their stderr stream, we should not send warnings to stdout instead, we should provide some additional documentation/examples to users of how to use shell redirects to handle stdout/stderr.