Ability to run tools and workflows offline

A user wanted to be able to run tools and workflows with no internet connection from the CLI.

Workaround:

1. Download descriptor files locally
2. Add the following to the config file: cwltool-extra-parameters: --disable-pull
3. Follow steps here https://stackoverflow.com/a/48125632 to get a copy of the docker image loaded locally
4. Launch the tool dockstore tool launch --local-entry <local-file-path> --json <input-json-path>

We should update the documentation to include this.

Note: The user had an issue getting the image working on a cluster. On the head node the image existed and was named with a tag, however on the compute nodes the repository and tag were missing. Will update here once a solution is found.

┆Issue is synchronized with this Jira Story ┆Fix Versions: Dockstore 1.6 ┆Issue Number: DOCK-422 ┆Sprint: Seabright Sprint 5 Electric ┆Issue Type: Story

[agduncan94]

What they wanted:

• to run a CWL tool on the cluster. The head node is connected to the public internet, but the compute nodes are not.

Problem:

• no internet connection means the compute node won’t be able to download descriptor files or download a docker image

Solution:

• download the docker image on the head node (with internet access) and save it into a file (which the compute nodes have access to)
• load the image onto the compute node
• download the descriptor files to the compute node
• launch using --local-entry

Situation explained by user:

was able to run the dockstore container just fine on the head node of our commodity cluster, which has internet access. Of course the point of having a cluster is to run software on compute nodes, which do not have public internet access. So when trying to execute the same command on a compute node it was complaining that it couldn’t pull down the docker image

So we saved the docker image with “docker save -o dockstore-cgpwgs_with_tags quay.io/wtsicgp/dockstore-cgpwgs:1.1.4” (executed on the head node to some cluster-globally-accessible directory), then added it to the compute node(s) with “docker load -i /path/to/image.tar” (executed on the compute node)

What we had done before, which caused the lack of entries in the repo and tag column was “docker save -o dockstore-cgpwgs.tar 8f2d9de5ab9d”, so just giving the tag id doesn’t place nicely with dockstore

After that, the command “dockstore tool launch --local-entry Dockstore.cwl --json example.json” worked just fine on our offline compute node

[wshands]

Tested on staging and works, but image tar files need to be in their own directory with no other types of files. Also works if you specify dockerImageId instead of dockerPull in the requirements section of the CWL.