Add filters to .dockstore.yml

Is your feature request related to a problem? Please describe. Some users create multiple workflows from a single GitHub repo. This more easily allows the different workflows to share code. The .dockstore.yml supports this via a list of workflows.

In this “one GitHub repo, many workflows” model, versions from different workflows show up associated with every workflow from the repo. This makes it hard for a user to view the relevant versions for a particular workflow.

Describe the solution you’d like The .dockstore.yml should have a filters section. See CircleCI filters as an example of one way to implement this.

Another benefit of this approach is that it gives users control over whether to publish branches or not. We’ve discussed this a few times whether Dockstore should or shouldn’t publish branches; this gives the workflow developer fine-grained control over it.

A third benefit would be that it could limit the amount of data Dockstore indexes.

Slightly Detailed Proposals If the filters section is not specified, then the branch/tag is processed by Dockstore.

If the filters section is specified, then it’s processed by Dockstore if it meets the filter condition.

CircleCI divides filters into branches and tags; Dockstore could do the same, and/or also support releases. Or we have a simple filter. Rough ideas:

workflows:
  - name: gwas
    primaryDescriptorPath: /gwas.wdl
    filter: 
      - gwas*
      - develop

Or

workflows:
  - name: gwas
    primaryDescriptorPath: /gwas.wdl
    filter: 
      - tags:
        - gwas*
      - branches
       - develop
       - master

Describe alternatives you’ve considered

Hide the version after publishing

The Dockstore API lets you hide versions for published workflows. This approach has some limitations.

1. You can only hide the version after exists.
2. User would need to figure out how to automatically invoke it. Push to GitHub triggers Dockstore to read version; what then triggers the API call to hide it?
3. The versions would still be there in the database and in the My Workflows view.

Push to another repo

One of our users is doing a version of this. This approach will work great, and will be the best in the that it gives the user a chance to run tests and flatten descriptors.

1. Workflow is pushed to GitHub repo A.
2. CI server, e.g., Travis or CircleCI, auto runs tests; if they pass
   1. Optionally flattens the descriptors (combines workflows’ descriptors into one descriptor)
   2. Pushes to GitHub repos B, C, and D (one for each workflow)
   3. GitHub repos B, C, D are configured with Dockstore GitHub app; Dockstore gets notified and reads the latest.

The main downside of this approach is initial set-up work, but once you have it, it gives you a lot of flexibility, and you’ll have the confidence of tests validating your code.

Have different .dockstore.yml’s in different branches and/or tags Before creating a tag/release, change the .dockstore.yml to match the the tag being released.

Starting point:

...
workflows:
  - name:  gwas
    primaryDescriptorPath: /gwas.wdl
  -name: align
    primaryDescriptorPath: /aligner.wdl

Before creating GWAS release, change the .dockstore.yml to get rid of align:

...
workflows:
  - name:  gwas
    primaryDescriptorPath: /gwas.wdl

Now before creating align release, the change the .dockstore.yml to get rid of gwas and add align

...
workflows:
  -name: align
    primaryDescriptorPath: /aligner.wdl

Downsides:

1. Cannot release multiple workflows off the same commit
2. .dockstore.yml will be changing frequently. Not necessarily a bad thing, but it’s typically fairly static in other user cases.

┆Issue is synchronized with this Jira Story ┆Epic: Automated sync of Dockstore and GitHub apps
 ┆Issue Number: DOCK-1513 ┆Sprint: Sprint 41 Pufferfish ┆Issue Type: Story

[coverbeck]

The different branches do not need to match. What matters is the .dockstore.yml that is present for an event. IOW, if you push branch1, Dockstore will read the .dockstore.yml for branch1 and process that version accordingly. If you then commit a different .dockstore.yml to branch2, and push branch2, Dockstore will read the .dockstore.yml in branch 2 and process that branch accordingly.

[kbergin]

@coverbeck

1. We don’t usually use tags that aren’t releases, I’m not sure if that’s typical or not.
2. Yea the multi repo vs single repo thing usually comes down to preference.