Cross site indexing of content

Feature Request

In response to this discussion https://twitter.com/DockstoreOrg/status/937730314746712064

It would be great to have a community developed, cross index of Dockerstore (and other container resource) content. I can add some detail about Singularity Hub / Registry, is that helps.

• Singularity Registry and Hub both serve image manifests to download images. Singularity Hub comes from Google Cloud Storage, and Singularity Registry comes from a served filesystem.
• The images themselves serve a json manifest with inspect that conforms to http://jsonapi.org/format/
• The manifests match the models in the database, not any kind of standard.

Minimally, given that a manifest is associated with some kind of container content, we would want to be able to search them. The general container naming convention that Singularity Hub uses:

<registry>/<namespace>/<container>:<digest>

Is also a nice search strategy in that it maps well to any kind of storage.

I don’t have specific suggestions or ideas, but just want to open up the conversation. I am an advocate for a strategy that is easy, maybe even fun, and can be flexible to allow for many different resources (e.g., a formal registry, or a Github repository)

┆Issue is synchronized with this Jira Story ┆fixVersions: Dockstore 2.X ┆friendlyId: DOCK-299 ┆sprint: Backlog ┆taskType: Story

[denis-yuen]

Ok, so if I understand correctly. Individual institutions can spin up a Singularity Registry to provide Singularity containers in addition to using those from Singularity Hub. Currently, you have a strategy of recording what sites have Singularity Manifests by listing them in GitHub and generating a page using GitHub Pages based on that information.

We have a similar idea in that we were thinking of creating a Badge system to validate and check that systems that implement the Tool Registry Schema are valid and then creating list of them.

[vsoch]

Singularity Hub is a cloud service that provides building, from Github repos. Singularity Registry is optimized for a local institution to deploy, and then push images to it. Both serve the same manifests for downloading, but the registry comes from a filesystem, and Singularity Hub is Google Cloud Storage.

Additional functions to parse the manifests that the registries serve (to then make them searchable from one box) would fit the model I was describing. If we extend that model to manifests of different kinds, then I think we have a good start!

But note the strategy I came up with is as simple as I possible could do - the entire page I linked is served statically (rendered automatically) via Github pages, and other registries are added by adding a markdown file and doing a PR. Any method we do to collect manifests should be that simple - if we have a server, then it could be just an interactive web interface to do it.