Support getting summary metadata information quickly in bulk

Motivation

The Job Manager UI is hoping to support a dashboard view, pictured below. We’ve heard from multiple users that this view would be very helpful to them. In addition, this capability is of especially high priority for our partners in this, Verily. Verily also needs to make changes to dsub to support this.

What is it

This may be more information than you need, but I decided to err on providing more info over less.

The dashboard view has panels of grouped information, categorized by labels on the jobs. Imagine a user had an owner label and a project label on all of their jobs. The dashboard panels would be pivoted by project and owner, and show probably the first ~5-10 labels that have the most running jobs with that label. These panels would be populated with a header that is the key of the cromwell label, a list of values that match that key that the users have access to, and then a summary of their statuses.

The dashboard will be filterable by other labels, but maybe not at first. A use case example there is filtering the image above by a label key:value of flag:archived. There is a concept of flagging jobs as archived so you don’t see them anymore, as a way to get your failures list down to “inbox 0” and say “I’ve addressed those jobs, I don’t want to see them anymore”. So it’s possible a user could want to filter those jobs out of their dashboard view as well.

v1 will not have this chart pictured and will not have the left panel of server information.

Ticket Prioritization Suggestions

1. I would like to start with a spike/design doc and scoping out the amount of effort it would take to support this in Cromwell before end of Q1.
2. This ticket can also represent the implementation if Cromwell wants, which we need by end of May to be able to do the front end work before end of Q2.

Current Status

Currently, I think this view would require many pings to the cromwell query endpoint with different queries to get back all of the numbers and results.

Risks I know of

I don’t think this is blocked by the labels endpoint update in #3233, but wanted to mention it in case it is a risk.

ACs

• The Job Manager Dashboard can be quickly filled in
• The user can choose which labels they want to get this summary information on (i.e. it’s not only a fixed set of labels that are supported by Cromwell)

[kbergin]

@ruchim I think we talked about this offline, but it’d be workflow level info. Except I guess for the aforementioned idea of grouping by task failures.

[kbergin]

A discussion with @ruchim yielded these notes on this idea: To support dashboard view in Job Manager a first step could be enabling searching by label key (values undefined) with other query parameters. i.e.: Show me all jobs that “were running at some point in the last X hours/days” and have the key:comment instead of making a new endpoint.