google-v2 support (Pipelines API v2alpha1)

We have now implemented support for Google Genomics new Pipelines API v2alpha1.

All existing unit and integration tests pass. We encourage people who are interested in testing the new API to please start to do so. Some feature requests for dsub have been blocked since Pipelines v1alpha2 did not support them. Now we can start to add support for these features, including #111 and #99.

A few changes to note, regarding the google-v2 provider as compared to the google provider:

• --zones is supported, but now so is --region.
• --min-cores and --min-ram are ignored.

To specify the VM you want, use the --machine-type parameter. Set it to a value from https://cloud.google.com/compute/docs/machine-types.

We are interested in feedback on whether supporting --min-cores and --min-ram is desirable and how we might implement it.

The Pipelines API v1alpha2 did not support explicit machine types and it did not support Custom Machine Types. Instead it found the “smallest” Predefined Machine Type that fit the --min-cores and --min-ram request.

Once Custom Machine Types were introduced, this meant that the Pipelines API was in a sense over-provisioning the VM selection. If you actually wanted 1 core and 2 GB of RAM, you instead got 1 core and 3.75 GB of RAM.

For google-v2, our intention is to implement --min-cores and --min-ram with a straight translation to a custom VM shape. This would mean shedding the legacy behavior of the Pipelines API v1alpha2.

However, we suspect that some people will want a “compatibility mode” when moving from the google provider to google-v2 provider. This would mean attempting to emulate the v1alpha2 behavior. We could implement this, either as the default or with a flag. Again it feels like shedding the legacy behavior of Pipelines API v1alpha2 would be preferred. When people port their existing dsub code, they can take a look at the resources they are requesting and ensure that the request fits the task and potentially save money by requesting the VM shape that is really needed.

[slagelwa]

Personally I like having to specify the machine type. Its what you do in the console when you create a instance or have to do on the command line. Then also I know exactly what I’m getting. Losing the capability won’t affect any users that I work with. So I’d vote for just shedding the legacy behavior.

[mbookman]

The google-v2 provider is now at at feature parity with the google provider and includes new features like support for Cloud Storage FUSE. Closing this issue with #126.