google-v2 support (Pipelines API v2alpha1)See original GitHub issue
We have now implemented support for Google Genomics new Pipelines API
All existing unit and integration tests pass.
We encourage people who are interested in testing the new API to please start to do so.
Some feature requests for
dsub have been blocked since Pipelines
v1alpha2 did not support them.
Now we can start to add support for these features, including #111 and #99.
A few changes to note, regarding the
google-v2 provider as compared to the
--zonesis supported, but now so is
To specify the VM you want, use the
Set it to a value from https://cloud.google.com/compute/docs/machine-types.
We are interested in feedback on whether supporting
--min-ram is desirable and how we might implement it.
The Pipelines API
v1alpha2 did not support explicit machine types and it did not support Custom Machine Types. Instead it found the “smallest” Predefined Machine Type that fit the
Once Custom Machine Types were introduced, this meant that the Pipelines API was in a sense over-provisioning the VM selection. If you actually wanted 1 core and 2 GB of RAM, you instead got 1 core and 3.75 GB of RAM.
google-v2, our intention is to implement
--min-ram with a straight translation to a custom VM shape. This would mean shedding the legacy behavior of the Pipelines API
However, we suspect that some people will want a “compatibility mode” when moving from the
google-v2 provider. This would mean attempting to emulate the
v1alpha2 behavior. We could implement this, either as the default or with a flag. Again it feels like shedding the legacy behavior of Pipelines API
v1alpha2 would be preferred. When people port their existing
dsub code, they can take a look at the resources they are requesting and ensure that the request fits the task and potentially save money by requesting the VM shape that is really needed.
- Created 5 years ago
- Comments:6 (4 by maintainers)
Top GitHub Comments
Personally I like having to specify the machine type. Its what you do in the console when you create a instance or have to do on the command line. Then also I know exactly what I’m getting. Losing the capability won’t affect any users that I work with. So I’d vote for just shedding the legacy behavior.