Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

google-v2 support (Pipelines API v2alpha1)

See original GitHub issue

We have now implemented support for Google Genomics new Pipelines API v2alpha1.

All existing unit and integration tests pass. We encourage people who are interested in testing the new API to please start to do so. Some feature requests for dsub have been blocked since Pipelines v1alpha2 did not support them. Now we can start to add support for these features, including #111 and #99.

A few changes to note, regarding the google-v2 provider as compared to the google provider:

  • --zones is supported, but now so is --region.
  • --min-cores and --min-ram are ignored.

To specify the VM you want, use the --machine-type parameter. Set it to a value from

We are interested in feedback on whether supporting --min-cores and --min-ram is desirable and how we might implement it.

The Pipelines API v1alpha2 did not support explicit machine types and it did not support Custom Machine Types. Instead it found the “smallest” Predefined Machine Type that fit the --min-cores and --min-ram request.

Once Custom Machine Types were introduced, this meant that the Pipelines API was in a sense over-provisioning the VM selection. If you actually wanted 1 core and 2 GB of RAM, you instead got 1 core and 3.75 GB of RAM.

For google-v2, our intention is to implement --min-cores and --min-ram with a straight translation to a custom VM shape. This would mean shedding the legacy behavior of the Pipelines API v1alpha2.

However, we suspect that some people will want a “compatibility mode” when moving from the google provider to google-v2 provider. This would mean attempting to emulate the v1alpha2 behavior. We could implement this, either as the default or with a flag. Again it feels like shedding the legacy behavior of Pipelines API v1alpha2 would be preferred. When people port their existing dsub code, they can take a look at the resources they are requesting and ensure that the request fits the task and potentially save money by requesting the VM shape that is really needed.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:6 (4 by maintainers)

github_iconTop GitHub Comments

slagelwacommented, Jun 30, 2018

Personally I like having to specify the machine type. Its what you do in the console when you create a instance or have to do on the command line. Then also I know exactly what I’m getting. Losing the capability won’t affect any users that I work with. So I’d vote for just shedding the legacy behavior.

mbookmancommented, Oct 5, 2018

The google-v2 provider is now at at feature parity with the google provider and includes new features like support for Cloud Storage FUSE. Closing this issue with #126.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Runs a pipeline. The returned Operation's metadata field will contain a google.genomics.v2alpha1.Metadata object describing the status of the pipeline ...
Read more >
Namespace Google.Apis.Genomics.v2alpha1
The "pipelines" collection of methods. PipelinesResource.RunRequest. Runs a pipeline. The returned Operation's metadata field will contain a google.genomics.
Read more >
Migrate to Cloud Life Sciences v2beta
The process of migrating Google Genomics v2alpha1 requests to Cloud Life Sciences API v2beta requests primarily consists of replacing field names and ......
Read more >
dsub/ at main · DataBiosphere/dsub - GitHub
The google-v2 provider utilizes the Google Genomics Pipelines API v2alpha1 while the google-cls-v2 provider utilizes the Google Cloud Life Sciences Piplines ...
Read more >
Google - Cromwell - Read the Docs
Google Genomics Pipelines API is a Docker-as-a-service from Google. ... providers { PAPIv2 { actor-factory = "
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found