runner: support GCP OIDC auth

I’m attempting to create a CML template workflow to use a self-hosted runner in GCP. To authenticate into GCP I would like to use the newly available OIDC method (described here).

After using the google-github-actions/auth a authentication file is created, but when I run the cml runner command I get the following traceback:

{"level":"warn","message":"ignoring RUNNER_NAME environment variable, use CML_RUNNER_NAME or --name instead"}
{"level":"info","message":"Preparing workdir /home/runner/.cml/cml-8cqnfumhcx..."}
{"level":"info","message":"Deploying cloud runner plan..."}
{"level":"info","message":"Terraform apply..."}
{"level":"error","message":"terraform -chdir='/home/runner/.cml/cml-8cqnfumhcx' apply -auto-approve\n\t\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  + create\n\nTerraform will perform the following actions:\n\n  # iterative_cml_runner.runner will be created\n  + resource \"iterative_cml_runner\" \"runner\" {\n      + cloud                = \"gcp\"\n      + driver               = \"github\"\n      + id                   = (known after apply)\n      + idle_timeout         = 300\n      + image                = (known after apply)\n      + instance_hdd_size    = 35\n      + instance_ip          = (known after apply)\n      + instance_launch_time = (known after apply)\n      + instance_type        = \"m+k80\"\n      + labels               = \"cml-gpu\"\n      + name                 = \"cml-8cqnfumhcx\"\n      + region               = \"us-west\"\n      + repo                 = \"https://github.com/developmentseed/ml-project-seed\"\n      + single               = false\n      + spot                 = false\n      + spot_price           = -1\n      + ssh_public           = (known after apply)\n      + token                = (sensitive value)\n    }\n\nPlan: 1 to add, 0 to change, 0 to destroy.\niterative_cml_runner.runner: Creating...\n\n\t╷\n│ Error: Failed disposing the machine: Couldn't extract the project identifier from the given credentials!\n│ \n│   with iterative_cml_runner.runner,\n│   on main.tf line 14, in resource \"iterative_cml_runner\" \"runner\":\n│   14: resource \"iterative_cml_runner\" \"runner\" {\n│ \n╵\n╷\n│ Error: Failed creating the machine: Couldn't extract the project identifier from the given credentials!\n│ \n│   with iterative_cml_runner.runner,\n│   on main.tf line 14, in resource \"iterative_cml_runner\" \"runner\":\n│   14: resource \"iterative_cml_runner\" \"runner\" {\n│ \n╵\n","stack":"Error: terraform -chdir='/home/runner/.cml/cml-8cqnfumhcx' apply -auto-approve\n\t\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  + create\n\nTerraform will perform the following actions:\n\n  # iterative_cml_runner.runner will be created\n  + resource \"iterative_cml_runner\" \"runner\" {\n      + cloud                = \"gcp\"\n      + driver               = \"github\"\n      + id                   = (known after apply)\n      + idle_timeout         = 300\n      + image                = (known after apply)\n      + instance_hdd_size    = 35\n      + instance_ip          = (known after apply)\n      + instance_launch_time = (known after apply)\n      + instance_type        = \"m+k80\"\n      + labels               = \"cml-gpu\"\n      + name                 = \"cml-8cqnfumhcx\"\n      + region               = \"us-west\"\n      + repo                 = \"https://github.com/developmentseed/ml-project-seed\"\n      + single               = false\n      + spot                 = false\n      + spot_price           = -1\n      + ssh_public           = (known after apply)\n      + token                = (sensitive value)\n    }\n\nPlan: 1 to add, 0 to change, 0 to destroy.\niterative_cml_runner.runner: Creating...\n\n\t╷\n│ Error: Failed disposing the machine: Couldn't extract the project identifier from the given credentials!\n│ \n│   with iterative_cml_runner.runner,\n│   on main.tf line 14, in resource \"iterative_cml_runner\" \"runner\":\n│   14: resource \"iterative_cml_runner\" \"runner\" {\n│ \n╵\n╷\n│ Error: Failed creating the machine: Couldn't extract the project identifier from the given credentials!\n│ \n│   with iterative_cml_runner.runner,\n│   on main.tf line 14, in resource \"iterative_cml_runner\" \"runner\":\n│   14: resource \"iterative_cml_runner\" \"runner\" {\n│ \n╵\n\n    at /usr/local/lib/node_modules/@dvcorg/cml/src/utils.js:21:27\n    at ChildProcess.exithandler (node:child_process:404:5)\n    at ChildProcess.emit (node:events:390:28)\n    at maybeClose (node:internal/child_process:1064:16)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)","status":"terminated"}

Looking through the terraform documentation, they state:

Note: Currently the only supported service account credentials are credentials downloaded from Cloud Console or generated by gcloud.

This could be the reason why this OIDC generated authentication file is not accepted, but I’m not sure.

Have you encountered this issue before? Did you experiment with OIDC for GCP? Any help would be greatly appreciated.

https://github.com/developmentseed/ml-project-seed/pull/6

For reference here is the current state of my github actions workflow:

name: CML (GCP)
on: [push]
permissions:
  id-token: write
  contents: read

jobs:
  deploy-runner:
    runs-on: ubuntu-latest
    steps:
      - uses: iterative/setup-cml@v1
      - uses: actions/checkout@v2
      - name: Authenticate to GCP
        id: auth
        uses: google-github-actions/auth@v0
        with:
            workload_identity_provider: 'projects/348645304830/locations/global/workloadIdentityPools/my-pool/providers/my-provider'
            service_account: '[e-mail]'
            project_id: 348645304830
      - name: Deploy runner on EC2
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }} 
          GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.auth.outputs.credentials_file_path }}
        run: |
          gcloud services list 
          cml runner \
              --cloud=gcp \
              --cloud-region=us-west \
              --cloud-type=m+k80 \
              --labels=cml-gpu

[dacbd]

@rodrigoalmeida94 this is supported by the new TPI release: https://github.com/iterative/terraform-provider-iterative/releases/tag/v0.10.8 I am currently slowly working on https://github.com/iterative/cml.dev/pull/208 to document some of the gotchas.

I think that it’s safe to this is not the currently recommended mode of authentication for using cml runner as if your workflow (with no changes to the default authentication resource) lasts longer than 1hr the credentials used by cml runner will expire and the instance self-termination will fail.

I intend to also document/create an easy/convenient method to update the credentials in use. You can follow updates here: https://github.com/iterative/cml.dev/issues/391

Basic example:

#...
      - uses: actions/checkout@v3
      - name: 'Authenticate to Google Cloud'
        uses: 'google-github-actions/auth@v0'
        with:
          workload_identity_provider: 'projects/200200923017/locations/global/workloadIdentityPools/gh-actions-pool/providers/github-actions'
          service_account: '[e-mail]'
      - run: |
          cml runner \
            --single \
            --labels=test-oidc \
            --token=${{ secrets.DACBD_PAT }} \
            --cloud=gcp \
            --cloud-region=us-west \
            --cloud-type=e2-highcpu-2 \
#...

[rodrigoalmeida94]

Good to know about the token expiry, hadn’t thought about this! Thanks @dacbd