Using GCP service account scopes in self hosted runner

I’m attempting to mount a GCS bucket in a self-hosted runner from CML and encountering multiple authentication problems with gcsfuse.

We are using this definition for our cml runner:

cml runner \
              --cloud=gcp \
              --cloud-region=us-west \
              --cloud-type=m+k80 \
              --labels=cml-gpu \
              --cloud-permission-set=[e-mail]

We then mount buckets in our project using gcsfuse:

gcsfuse --debug_gcs --implicit-dirs data/

And this returns the following error:

2022/02/15 19:03:17.273266 Start gcsfuse/0.40.0 (Go version go1.17.6) for app "" using mount point: /__w/ml-project-seed/ml-project-seed/data
2022/02/15 19:03:17.287982 Opening GCS connection...
2022/02/15 19:03:17.291621 Mounting file system "gcsfuse"...
2022/02/15 19:03:17.293180 File system has been successfully mounted.
Here are the contents of the mounted path
$ cd data/bp-padang/cloudcover
/__w/_temp/0c805963-6ad9-44af-931d-9f971accd261.sh: 24: cd: can't cd to data/bp-padang/cloudcover

The service account [e-mail] has been assigned Storage Admin and Compute Admin roles, so theoretically it should have access to the buckets.

After multiple trial and errors, we were able to setup an instance via terraform and successfully mount the buckets with gcsfuse by using these settings:

resource "google_compute_instance" "jupyter" {
  ....
  service_account { scopes = ["storage-full", "cloud-platform"] }
  ...
}

Looks like the scopes are quite important in order to provide instances with permissions in GCS resources. It would be great if we could set those along with other parameters in the cml runner command.

If you have any other experiences mounting GCS buckets in CML based runners, would be happy to hear how you accomplished it without the access scopes. Any help would be really appreciated!

[dacbd]

Thanks @DavidGOrtega, and I can confirm that I had a whole pipeline work successfully with:

          cml-runner \
            ...
            --cloud=gcp \
            --cloud-permission-set=[e-mail],scopes=storage-rw \
            ...

And DVC on the instance automagically had correct permission to the remote bucket 🥳 🎈

[rodrigoalmeida94]

I can also confirm I was able to run a workflow successfully when mounting a bucket using gcsfuse! Thanks so much everyone for the really fast turn around. 🐎 🥳