GCP ValueError: Service account info was not in the expected format, missing fields token_uri

Trying to create a GCPCluster after specifying my GCP projectid in ~/.config/dask/cloudprovider.yaml I get below error related to ValueError: Service account info was not in the expected format, missing fields token_uri.

from dask_cloudprovider.gcp import GCPCluster
from dask.distributed import wait

cluster = GCPCluster(
    zone="us-central1-a",
    machine_type="n1-standard-8",
    n_workers=1,
    docker_image="rapidsai/rapidsai:cuda11.0-runtime-ubuntu18.04-py3.8",
    worker_class="dask_cuda.CUDAWorker",
    env_vars={"EXTRA_PIP_PACKAGES": "gcsfs"}
)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/anaconda3/envs/daskcp/lib/python3.7/site-packages/google/auth/_default.py in load_credentials_from_file(filename, scopes, quota_project_id)
    134             credentials = service_account.Credentials.from_service_account_info(
--> 135                 info, scopes=scopes
    136             )

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/google/oauth2/service_account.py in from_service_account_info(cls, info, **kwargs)
    210         signer = _service_account_info.from_dict(
--> 211             info, require=["client_email", "token_uri"]
    212         )

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/google/auth/_service_account_info.py in from_dict(data, require)
     50             "Service account info was not in the expected format, missing "
---> 51             "fields {}.".format(", ".join(missing))
     52         )

ValueError: Service account info was not in the expected format, missing fields token_uri.

The above exception was the direct cause of the following exception:

DefaultCredentialsError                   Traceback (most recent call last)
<ipython-input-23-066423d9340a> in <module>
     17     docker_image="rapidsai/rapidsai:cuda11.0-runtime-ubuntu18.04-py3.8",
     18     worker_class="dask_cuda.CUDAWorker",
---> 19     env_vars={"EXTRA_PIP_PACKAGES": "gcsfs"}
     20 )

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/dask_cloudprovider/gcp/instances.py in __init__(self, projectid, zone, machine_type, source_image, docker_image, ngpus, gpu_type, filesystem_size, auto_shutdown, bootstrap, **kwargs)
    512     ):
    513 
--> 514         self.compute = GCPCompute()
    515 
    516         self.config = dask.config.get("cloudprovider.gcp", {})

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/dask_cloudprovider/gcp/instances.py in __init__(self)
    551 
    552     def __init__(self):
--> 553         self._compute = self.refresh_client()
    554 
    555     def refresh_client(self):

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/dask_cloudprovider/gcp/instances.py in refresh_client(self)
    570                     # take first row
    571                     f_.write(creds_rows[0][1])
--> 572                 creds, _ = google.auth.load_credentials_from_file(filename=f)
    573             return googleapiclient.discovery.build("compute", "v1", credentials=creds)
    574 

~/anaconda3/envs/daskcp/lib/python3.7/site-packages/google/auth/_default.py in load_credentials_from_file(filename, scopes, quota_project_id)
    138             msg = "Failed to load service account credentials from {}".format(filename)
    139             new_exc = exceptions.DefaultCredentialsError(msg, caught_exc)
--> 140             six.raise_from(new_exc, caught_exc)
    141         if quota_project_id:
    142             credentials = credentials.with_quota_project(quota_project_id)

~/.local/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

DefaultCredentialsError: ('Failed to load service account credentials from /tmp/tmp79wsxl8a.', ValueError('Service account info was not in the expected format, missing fields token_uri.'))

Anything else we need to know?:

Environment:

• Dask version: 2.30.0
• Python version: 3.7.9
• Operating System: Ubuntu 18.04.4 LTS
• Install method (conda, pip, source): conda

[jacobtomlinson]

I agree with @bradmiro it looks like your project ID is not correct or missing.

Once you have that fixed you are on the right track with configuring accelerators. Here are a few additional comments:

• The a2 instance type family implies A100 GPUs, this is tightly coupled.
• Other instance families such as n1 can have accelerators assigned optionally, so you need to use the gpu_type and ngpus arguments to specify which accelerator to use and how many.
• You can list available accelerators with gcloud compute accelerator-types list.

[bradmiro]

For #2 you may be missing projectid: https://github.com/dask/dask-cloudprovider/blob/53d3c92098ff58029d1d98041b38d3eebf9c7713/dask_cloudprovider/gcp/instances.py#L356-L365