Unable to access GCS storageUri using ServiceAccount credentials

/kind bug

What steps did you take and what happened:

I’m trying to deploy an inferenceservice in kind with a GCS storageUri using the README instructions, but I’m seeing the storage-initializer container fail to read from GCS despite having a ServiceAccount with appropriate access.

To setup KF-Serving, I cloned the repo and rankind create cluster followed by ./hack/quick_install.sh.

> kubectl get pods -n kfserving-system
NAME                             READY   STATUS    RESTARTS   AGE
kfserving-controller-manager-0   2/2     Running   0          92s

We create a secret containing the service account json file.

> kubectl create secret generic gcs-creds --from-file=gcloud-application-credentials.json=/path/to/service-account.json
> kubectl get secret user-gcp-sa -o=yaml
apiVersion: v1
data:
  gcloud-application-credentials.json: <base64 encoded file>

Also define a ServiceAccount using the secret -

apiVersion: v1
kind: ServiceAccount
metadata:
  name: user-gcp-sa
secrets:
  - name: gcs-creds

Use ServiceAccount in InferenceService -

apiVersion: serving.kubeflow.org/v1alpha2
kind: InferenceService
metadata:
  name: example-inference
spec:
  default:
    predictor:
      serviceAccountName: user-gcp-sa
      pytorch:
        modelClassName: ExampleNet
        resources:
          limits:
            cpu: 100m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 1Gi
        storageUri: gs://path/to/bucket

> kubectl logs example-inference-predictor-default-qsvkn-deployment-6d4dffb67f-g6gvd storage-initializer
[I 200330 19:29:33 initializer-entrypoint:13] Initializing, args: src_uri [gs://***] dest_path[ [/mnt/models]
[I 200330 19:29:33 storage:35] Copying contents of gs://*** to local
[I 200330 19:29:37 _metadata:95] Compute Engine Metadata server unavailable onattempt 1 of 3
[I 200330 19:29:40 _metadata:95] Compute Engine Metadata server unavailable onattempt 2 of 3
[I 200330 19:29:40 _metadata:95] Compute Engine Metadata server unavailable onattempt 3 of 3
[I 200330 19:29:40 requests:233] Refreshing credentials due to a 401 response. Attempt 1/2.
Traceback (most recent call last):
  File "/storage-initializer/scripts/initializer-entrypoint", line 14, in <module>
    kfserving.Storage.download(src_uri, dest_path)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 48, in download
    Storage._download_gcs(uri, out_dir)
  File "/usr/local/lib/python3.7/site-packages/kfserving/storage.py", line 100, in _download_gcs
    for blob in blobs:
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 204, in _items_iter
    for page in self._page_iter(increment=False):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 235, in _page_iter
    page = self._next_page()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 361, in _next_page
    response = self._get_next_page_response()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/page_iterator.py", line 411, in _get_next_page_response
    method=self._HTTP_METHOD, path=self.path, query_params=params
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_http.py", line 389, in api_request
    target_object=_target_object,
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_http.py", line 266, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/_http.py", line 295, in _do_request
    return self.http.request(url=url, method=method, headers=headers, data=data)
  File "/usr/local/lib/python3.7/site-packages/google/auth/transport/requests.py", line 239, in request
    self.credentials.refresh(auth_request_with_timeout)
  File "/usr/local/lib/python3.7/site-packages/google/auth/credentials.py", line 148, in refresh
    raise ValueError("Anonymous credentials cannot be refreshed.")
ValueError: Anonymous credentials cannot be refreshed.

The init container doesn’t show a GOOGLE_APPLICATION_CREDENTIALS env var being set.

 storage-initializer:
    Container ID:  containerd://262d813082b96824fc0522518197420a46c6277f1aeef5ae57d5e64ba352aa83
    Image:         gcr.io/kfserving/storage-initializer:0.2.2
    Image ID:      gcr.io/kfserving/storage-initializer@sha256:7a7d3cf4c5121a3e6bad0acc9e88bbdfa9c7f774d80bd64d8e35a84dcfef8890
    Port:          <none>
    Host Port:     <none>
    Args:
      gs://vimeo-ml/thumbnail_models/ssd128
      /mnt/models
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 30 Mar 2020 15:29:01 -0400
      Finished:     Mon, 30 Mar 2020 15:29:02 -0400
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 30 Mar 2020 15:28:46 -0400
      Finished:     Mon, 30 Mar 2020 15:28:47 -0400
    Ready:          False
    Restart Count:  2
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:        100m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /mnt/models from kfserving-provision-location (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from user-gcp-sa-token-zprt6 (ro)

What did you expect to happen:

I expected the storage-initializer container to have a GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to the provided credentials and to successfully read the GCS bucket.

Anything else you would like to add:

Environment:

• Istio Version: 1.3.6
• Knative Version: 0.12.0
• KFServing Version: 0.22
• Kubeflow version: n/a
• Kind version: kind v0.7.0 go1.13.6 darwin/amd64
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2020-01-14T00:09:19Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):

[yuzisun]

/close

[hoanhvong]

That’s so good. I hope you can also update this approach to the official doc in kserve with GCP. Thank you.