Failed to pull image "gcr.io/kfserving/kfserving-controller:v0.5.1"

/kind bug

What steps did you take and what happened:

Attempted to deploy kfserving from kustomize of KF 1.3.0

Failed to pull image “gcr.io/kfserving/kfserving-controller:v0.5.1”: rpc error: code = Unknown desc = Error response from daemon: Head https://gcr.io/v2/kfserving/kfserving-controller/manifests/v0.5.1: denied: Project kfserving has been deleted.

What did you expect to happen:

expected the images to be present at the configured location

Anything else you would like to add:

This appears to be a relatively recent problem but I’m not entirely sure of that.

I also attempted to pull v0.6.0 and that also failed.

Environment:

• Istio Version: v1.9.0
• Knative Version: ???
• KFServing Version: v1.5.1
• Kubeflow version: v1.3.0
• Kfdef:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
• Kubernetes version: Client Version: version.Info{Major:“1”, Minor:“21”, GitVersion:“v1.21.1”, GitCommit:“5e58841cce77d4bc13713ad2b91fa0d961e69192”, GitTreeState:“clean”, BuildDate:“2021-05-12T14:18:45Z”, GoVersion:“go1.16.4”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“21”, GitVersion:“v1.21.3”, GitCommit:“ca643a4d1f7bfe34773c74f79527be4afd95bf39”, GitTreeState:“clean”, BuildDate:“2021-07-15T20:59:07Z”, GoVersion:“go1.16.6”, Compiler:“gc”, Platform:“linux/amd64”}
• OS (e.g. from /etc/os-release): Ubuntu 20.04

[dtpryce]

@yuzisun has this happened again we are experiencing Failed to pull image "gcr.io/kfserving/storage-initializer:v0.4.0": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/kfserving/storage-initializer/manifests/v0.4.0: denied: Permission denied: Consumer 'project:kfserving' has been suspended

[yuzisun]

@ReggieCarey @siddharth-mitra @salanki can you try now? the kfserving gcr project was incorrectly deleted it is undeleted now.