[backend] GCR images not accessible: `Consumer 'project:ml-pipeline' has been suspended.`

Environment

• How did you deploy Kubeflow Pipelines (KFP)?

Standalone installation with Kustomize on EKS.

• KFP version: 1.7.0

My KFP runs are failing and when I tried to pull this image: gcr.io/ml-pipeline/argoexec:v3.1.14-license-compliance manually from another unrelated machine, that failed with the following error:

Error response from daemon: Head https://gcr.io/v2/ml-pipeline/argoexec/manifests/v3.1.14-license-compliance: denied: Permission denied: Consumer 'project:ml-pipeline' has been suspended.

I apologize if this is something from my side, but it seems serious if it is not.

Edit:

Suggestions

1. Your jobs might not actually be failing. For us, the main container runs without issue. Only the wait container with the gcr.io/ml-pipeline/argoexec:v3.1.14-license-compliance image is stuck. This comment details how to deal with that temporarily.
2. If you have your core Kubeflow Pipelines components on machines that have the potential to roll. On spot instance, for example, then: a. Try freezing them if you can. For example, certain K8s scaling systems allow you to restrict scale down on certain node groups. b. Copy over the core images to one of your private container registries as soon as possible. After that, you should be able to update your Kubeflow Pipelines manifests with these private images.

---

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[VishDev12]

Could we please get a short RCA from the developers before we close this issue? I want to evaluate the risks of something like this repeating.

[chensun]

Could we please get a short RCA from the developers before we close this issue? I want to evaluate the risks of something like this repeating.

Apologize for the trouble, our project was mistakenly suspended for a short period due to some process error, and we’ll be working on a postmortem to prevent this from happening again.