Failure message: wrapping host binaries: pulling image: retry budget exhausted (10 attempts): running ["docker" "pull" "bash"]: exit status 1 (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/

This is a new error message for me, and I checked the GCP status page in case it was transient but don’t see any active issues.

I am launching a bunch of tasks across a bunch of zones (within a single job) using the google-cls-v2 API with dsub 0.4.3. I am getting the following error, but only on some tasks within this job (while others within the same job are launching and succeeding):

Failure message: wrapping host binaries: pulling image: retry budget exhausted (10 attempts): running ["docker" "pull" "bash"]: exit status 1 (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n")

This strikes me as an odd message, because (1) I’m running these machines without a public IP, and so (2) I’m only using gcr.io Docker instances with them. I don’t know why they’d be pointing to registry-1.docker.io.

If this isn’t a dsub issue, I can point this to the mailing list. (And if there is a quick way for me to get a flavor for “dsub issue vs not dsub issue”, just let me know so I can self-triage.)

Thanks!

[wnojopra]

Latest message from the Pipelines team is that they’ve heard from a few people the very same issue, and are working to have it fixed as soon as possible.

[cblacknygc]

We were hitting this issue thru cromwell in us-central1 and can confirm we haven’t seen it since Google told us there was a fix April 9th.