Failure message: wrapping host binaries: pulling image: retry budget exhausted (10 attempts): running ["docker" "pull" "bash"]: exit status 1 (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/
See original GitHub issueThis is a new error message for me, and I checked the GCP status page in case it was transient but don’t see any active issues.
I am launching a bunch of tasks across a bunch of zones (within a single job) using the google-cls-v2 API with dsub 0.4.3. I am getting the following error, but only on some tasks within this job (while others within the same job are launching and succeeding):
Failure message: wrapping host binaries: pulling image: retry budget exhausted (10 attempts): running ["docker" "pull" "bash"]: exit status 1 (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n")
This strikes me as an odd message, because (1) I’m running these machines without a public IP, and so (2) I’m only using gcr.io
Docker instances with them. I don’t know why they’d be pointing to registry-1.docker.io.
If this isn’t a dsub issue, I can point this to the mailing list. (And if there is a quick way for me to get a flavor for “dsub issue vs not dsub issue”, just let me know so I can self-triage.)
Thanks!
Issue Analytics
- State:
- Created 3 years ago
- Comments:12 (6 by maintainers)
Latest message from the Pipelines team is that they’ve heard from a few people the very same issue, and are working to have it fixed as soon as possible.
We were hitting this issue thru cromwell in us-central1 and can confirm we haven’t seen it since Google told us there was a fix April 9th.