Failed to fetch anonymous token when trying to pull Triton Docker image in CI

Description Every few CI runs I see this exact error when I try to pull the Triton Docker image. It causes my CI to fail, which is really frustrating!

#5 [internal] load metadata for nvcr.io/nvidia/tritonserver:21.09-py3
#5 sha256:427d8fa572bc82e08f6dcef7f8d657525a6eca13d4e4fe03fea3258a157c0558
#5 ERROR: failed to authorize: rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 401 Unauthorized
#3 [internal] load metadata for nvcr.io/nvidia/tritonserver:21.09-py3-min
#3 sha256:ab4baa3f48727d25fa6024bc84cf599d16c85f5a907845f4bb3b82991ea5d04b
#3 ERROR: failed to authorize: rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 401 Unauthorized

Triton Information 21.09

Are you using the Triton container or did you build it yourself?

I am trying to build Triton myself, which usually works well up until a few weeks ago when I started seeing this issue.

To Reproduce Try to pull the docker image a few times in a row. My CI runs every 10 minutes or so.

Expected behavior The docker image pull does not give me a 401.

[dyastremsky]

Thank you for the update, Bryan! Appreciate that you were able to find a workaround.

I’ll leave this ticket open to work on until Hiren closes it, as this NGC issue is still impacting users. (CC: @cnegron-nv)

[bryanmcgrane]

Just to follow up. I have resolved the issue for my team. We now back up Triton in our own private container registry. When we build, we pull from our registry.