Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Azure Container Registry token works for push but not pull

See original GitHub issue

Firstly, thanks for the amazing tool!

Environment:

  • Jib version: 2.5.2
  • Build tool: Apache Maven 3.6.0
  • OS: Ubuntu 18.04.4 LTS

Description of the issue: In short: Jib is able to authenticate to the Azure Container Registry for pushing, but not for pulling, despite using the same token credentials which have both push and pull rights. Perhaps related to ignoring provided "WWW-Authenticate: Bearer… " header in the pull authentication flow.

In long: I’m running an AzureDevOps Pipelines build agent which is pulling a custom base image from Azure Container Registry, building an image, and pushing to the same registry. I’m supplying credentials to jib using the <to/from><auth> command lines, like so:

/usr/bin/mvn -f /home/azureuser/myagent/_work/3/s/pom.xml --batch-mode -X \
  -Djava.util.logging.config.file=logging.properties \
  -Djib.serialize=true \
  -Djib.from.auth.username=*** -Djib.from.auth.password=*** \
  -Djib.to.auth.username=*** -Djib.to.auth.password=*** \
  -Djib.to.image=***testingmicroservices.azurecr.io/target-image:0.0.723 \
  jib:build

When I do so, I encounter the following error:

[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.5.2:build (default-cli) on project inbox-service-deployable: Build image failed, perhaps you should make sure your credentials for '***testingmicroservices.azurecr.io/base' are set up correctly. See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-should-i-do-when-the-registry-responds-with-unauthorized for help: Unauthorized for ***testingmicroservices.azurecr.io/base: 401 Unauthorized
[ERROR] {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information.","detail":[{"Type":"repository","Name":"base","Action":"pull"}]}]}
[ERROR] -> [Help 1]

The to/from username and password are the same and are using a token which is configured to have both push and pull rights to all of my repositories (_repositories_admin has this property):

Screen Shot 2020-09-24 at 7 37 43 AM

I’ve confirmed that logging into docker directly using these credentials on the same build agent allows me to pull the base image, so the credentials should have pull access.

I enabled debugging per the FAQ instructions and took a look at the logs, included in the logging section below. What stood out to me (I’m no expert on authn/authz), was that the push target image authentication appears to succeed while the pull base image authentication fails.

For the push image authn:

  1. Make an unauthenticated request to .../azurecr.io/v2 -> get a 401 back with “WWW-Auth: Bearer with link”
  2. Make a request using that link -> get a 200 OK back.

For the pull image authn:

  1. Make an unauthenticated request for base image -> get a 401 with “WWW-Authenticate: Bearer with link”
  2. Make a “basic auth request” without using link -> get a 401 with “WWW-Authenticate: Basic”
  3. Make a request. to .../azurecr.io/v2 -> get a 401 back with “WWW-Authenticate: Basic”
  4. Raise error [ERROR] The registry asked for basic authentication, but the registry had refused basic authentication previously

What stood out to me here is that for the pull image flow, we appear to be ignoring the original bearer link and then Azure starts suggesting basic auth instead of bearer link thereafter.

Expected behavior: Jib uses the supplied credentials to pull from the container registry, build the image, and push to the same registry.

jib-maven-plugin Configuration: 

      <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>jib-maven-plugin</artifactId>
        <version>${jib.version}</version>
        <configuration>
          <from>
            <image>mwolffetestingmicroservices.azurecr.io/base@sha256:sha256DigestForBaseImage</image>
          </from>
          <to>
            <image>mwolffetestingmicroservices.azurecr.io/target-image</image>
          </to>
        </configuration>
      </plugin>

Log output: I’ve find-replaced what appear to be tokens in this log output, this is generated using the following logging.properties file:

handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=ALL

# CONFIG hides authentication data
# ALL includes authentication data
com.google.api.client.http.level=ALL

Using the command:

2020-09-24T14:34:37.8730421Z [command]/usr/bin/mvn -f /home/azureuser/myagent/_work/3/s/pom.xml --batch-mode -X -Djava.util.logging.config.file=logging.properties -Djib.serialize=true -Djib.from.auth.username=*** -Djib.from.auth.password=*** -Djib.to.auth.username=*** -Djib.to.auth.password=*** -Djib.to.image=***testingmicroservices.azurecr.io/target-image:0.0.723 jib:build

I’ve only included the authentication logging below, happy to pull further logs.

debug log (click to expand) 
2020-09-24T14:34:42.4249224Z [DEBUG] TIMING	Authenticating push to ***testingmicroservices.azurecr.io
2020-09-24T14:34:42.4255888Z [INFO] Using credentials from <to><auth> for ***testingmicroservices.azurecr.io/target-image:0.0.723
2020-09-24T14:34:42.4277747Z [DEBUG] attempting bearer auth for ***testingmicroservices.azurecr.io/target-image...
2020-09-24T14:34:42.8820338Z Sep 24, 2020 2:34:42 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:42.8822464Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:42.8823556Z GET https://***testingmicroservices.azurecr.io/v2/
2020-09-24T14:34:42.8824172Z Accept: 
2020-09-24T14:34:42.8825517Z Accept-Encoding: gzip
2020-09-24T14:34:42.8826673Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:42.8827343Z 
2020-09-24T14:34:42.8964866Z Sep 24, 2020 2:34:42 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:42.8968329Z CONFIG: curl -v --compressed -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/v2/'
2020-09-24T14:34:43.2620699Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:43.2625419Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:43.2626329Z HTTP/1.1 401 Unauthorized
2020-09-24T14:34:43.2627080Z Server: openresty
2020-09-24T14:34:43.2627838Z Date: Thu, 24 Sep 2020 14:34:43 GMT
2020-09-24T14:34:43.2629090Z Content-Type: application/json; charset=utf-8
2020-09-24T14:34:43.2630199Z Content-Length: 149
2020-09-24T14:34:43.2631357Z Connection: keep-alive
2020-09-24T14:34:43.2632521Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T14:34:43.2633712Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T14:34:43.2635118Z Access-Control-Expose-Headers: Link
2020-09-24T14:34:43.2636388Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T14:34:43.2637581Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T14:34:43.2638832Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.2641320Z Www-Authenticate: Bearer realm="https://***testingmicroservices.azurecr.io/oauth2/token",service="***testingmicroservices.azurecr.io"
2020-09-24T14:34:43.2660018Z X-Content-Type-Options: nosniff
2020-09-24T14:34:43.2661412Z X-Ms-Correlation-Request-Id: 92070ced-e396-4815-8107-bb67cf01e967
2020-09-24T14:34:43.2663292Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.2664081Z 
2020-09-24T14:34:43.2768625Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.2769512Z CONFIG: Total: 149 bytes
2020-09-24T14:34:43.2770059Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.2770950Z CONFIG: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information.","detail":null}]}
2020-09-24T14:34:43.2772101Z 
2020-09-24T14:34:43.2944574Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.2946816Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:43.2970481Z GET https://***testingmicroservices.azurecr.io/oauth2/token?service=***testingmicroservices.azurecr.io&scope=repository:target-image:pull,push&scope=repository:base:pull
2020-09-24T14:34:43.2971899Z Accept: */*
2020-09-24T14:34:43.2973252Z Accept-Encoding: gzip
2020-09-24T14:34:43.2974292Z Authorization: Basic token1
2020-09-24T14:34:43.2976090Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:43.2976947Z 
2020-09-24T14:34:43.2977836Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.2982676Z CONFIG: curl -v --compressed -H 'Accept: */*' -H 'Accept-Encoding: gzip' -H 'Authorization: Basic token1' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/oauth2/token?service=***testingmicroservices.azurecr.io&scope=repository:target-image:pull,push&scope=repository:base:pull'
2020-09-24T14:34:43.3673325Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:43.3679867Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:43.3683519Z HTTP/1.1 200 OK
2020-09-24T14:34:43.3687031Z Server: openresty
2020-09-24T14:34:43.3690378Z Date: Thu, 24 Sep 2020 14:34:43 GMT
2020-09-24T14:34:43.3693923Z Content-Type: application/json; charset=utf-8
2020-09-24T14:34:43.3697164Z Transfer-Encoding: chunked
2020-09-24T14:34:43.3701392Z Connection: keep-alive
2020-09-24T14:34:43.3705534Z X-Ms-Correlation-Request-Id: f31402bd-c1ce-4581-a7c2-c541b7b127bd
2020-09-24T14:34:43.3709701Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.3713457Z 
2020-09-24T14:34:43.3844816Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.3849164Z CONFIG: Total: 1,038 bytes
2020-09-24T14:34:43.3858018Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.3875691Z CONFIG: {"access_token":"accessTokenResponse1"}
2020-09-24T14:34:43.3890373Z [DEBUG] bearer auth succeeded for ***testingmicroservices.azurecr.io/target-image
2020-09-24T14:34:43.3895316Z [DEBUG] TIMED	Authenticating push to ***testingmicroservices.azurecr.io : 965.0 ms
2020-09-24T14:34:43.3927045Z [INFO] Getting manifest for base image ***testingmicroservices.azurecr.io/base@sha256:baseImageSha256Digest...
2020-09-24T14:34:43.4035820Z [DEBUG] TIMING	Pulling base image manifest
2020-09-24T14:34:43.4076877Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.4081410Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:43.4086317Z GET https://***testingmicroservices.azurecr.io/v2/base/manifests/sha256:baseImageSha256Digest
2020-09-24T14:34:43.4091248Z Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json
2020-09-24T14:34:43.4095080Z Accept-Encoding: gzip
2020-09-24T14:34:43.4098562Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:43.4127205Z 
2020-09-24T14:34:43.4134772Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.4144625Z CONFIG: curl -v --compressed -H 'Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/v2/base/manifests/sha256:baseImageSha256Digest'
2020-09-24T14:34:43.4484733Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:43.4487446Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:43.4488590Z HTTP/1.1 401 Unauthorized
2020-09-24T14:34:43.4489388Z Server: openresty
2020-09-24T14:34:43.4490496Z Date: Thu, 24 Sep 2020 14:34:43 GMT
2020-09-24T14:34:43.4491750Z Content-Type: application/json; charset=utf-8
2020-09-24T14:34:43.4492925Z Content-Length: 198
2020-09-24T14:34:43.4494076Z Connection: keep-alive
2020-09-24T14:34:43.4495297Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T14:34:43.4499490Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T14:34:43.4501856Z Access-Control-Expose-Headers: Link
2020-09-24T14:34:43.4503365Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T14:34:43.4504898Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T14:34:43.4507774Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.4510078Z Www-Authenticate: Bearer realm="https://***testingmicroservices.azurecr.io/oauth2/token",service="***testingmicroservices.azurecr.io",scope="repository:base:pull"
2020-09-24T14:34:43.4515044Z X-Content-Type-Options: nosniff
2020-09-24T14:34:43.4516785Z X-Ms-Correlation-Request-Id: 2191d58f-b57c-43ec-8eea-0e106166c19c
2020-09-24T14:34:43.4518092Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.4518727Z 
2020-09-24T14:34:43.4519637Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.4520509Z CONFIG: Total: 198 bytes
2020-09-24T14:34:43.4521162Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.4523205Z CONFIG: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information.","detail":[{"Type":"repository","Name":"base","Action":"pull"}]}]}
2020-09-24T14:34:43.4524146Z 
2020-09-24T14:34:43.4524997Z [INFO] The base image requires auth. Trying again for ***testingmicroservices.azurecr.io/base@sha256:baseImageSha256Digest...
2020-09-24T14:34:43.4526527Z [INFO] Using credentials from <from><auth> for ***testingmicroservices.azurecr.io/base@sha256:baseImageSha256Digest
2020-09-24T14:34:43.4527698Z [DEBUG] Trying basic auth for ***testingmicroservices.azurecr.io/base@sha256:baseImageSha256Digest...
2020-09-24T14:34:43.4528766Z [DEBUG] configured basic auth for ***testingmicroservices.azurecr.io/base
2020-09-24T14:34:43.4542773Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.4544316Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:43.4545930Z GET https://***testingmicroservices.azurecr.io/v2/base/manifests/sha256:baseImageSha256Digest
2020-09-24T14:34:43.4548092Z Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json
2020-09-24T14:34:43.4549605Z Accept-Encoding: gzip
2020-09-24T14:34:43.4551787Z Authorization: Basic bXdvbGZmZS10ZXN0aW5nLWNpLXBpcGVsaW5lLXRva2VuOm13b2xmZmUtdGVzdGluZy1jaS1waXBlbGluZS10b2tlbg==
2020-09-24T14:34:43.4555096Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:43.4556172Z 
2020-09-24T14:34:43.4557045Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.4561294Z CONFIG: curl -v --compressed -H 'Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json' -H 'Accept-Encoding: gzip' -H 'Authorization: Basic bXdvbGZmZS10ZXN0aW5nLWNpLXBpcGVsaW5lLXRva2VuOm13b2xmZmUtdGVzdGluZy1jaS1waXBlbGluZS10b2tlbg==' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/v2/base/manifests/sha256:baseImageSha256Digest'
2020-09-24T14:34:43.5908561Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:43.5910623Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:43.5911370Z HTTP/1.1 401 Unauthorized
2020-09-24T14:34:43.5911934Z Server: openresty
2020-09-24T14:34:43.5912496Z Date: Thu, 24 Sep 2020 14:34:43 GMT
2020-09-24T14:34:43.5913547Z Content-Type: application/json; charset=utf-8
2020-09-24T14:34:43.5915335Z Content-Length: 198
2020-09-24T14:34:43.5916315Z Connection: keep-alive
2020-09-24T14:34:43.5917395Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T14:34:43.5918595Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T14:34:43.5919945Z Access-Control-Expose-Headers: Link
2020-09-24T14:34:43.5921104Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T14:34:43.5922428Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T14:34:43.5923747Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.5924828Z Www-Authenticate: Basic realm="Azure Container Registry"
2020-09-24T14:34:43.5925990Z X-Content-Type-Options: nosniff
2020-09-24T14:34:43.5927052Z X-Ms-Correlation-Request-Id: 7bf8fe87-09a7-4a01-8651-b64790a70eda
2020-09-24T14:34:43.5928363Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.5929244Z 
2020-09-24T14:34:43.5929873Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.5930610Z CONFIG: Total: 198 bytes
2020-09-24T14:34:43.5931280Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.5932531Z CONFIG: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information.","detail":[{"Type":"repository","Name":"base","Action":"pull"}]}]}
2020-09-24T14:34:43.5933562Z 
2020-09-24T14:34:43.5934334Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.5935421Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:43.5936250Z GET https://***testingmicroservices.azurecr.io/v2/
2020-09-24T14:34:43.5936824Z Accept: 
2020-09-24T14:34:43.5937852Z Accept-Encoding: gzip
2020-09-24T14:34:43.5939012Z Authorization: Basic bXdvbGZmZS10ZXN0aW5nLWNpLXBpcGVsaW5lLXRva2VuOm13b2xmZmUtdGVzdGluZy1jaS1waXBlbGluZS10b2tlbg==
2020-09-24T14:34:43.5940642Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:43.5941485Z 
2020-09-24T14:34:43.5942126Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:43.5944897Z CONFIG: curl -v --compressed -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'Authorization: Basic bXdvbGZmZS10ZXN0aW5nLWNpLXBpcGVsaW5lLXRva2VuOm13b2xmZmUtdGVzdGluZy1jaS1waXBlbGluZS10b2tlbg==' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/v2/'
2020-09-24T14:34:43.5946644Z [DEBUG] Trying bearer auth for ***testingmicroservices.azurecr.io/base@sha256:baseImageSha256Digest...
2020-09-24T14:34:43.5947554Z [DEBUG] attempting bearer auth for ***testingmicroservices.azurecr.io/base...
2020-09-24T14:34:43.6540153Z Sep 24, 2020 2:34:43 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:43.6542852Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:43.6544297Z HTTP/1.1 401 Unauthorized
2020-09-24T14:34:43.6545143Z Server: openresty
2020-09-24T14:34:43.6545941Z Date: Thu, 24 Sep 2020 14:34:43 GMT
2020-09-24T14:34:43.6547672Z Content-Type: application/json; charset=utf-8
2020-09-24T14:34:43.6551877Z Content-Length: 149
2020-09-24T14:34:43.6553468Z Connection: keep-alive
2020-09-24T14:34:43.6556159Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T14:34:43.6557576Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T14:34:43.6558922Z Access-Control-Expose-Headers: Link
2020-09-24T14:34:43.6560310Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T14:34:43.6563378Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T14:34:43.6565165Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.6566853Z Www-Authenticate: Basic realm="Azure Container Registry"
2020-09-24T14:34:43.6578265Z X-Content-Type-Options: nosniff
2020-09-24T14:34:43.6579734Z X-Ms-Correlation-Request-Id: c280b785-d194-47e3-bbdd-c7eb74eec80e
2020-09-24T14:34:43.6582435Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:43.6582855Z 
2020-09-24T14:34:43.6584240Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.6584827Z CONFIG: Total: 149 bytes
2020-09-24T14:34:43.6585385Z Sep 24, 2020 2:34:43 PM com.google.api.client.util.LoggingByteArrayOutputStream close
2020-09-24T14:34:43.6587170Z CONFIG: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information.","detail":null}]}
2020-09-24T14:34:43.6588498Z 
2020-09-24T14:34:43.6589104Z [DEBUG] server requires basic auth for ***testingmicroservices.azurecr.io/base
2020-09-24T14:34:43.6590621Z [ERROR] The registry asked for basic authentication, but the registry had refused basic authentication previously
2020-09-24T14:34:43.6591329Z [DEBUG] TIMED	Pulling base image manifest : 251.0 ms
2020-09-24T14:34:43.6621287Z [DEBUG] TIMING	Preparing application layer builders
2020-09-24T14:34:43.6631007Z [DEBUG] TIMED	Preparing application layer builders : 1.0 ms
2020-09-24T14:34:43.6641263Z [INFO] Building dependencies layer...
2020-09-24T14:34:43.6644371Z [DEBUG] TIMING	Building dependencies layer
2020-09-24T14:34:45.0809577Z [DEBUG] Building dependencies layer built sha256:targetImageSha256
2020-09-24T14:34:45.0826552Z [DEBUG] TIMED	Building dependencies layer : 1418.0 ms
2020-09-24T14:34:45.0840672Z [INFO] Building resources layer...
2020-09-24T14:34:45.0848212Z [DEBUG] TIMING	Building resources layer
2020-09-24T14:34:45.1048175Z [DEBUG] Building resources layer built sha256:1d0b9a031e93e686a32f3f60566cfd02f9e52a0e8fed13c4ecfaa3037d4054cf
2020-09-24T14:34:45.1053357Z [DEBUG] TIMED	Building resources layer : 21.0 ms
2020-09-24T14:34:45.1071042Z [INFO] Building classes layer...
2020-09-24T14:34:45.1083081Z [DEBUG] TIMING	Building classes layer
2020-09-24T14:34:45.1286480Z [DEBUG] Building classes layer built sha256:2b22dea88866add0448162295beab663ce4cdb41d7a72602bb6018355c6c5318
2020-09-24T14:34:45.1291631Z [DEBUG] TIMED	Building classes layer : 21.0 ms
2020-09-24T14:34:45.1319166Z [DEBUG] TIMING	Preparing layer pushers
2020-09-24T14:34:45.1330904Z [DEBUG] TIMED	Preparing layer pushers : 1.0 ms
2020-09-24T14:34:45.1345258Z [DEBUG] TIMING	Pushing BLOB digest: sha256:targetImageSha256, size: 17016695
2020-09-24T14:34:45.1390901Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:45.1395520Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:45.1411105Z HEAD https://***testingmicroservices.azurecr.io/v2/target-image/blobs/sha256:targetImageSha256
2020-09-24T14:34:45.1416219Z Accept: 
2020-09-24T14:34:45.1421939Z Accept-Encoding: gzip
2020-09-24T14:34:45.1463269Z Authorization: Bearer accessTokenResponse1
2020-09-24T14:34:45.1467837Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:45.1474595Z 
2020-09-24T14:34:45.1482604Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:45.1521321Z CONFIG: curl -v --compressed -X HEAD -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'Authorization: Bearer accessTokenResponse1' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/v2/target-image/blobs/sha256:targetImageSha256'
2020-09-24T14:34:45.3842225Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:45.3843977Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:45.3844649Z HTTP/1.1 404 Not Found
2020-09-24T14:34:45.3845014Z Server: openresty
2020-09-24T14:34:45.3845460Z Date: Thu, 24 Sep 2020 14:34:45 GMT
2020-09-24T14:34:45.3846397Z Content-Type: application/json; charset=utf-8
2020-09-24T14:34:45.3847085Z Content-Length: 157
2020-09-24T14:34:45.3847740Z Connection: keep-alive
2020-09-24T14:34:45.3848860Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T14:34:45.3849725Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T14:34:45.3850856Z Access-Control-Expose-Headers: Link
2020-09-24T14:34:45.3851723Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T14:34:45.3852431Z Cache-Control: no-cache
2020-09-24T14:34:45.3853198Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T14:34:45.3854053Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:45.3855023Z X-Content-Type-Options: nosniff
2020-09-24T14:34:45.3855685Z X-Ms-Client-Request-Id: 
2020-09-24T14:34:45.3856533Z X-Ms-Correlation-Request-Id: bd322c68-58e7-4218-bb14-bf350d32682b
2020-09-24T14:34:45.3857330Z X-Ms-Request-Id: 6ccf72f7-a28e-4836-9ed4-e20d10fb3178
2020-09-24T14:34:45.3859306Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:45.3859643Z 
2020-09-24T14:34:45.4162266Z [DEBUG] TIMING	pushBlob
2020-09-24T14:34:45.4162975Z [DEBUG] 	TIMING	pushBlob POST sha256:targetImageSha256
2020-09-24T14:34:45.4213689Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:45.4215668Z CONFIG: -------------- REQUEST  --------------
2020-09-24T14:34:45.4217386Z POST https://***testingmicroservices.azurecr.io/v2/target-image/blobs/uploads/?mount=sha256:targetImageSha256&from=base
2020-09-24T14:34:45.4218085Z Accept: 
2020-09-24T14:34:45.4219362Z Accept-Encoding: gzip
2020-09-24T14:34:45.4224707Z Authorization: Bearer accessTokenResponse1
2020-09-24T14:34:45.4228712Z User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)
2020-09-24T14:34:45.4229129Z 
2020-09-24T14:34:45.4229520Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpRequest execute
2020-09-24T14:34:45.4250394Z CONFIG: curl -v --compressed -X POST -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'Authorization: Bearer accessTokenResponse1' -H 'User-Agent: jib 2.5.2 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)' -- 'https://***testingmicroservices.azurecr.io/v2/target-image/blobs/uploads/?mount=sha256:targetImageSha256&from=base'
2020-09-24T14:34:45.6467158Z Sep 24, 2020 2:34:45 PM com.google.api.client.http.HttpResponse <init>
2020-09-24T14:34:45.6468915Z CONFIG: -------------- RESPONSE --------------
2020-09-24T14:34:45.6469382Z HTTP/1.1 202 Accepted
2020-09-24T14:34:45.6469901Z Server: openresty
2020-09-24T14:34:45.6470435Z Date: Thu, 24 Sep 2020 14:34:45 GMT
2020-09-24T14:34:45.6471133Z Content-Length: 0
2020-09-24T14:34:45.6471704Z Connection: keep-alive
2020-09-24T14:34:45.6472498Z Access-Control-Expose-Headers: Docker-Content-Digest
2020-09-24T14:34:45.6473325Z Access-Control-Expose-Headers: WWW-Authenticate
2020-09-24T14:34:45.6475505Z Access-Control-Expose-Headers: Link
2020-09-24T14:34:45.6476521Z Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
2020-09-24T14:34:45.6477411Z Docker-Distribution-Api-Version: registry/2.0
2020-09-24T14:34:45.6478286Z Docker-Upload-Uuid: dockerUploadId
2020-09-24T14:34:45.6480219Z Location: /v2/target-image/blobs/uploads/dockerUploadId?_nouploadcache=false&_state=9Zz4cun1jV2G6HgfuWfSwPuhjJcF_l2fjoJFiTRypoR7Ik5hbWUiOiJpbmJveC1zZXJ2aWNlIiwiVVVJRCI6ImY3MzljNzA1LTdhN2MtNDYzZC05OTU4LTNmYzljMTgyZjNlNSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMC0wOS0yNFQxNDozNDo0NS42Mjc5NzIyMzJaIn0%3D
2020-09-24T14:34:45.6481569Z Range: 0-0
2020-09-24T14:34:45.6482366Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:45.6483171Z X-Content-Type-Options: nosniff
2020-09-24T14:34:45.6483861Z X-Ms-Client-Request-Id: 
2020-09-24T14:34:45.6484687Z X-Ms-Correlation-Request-Id: 8af63e2f-fd51-40c2-8c69-62f4cf2827ba
2020-09-24T14:34:45.6485670Z X-Ms-Request-Id: b75261dc-d8f4-45d3-a6a7-fd25a72f8556
2020-09-24T14:34:45.6486745Z Strict-Transport-Security: max-age=31536000; includeSubDomains
2020-09-24T14:34:45.6487066Z 
2020-09-24T14:34:45.6487754Z [DEBUG] 	TIMED	pushBlob PATCH sha256:targetImageSha256 : 231.0 ms

Additional Information: I’ve read the FAQ and tried using a credHelper, the credentials are found, but encounter the same issue.

Happy to provide more information to help debug this, apologies in advance if this is simply user error 😬

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:1
  • Comments:16 (9 by maintainers)

github_iconTop GitHub Comments

3reactions
chanseokohcommented, Sep 29, 2020

@carstenrathmann it’s a different issue, and I’d say in some part the issue is on the GitHub Container Registry.

ghcr.io correctly returns 401 Unauthorized with WWW-Authenticate: Bearer when starting the bearer auth flow.

$ curl -D - https://ghcr.io/v2/
HTTP/1.1 401 Unauthorized
Content-Type: application/json
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://ghcr.io/token",scope="repository:user/image:pull"
Date: Tue, 29 Sep 2020 18:34:25 GMT
Content-Length: 73
X-GitHub-Request-Id: B630:6A82:7D064:14637D:5F737E30

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required"}]}

However, when a client sends some credentials (can be an arbitrary garbage string) as part of the request, it returns 200 OK, implying that it doesn’t require authentication.

$ curl -D - -H 'Authorization: WHATEVER GARBAGE' https://ghcr.io/v2/
HTTP/1.1 200 OK
Content-Type: application/json
Docker-Distribution-Api-Version: registry/2.0
Date: Tue, 29 Sep 2020 18:34:29 GMT
Content-Length: 0
X-GitHub-Request-Id: B640:0AA7:1D606:5908C:5F737E35

The good news is that, with the optimized auth flow proposed in #2789, I confirmed it resolves your issue, as #2789 won’t get through this failing execution path.

I see that ghcr.io is public beta as of now. Even though #2789 resolves your issue, I suggest filing a bug against them anyway. Properly fixing the issue on their side will make Jib <=2.5.2 work again.

2reactions
chanseokohcommented, Oct 8, 2020

@EugenDueck I’ve opened #2812. Please follow up there.

@maxwolffe @carstenrathmann thanks for confirming!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshoot login to registry - Azure Container Registry
Symptoms, causes, and resolution of common problems when logging into an Azure container registry.
Read more >
Permissions to repositories in Azure Container Registry
Create a token with permissions scoped to specific repositories in a Premium registry to pull or push images, or perform other actions.
Read more >
Troubleshoot network issues with registry - Azure Container ...
Unable to add or modify virtual network settings or public access rules. ACR Tasks is unable to push or pull images.
Read more >
Authenticate with an Azure container registry - Microsoft Learn
Authentication options for a private Azure container registry, ... No, always pull and push access, Single account per registry, not ...
Read more >
Enable anonymous pull access - Azure Container Registry
By default, access to pull or push content from an Azure container registry is only available to authenticated users.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Set environment variables for credHelper

Next page

[Jib Gradle Plugin] BuildStepsExecutionException: Cannot run program "docker": error=2, No such file or directory