Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Regression HTTPS verification javax.net.ssl.SSLPeerUnverifiedException on docker layer pull

See original GitHub issue

On upgrade to 2.7.1 a dependency change breaks host verification for docker pull

Change seen within the http request trace

2.7.0  => User-Agent: jib 2.7.0 jib-gradle-plugin Google-HTTP-Java-Client/1.34.0 (gzip)

2.7.1   => User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)

This is possibly a host verification regression.

Environment:

  • *Jib version: 2.7.1
  • *Build tool: Gradle v6.7.1
  • *OS: Ubuntu-16.04 + Ubuntu 20.04
  • *JAVA 8, JAVA 11

Description of the issue: Pulling of image fails on hostname verification

The base image requires auth. Trying again for amazon/aws-lambda-java:11...
No credentials could be retrieved for registry-1.docker.io/amazon/aws-lambda-java
Using base image with digest: sha256:801e43678e238903893085c6e9105a0e9d7efc379b6d2a38752a6a16ff96b83a
I/O error for image [registry-1.docker.io/amazon/aws-lambda-java]:
    javax.net.ssl.SSLPeerUnverifiedException
    Certificate for <docker-images-prod.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: com.google.cloud.tools.jib.api.InsecureRegistryException: Failed to verify the server at https://registry-1.docker.io/v2/amazon/aws-lambda-java/blobs/sha256:382c81da26b147bcaa0e0a46640a30f9f65364c380cc7c30336acccc1d29eb74 because only secure connections are allowed.

Plugin version and jib configuration required

plugins {
    id("com.google.cloud.tools.jib") version "2.7.1" // 2.7.0 works
}
jib {
    to.image = "${project.name}-lambda"
    from.image = "amazon/aws-lambda-java:11"
}

Log output:

./gradlew -g localcache --no-daemon <projectNameRedacted> -x build -x test -x check --info -Djava.util.logging.config.file=logging.properties -Djib.serialize=true

## OUTPUT is redacted of keys/tokens
> pulling base image manifest

-------------- REQUEST  --------------
GET https://registry-1.docker.io/v2/amazon/aws-lambda-java/manifests/11
Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json
Accept-Encoding: gzip
User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)

curl -v --compressed -H 'Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)' -- 'https://registry-1.docker.io/v2/amazon/aws-lambda-java/manifests/11'
-------------- RESPONSE --------------
HTTP/1.1 401 Unauthorized
Content-Type: application/json
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:amazon/aws-lambda-java:pull"
Date: Mon, 15 Feb 2021 15:46:13 GMT
Content-Length: 165
Strict-Transport-Security: max-age=31536000
The base image requires auth. Trying again for amazon/aws-lambda-java:11...
No credentials could be retrieved for registry-1.docker.io/amazon/aws-lambda-java
Executing tasks:
[                              ] 0.0% complete
> pulling base image manifest

-------------- REQUEST  --------------
GET https://auth.docker.io/token?service=registry.docker.io&scope=repository:amazon/aws-lambda-java:pull
Accept: */*
Accept-Encoding: gzip
User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)

curl -v --compressed -H 'Accept: */*' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)' -- 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:amazon/aws-lambda-java:pull'
-------------- RESPONSE --------------
HTTP/1.1 200 OK
Content-Type: application/json
Date: Mon, 15 Feb 2021 15:46:13 GMT
Transfer-Encoding: chunked
Strict-Transport-Security: max-age=31536000

Total: 4,370 bytes
{"token":"<redacted>","expires_in":300,"issued_at":"2021-02-15T15:46:13.26417122Z"}

-------------- REQUEST  --------------
GET https://registry-1.docker.io/v2/amazon/aws-lambda-java/manifests/11
Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json
Accept-Encoding: gzip
Authorization: <Not Logged>
User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)

curl -v --compressed -H 'Accept: application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json' -H 'Accept-Encoding: gzip' -H 'Authorization: <Not Logged>' -H 'User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)' -- 'https://registry-1.docker.io/v2/amazon/aws-lambda-java/manifests/11'
-------------- RESPONSE --------------
HTTP/1.1 200 OK
Content-Length: 1580
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:801e43678e238903893085c6e9105a0e9d7efc379b6d2a38752a6a16ff96b83a
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:801e43678e238903893085c6e9105a0e9d7efc379b6d2a38752a6a16ff96b83a"
Date: Mon, 15 Feb 2021 15:46:13 GMT
Strict-Transport-Security: max-age=31536000
RateLimit-Limit: 100;w=21600
RateLimit-Remaining: 80;w=21600

Total: 1,580 bytes
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 2888,
      "digest": "sha256:382c81da26b147bcaa0e0a46640a30f9f65364c380cc7c30336acccc1d29eb74"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 100753295,
         "digest": "sha256:5db394153f458493e9a627e2acbc32cb1f653e134f68c03870f164ff29a8a541"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 75012,
         "digest": "sha256:a361664b70ca6bf0a1d3c8b4a206a79855dc09b3a38f26a07519dc37ad936356"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 418,
         "digest": "sha256:3adf1af4c33c0b6807cfa453014cc7b72254907cdbc78c08aec6eb9cac7ae03c"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 3310349,
         "digest": "sha256:03ac043af787c8cb06d6a7f8289c62cf97e630b08011717197ab2adb087d061d"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 65193128,
         "digest": "sha256:0361b35e6e0203ab55683be6a6faa54581eeb5038b0beef46203cf505d4c8633"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2716317,
Using base image with digest: sha256:801e43678e238903893085c6e9105a0e9d7efc379b6d2a38752a6a16ff96b83a
Executing tasks:
[                              ] 0.0% complete
> pulling base image manifest

-------------- REQUEST  --------------
GET https://registry-1.docker.io/v2/amazon/aws-lambda-java/blobs/sha256:382c81da26b147bcaa0e0a46640a30f9f65364c380cc7c30336acccc1d29eb74
Accept:
Accept-Encoding: gzip
Authorization: <Not Logged>
User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)

curl -v --compressed -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'Authorization: <Not Logged>' -H 'User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)' -- 'https://registry-1.docker.io/v2/amazon/aws-lambda-java/blobs/sha256:382c81da26b147bcaa0e0a46640a30f9f65364c380cc7c30336acccc1d29eb74'
-------------- RESPONSE --------------
HTTP/1.1 307 Temporary Redirect
Content-Type: application/octet-stream
Docker-Distribution-Api-Version: registry/2.0
Location: https://docker-images-prod.s3.amazonaws.com/registry-v2/docker/registry/v2/blobs/sha256/38/382c81da26b147bcaa0e0a46640a30f9f65364c380cc7c30336acccc1d29eb74/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<redacted>%2F20210215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210215T154613Z&X-Amz-Expires=1200&X-Amz-Security-Token=<REDACTED_TOKEN>&X-Amz-SignedHeaders=host&X-Amz-Signature=7a2fdd56250cfb3f68f3fd2473b897cc64697e4436c6e583f34fc0df3edc1f47
Date: Mon, 15 Feb 2021 15:46:13 GMT
Content-Length: 0
Strict-Transport-Security: max-age=31536000

-------------- REQUEST  --------------
GET https://docker-images-prod.s3.amazonaws.com/registry-v2/docker/registry/v2/blobs/sha256/38/382c81da26b147bcaa0e0a46640a30f9f65364c380cc7c30336acccc1d29eb74/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<redacted>%2F20210215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210215T154613Z&X-Amz-Expires=1200&X-Amz-Security-Token=<REDACTED_TOKEN>&X-Amz-SignedHeaders=host&X-Amz-Signature=7a2fdd56250cfb3f68f3fd2473b897cc64697e4436c6e583f34fc0df3edc1f47
Accept:
Accept-Encoding: gzip
User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)

curl -v --compressed -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 2.7.1 jib-gradle-plugin Google-HTTP-Java-Client/1.38.0 (gzip)' -- 'https://docker-images-prod.s3.amazonaws.com/registry-v2/docker/registry/v2/blobs/sha256/38/382c81da26b147bcaa0e0a46640a30f9f65364c380cc7c30336acccc1d29eb74/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<redacted>%2F20210215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210215T154613Z&X-Amz-Expires=1200&X-Amz-Security-Token=<REDACTED> I/O error for image [registry-1.docker.io/amazon/aws-lambda-java]:
    javax.net.ssl.SSLPeerUnverifiedException
    Certificate for <docker-images-prod.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

Additional Information:

There was a similar verification issue encountered with apache-httpcomponents

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:2
  • Comments:29 (18 by maintainers)

github_iconTop GitHub Comments

2reactions
chanseokohcommented, Apr 27, 2021

@mattupstate seems like you tried 2.7.1 dependencies. What about 2.7.0 dependencies? The issue reported here is that 2.7.0 had been working, but it stopped working since 2.7.1.

1reaction
racevedoocommented, Aug 12, 2021

Tested with jib 3.1.3 without forcing dependencies, got the same problem.

Read more comments on GitHub >

github_iconTop Results From Across the Web

javax.net.ssl.SSLPeerUnverifiedException: Hostname not ...
I am trying to use HTTPS connection ...
Read more >
About storage drivers - Docker Documentation
Docker uses storage drivers to store image layers, and to store data in the writable layer of a container. The container's writable layer...
Read more >
Troubleshooting errors with Docker commands when using ...
In some cases, running a Docker command against Amazon ECR may result in an error ... Error: "Filesystem Layer Verification Failed" when pulling...
Read more >
The Docker executor - GitLab Docs
Linked containers share their environment variables. Create a network for each job. Introduced in GitLab Runner 12.9. To enable network-per ...
Read more >
podman-pull
--disable-content-trust¶. This is a Docker-specific option to disable image verification to a container registry and is not supported by Podman. This option is ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Getting manifest for base image uses Proxy even if it is set in nonProxyHosts

Next page

Jib-gradle-plugin jib.httpTimeout not taken into account