Unable to push to harbor image with http

Environment:

• Jib version: 1.5.0
• Build tool: Gradle
• OS: Mac OSX
• Repository: Harbor

Description of the issue:

I have insecure harbor registry for now and trying to push it and pushing it with property sendCredentialsOverHttp=true but push always times out

Expected behavior:

Should be able to push

Steps to reproduce:

/gradlew jib -DsendCredentialsOverHttp=true -Djib.to.image=dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample -Djib.console=plain

jib-gradle-plugin Configuration:

jib {
    to {
        allowInsecureRegistries = true        
        tags = ['latest', '1.0.0-SNAPSHOT']
        auth {
            username = 'xyz'
            password = 'xyz'
        }
    }
    container {
        ports = ['8080']
        mainClass = "$mainClassName"
        args = ["run", mainVerticleName, "-conf staging.json"]
        jvmFlags = ['-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4j2LogDelegateFactory']
    }

}

Log output:

➜ ./gradlew jib -DsendCredentialsOverHttp=true -Djib.to.image=dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample -Djib.console=plain
Starting a Gradle Daemon, 1 busy and 1 incompatible Daemons could not be reused, use --status for details

> Task :jib
Authentication over HTTP is enabled. It is strongly recommended that you do not enable this on a public network!

Containerizing application to dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample, dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample, dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample:1.0.0-SNAPSHOT...
Retrieving registry credentials for dockerhub-1003850024.ap-south-1.elb.amazonaws.com...
Getting base image gcr.io/distroless/java:8...
Building dependencies layer...
Building project dependencies layer...
Building resources layer...
Building classes layer...

> Task :jib FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Connect to dockerhub-1003850024.ap-south-1.elb.amazonaws.com:443 [dockerhub-1003850024.ap-south-1.elb.amazonaws.com/35.154.123.120, dockerhub-1003850024.ap-south-1.elb.amazonaws.com/13.234.102.169] failed: Operation timed out (Connection timed out)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.5.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 2m 40s
8 actionable tasks: 1 executed, 7 up-to-date

Additional Information:

Push works fine with docker command-line

docker tag xyz:1.0.0-SNAPSHOT dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample/xyz:1.0.0-SNAPSHOT

docker push dockerhub-1003850024.ap-south-1.elb.amazonaws.com/sample/xyz:1.0.0-SNAPSHOT

[chanseokoh]

@raizoor both the default connection timeout and the read timeout are set to 20 seconds by default (configurable with -Djib.httpTimeout), so I’m actually confused why it isn’t enforced.

I did an investigation, and I think it is the bug in recent Google HTTP Client versions that does not honor the given connection timeout. (Haven’t tested read timeout yet.) Jib 1.5.0 is affected, and I am positive that the timeout will work correctly on 1.4.0.

[chanseokoh]

@riyaz looked at your last log. No wonder it takes > 20 minutes (with 1.5.0). At least it is indeed working but very inefficient because Jib first tries HTTPS (port 443) before trying HTTP (port 80) for every access to your registry. And this HTTPS-to-HTTP failover can happen only when the HTTPS attempt times out. The log shows that the timeout is about 75 seconds. That is, for every HTTP request Jib makes to your registry, it first has to wait for ~75 seconds. But I do see that the plain http:// is very fast.

22:11:01.880 [INFO] [com.google.api.client.http.HttpTransport] -------------- REQUEST  --------------
GET https://dockerhub-1003850024.ap-south-1.elb.amazonaws.com/v2/

22:12:17.265 [INFO] [org.gradle.api.Task] Failed to connect to https://dockerhub-1003850024.ap-south-1.elb.amazonaws.com/v2/ over HTTPS. Attempting again with HTTP: http://dockerhub-1003850024.ap-south-1.elb.amazonaws.com/v2/
22:12:17.269 [INFO] [com.google.api.client.http.HttpTransport] -------------- REQUEST  --------------

22:12:17.269 [INFO] [com.google.api.client.http.HttpTransport] curl -v --compressed -H 'Accept: ' -H 'Accept-Encoding: gzip' -H 'User-Agent: jib 1.5.0 jib-gradle-plugin Google-HTTP-Java-Client/1.30.0 (gzip)' -- 'http://dockerhub-1003850024.ap-south-1.elb.amazonaws.com/v2/'
22:12:17.484 [INFO] [com.google.api.client.http.HttpTransport] -------------- RESPONSE --------------
HTTP/1.1 401 Unauthorized
Date: Wed, 21 Aug 2019 16:42:17 GMT

22:12:17.500 [INFO] [com.google.api.client.http.HttpTransport] -------------- REQUEST  --------------
GET http://dockerhub-1003850024.ap-south-1.elb.amazonaws.com/service/token?service=harbor-registry&scope=repository:sample/starter:pull,push

22:12:17.500 [INFO] [com.google.api.client.http.HttpTransport] curl -v --compressed -H 'Accept: */*' -H 'Accept-Encoding: gzip' -H 'Authorization: <Not Logged>' -H 'User-Agent: jib 1.5.0 jib-gradle-plugin Google-HTTP-Java-Client/1.30.0 (gzip)' -- 'http://dockerhub-1003850024.ap-south-1.elb.amazonaws.com/service/token?service=harbor-registry&scope=repository:sample/starter:pull,push'
22:12:17.586 [INFO] [com.google.api.client.http.HttpTransport] -------------- RESPONSE --------------
HTTP/1.1 200 OK

That said, if you specify the port :80 in your image reference (e.g., ...elb.amazonaws.com:80/sample/...), I bet it will be very fast, as it will attempt the plain HTTP directly. Later you could add an additional tag using whatever means to have the correct name without the port part (:80). Another dirty hack would be to somehow configure your firewall or network so that any connection attempt to amazonaws.com:443 drops immediately as @briandealwis said.

But, I strongly recommend you make the port 443 work as a registry and close the insecure registry at port 80. I’m emphasizing again that by sending credentials over HTTP, anyone in the wild can capture and steal your username and password easily.

I have requested our infra team to setup ssl, meanwhile i have switched to aws ecr for now.

Given that you don’t have a secure registry, ECR seems rather the right choice. I would continue using it, at least until you set up SSL.

Regarding no network logs with gradle, --no-daemon seems to work sometime only, if we downgrade the gradle version to 4.10.3 it always works.

Thanks for the info. It’s good to know. We are baffled too.