Premature end of chunk coded message body after upgrading to 3.1.3

Environment:

• Jib version: 3.1.3
• Build tool: Gradle 7.1 (using jib plugin 3.1.3)
• OS: Linux via Gitlab pipelines

Description of the issue: Upgraded from 3.1.2 to 3.1.3 and now getting an error when pulling an image from GCR

I/O error for image [eu.gcr.io/***]:
    org.apache.http.ConnectionClosedException
    Premature end of chunk coded message body: closing chunk expected
I/O error for image [eu.gcr.io/***]:
    org.apache.http.ConnectionClosedException
    Premature end of chunk coded message body: closing chunk expected

This causes it to completely fail

Expected behavior:

It would still work the same from 3.1.2

Steps to reproduce:

1. Upgrade to 3.1.3 and it breaks

jib-gradle-plugin Configuration:

jib {
    from {
        image = "eu.gcr.io/***"
    }
    to {
        image = System.getenv("CONTAINER_IMAGE") ?: "***"
        tags = ["latest", System.getenv("CI_PIPELINE_ID") ?: "dev", System.getenv("CI_COMMIT_SHA") ?: "local"]
    }
    container {
        mainClass = "com.***.Application"
        args = ["server", System.getenv("DROPWIZARD_CONFIG") ?: "development"]
        jvmFlags = [
            "-XX:+UseG1GC",
            "-XX:+UseStringDeduplication",
            "-Dcontrast.dir=/tmp/contrast",
            "-javaagent:contrast.jar",
            "-Dcontrast.assess.sampling.enable=true",
            "-Dcontrast.assess.sampling.baseline=1",
            "-Dcontrast.assess.sampling.request_frequency=1200",
            "-Dcontrast.assess.sampling.window_ms=30000",
        ]
        ports = ["8080", "8081"]
    }
}

Log output:

Containerizing application to eu.gcr.io/***, eu.gcr.io/***, eu.gcr.io/***, eu.gcr.io/***...
Base image 'eu.gcr.io/***' does not use a specific image digest - build may not be reproducible
Getting manifest for base image eu.gcr.io/***...
Building dependencies layer...
Building project dependencies layer...
Building classes layer...
Building jvm arg files layer...
Using Google Application Default Credentials for eu.gcr.io/***
The base image requires auth. Trying again for eu.gcr.io/***...
Using Google Application Default Credentials for eu.gcr.io/***
I/O error for image [eu.gcr.io/***]:
    org.apache.http.ConnectionClosedException
    Premature end of chunk coded message body: closing chunk expected
I/O error for image [eu.gcr.io/***]:
    org.apache.http.ConnectionClosedException
    Premature end of chunk coded message body: closing chunk expected
Using base image with digest: ***
Container entrypoint set to [java, -Xms4g, -Xmx8g, -XX:+UseG1GC, -cp, @/app/jib-classpath-file, ***]
I/O error for image [eu.gcr.io/***]:
    org.apache.http.ConnectionClosedException
    Premature end of chunk coded message body: closing chunk expected
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':maintenance:jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Premature end of chunk coded message body: closing chunk expected

[chanseokoh]

@rmaclean-ee @ChristianCiach @steven-sheehy @danilofnmoreira @abithgs @creekpld @yuri-sergiichuk @nhmarujo @ikorolev93 @olivierchedru @osigida

We just released Jib plugins 3.1.4 which downgraded Google HTTP libraries to 1.34.0. I expect the new release will avoid the issue, unless Gradle overrides the versions. If Gradle overrides them (which often happens unfortunately), one option is to force versions.

[ChristianCiach]

Incredible… I can reproduce the issue, but only with ancient Apache httpd versions (up to 2.4.18, works starting with 2.4.20 (there doesn’t seem to be a 2.4.19)) and only when using SSL (dummy certs work fine to reproduce the issue).

It has nothing to do with the nginx or the registry. Even when just serving files, Google-HttpClient requesting https://localhost:8443 running an ancient Apache httpd will trigger the issue:

> mvn -U exec:java -Dexec.mainClass=GoogleHttpClientMain -Dexec.args="https://localhost:8443/"
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< GoogleHttpClient:GoogleHttpClient >------------------
[INFO] Building GoogleHttpClient 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ GoogleHttpClient ---
[WARNING] 
org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 45; received: 0)
    at org.apache.http.impl.io.ContentLengthInputStream.read (ContentLengthInputStream.java:178)
    at org.apache.http.impl.io.ContentLengthInputStream.read (ContentLengthInputStream.java:198)
    at org.apache.http.impl.io.ContentLengthInputStream.close (ContentLengthInputStream.java:101)
    at org.apache.http.conn.BasicManagedEntity.streamClosed (BasicManagedEntity.java:166)
    at org.apache.http.conn.EofSensorInputStream.checkClose (EofSensorInputStream.java:228)
    at org.apache.http.conn.EofSensorInputStream.close (EofSensorInputStream.java:172)
    at com.google.api.client.http.HttpResponse.ignore (HttpResponse.java:427)
    at com.google.api.client.http.HttpResponse.disconnect (HttpResponse.java:441)
    at GoogleHttpClientMain.main (GoogleHttpClientMain.java:43)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:829)

Of course, I had to hack your client from https://github.com/chanseokoh/docker-hub-upload-speed-test to ignore the invalid ssl cert:

  private static HttpRequestFactory reqFactory;
  static {
    try {
      reqFactory = new ApacheHttpTransport.Builder().doNotValidateCertificate().build().createRequestFactory();
    } catch(Exception e) {
      throw new RuntimeException(e);
    }
  }

I don’t know how to interpret this. The issue only appears with ancient HTTP servers with SSL enabled, but this doesn’t necessarily mean that the server is at fault.

You should be able to easily reproduce this example following these steps:

1. Extract the default httpd.conf from an ancient httpd container:

docker run --rm httpd:2.4.18 cat /usr/local/apache2/conf/httpd.conf > httpd.conf

1. Edit the config to include these lines:

Listen 443
LoadModule ssl_module modules/mod_ssl.so

<VirtualHost _default_:443>
    SSLEngine on
    SSLProtocol TLSv1.2
    SSLCipherSuite ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:!aNULL:!MD5:!DSS

    SSLCertificateFile      /server.crt
    SSLCertificateKeyFile   /server.key
</VirtualHost>

1. Start the container:

docker run --rm -p 8443:443 -v $PWD/server.crt:/server.crt -v $PWD/server.key:/server.key -v $PWD/httpd.conf:/usr/local/apache2/conf/httpd.conf httpd:2.4.18

1. Request https://localhost:8443 using the current version Google HttpClient as shown above.