Operation timed out (Read failed) during push to us.gcr.io

Environment:

• Jib version: 1.6.1
• Build tool: Maven 3.6.1
• OS: macOS 10.14.6

Description of the issue: I tried building an image for the first time for a new project. It failed thrice in a row with Operation timed out (Read failed). I increased the timeout to 60 sec but it failed again. I then used dockerBuild that built to docker daemon and tried pushing using docker push. It took very long time (~ 13 minutes) and reattempted after a failure in middle but was able to successfully push. The image size showed by docker is 169MB, GCR shows virtual size 86.5MB.

After docker successfully pushed to GCR, I executed jib buildagain with default timeout and surprisingly it succeeded and pushed in no time (32 sec) (as there was no change to project) whereas it was timing out just before docker pushed the image.

Expected behavior: I believe jib should’ve pushed it without any exception.

Steps to reproduce:

1. It’s a private project, thus I can’t share the project to build the image from, but if there is something I should check configuration wise, please let me know.
2. I just executed mvn -X -Djib.serialize=true -Djib.console=plain compile jib:build with the given configuration.
3. I’ve given the log output post processing of project classes that may be helpful.

jib-maven-plugin Configuration:

    <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>jib-maven-plugin</artifactId>
        <version>1.6.1</version>
        <configuration>
          <from>
            <credHelper>gcloud</credHelper>
          </from>
          <to>
            <image>us.gcr.io/${projectId}/zl-wzgp:0.1.0-SNAPSHOT</image>
            <credHelper>gcloud</credHelper>
            <tags>
              <tag>latest</tag>
            </tags>
          </to>
          <container>
            <jvmFlags>
              <jvmFlag>-Xms512m</jvmFlag>
              <jvmFlag>-XX:+UnlockExperimentalVMOptions</jvmFlag>
              <jvmFlag>-XX:+UseCGroupMemoryLimitForHeap</jvmFlag>
              <jvmFlag>-Dspring.profiles.active=production</jvmFlag>
            </jvmFlags>
            <mainClass>com.zl.wzgp.Launcher</mainClass>
            <ports>
              <port>4000</port>
            </ports>
          </container>
        </configuration>
     </plugin>

Log output:

[DEBUG] TIMING Building and pushing image [INFO] Retrieving registry credentials for us.gcr.io… [DEBUG] TIMING Retrieving registry credentials for us.gcr.io [DEBUG] Using credentials from docker-credential-gcloud for us.gcr.io DEBUG] TIMING Retrieving registry credentials for us.gcr.io [DEBUG] Using credentials from docker-credential-gcloud for us.gcr.io [DEBUG] TIMED Retrieving registry credentials for us.gcr.io : 1161.0 ms [DEBUG] TIMING Authenticating push to us.gcr.io [DEBUG] TIMED Authenticating push to us.gcr.io : 1923.0 ms [INFO] Getting base image gcr.io/distroless/java:8… [DEBUG] TIMING Pulling base image manifest [DEBUG] TIMED Pulling base image manifest : 6022.0 ms [DEBUG] TIMING Preparing base image layer pullers [DEBUG] TIMED Preparing base image layer pullers : 0.0 ms [DEBUG] TIMING Pulling base image layer sha256:e8d8785a314f385d3675a017f4e2df1707c528c06e7a7989663fdab4900bd8ff [DEBUG] TIMED Pulling base image layer sha256:e8d8785a314f385d3675a017f4e2df1707c528c06e7a7989663fdab4900bd8ff : 714.0 ms [DEBUG] TIMING Pulling base image layer sha256:e005d777a298a3529b1c8cf890883359e050cc966089ce84fea4d17b111907db [DEBUG] TIMED Pulling base image layer sha256:e005d777a298a3529b1c8cf890883359e050cc966089ce84fea4d17b111907db : 709.0 ms [DEBUG] TIMING Pulling base image layer sha256:3e010093287c245d72a774033b4cddd6451a820bfbb1948c97798e1838858dd2 [DEBUG] TIMED Pulling base image layer sha256:3e010093287c245d72a774033b4cddd6451a820bfbb1948c97798e1838858dd2 : 717.0 ms [DEBUG] TIMING Pulling base image layer sha256:408ad02cbf46b38f36ac42c09d911cf359a5be4dc3fee6e5956b1e4f96e98bdf [DEBUG] TIMED Pulling base image layer sha256:408ad02cbf46b38f36ac42c09d911cf359a5be4dc3fee6e5956b1e4f96e98bdf : 964.0 ms [DEBUG] TIMING Preparing application layer builders [DEBUG] TIMED Preparing application layer builders : 1.0 ms [INFO] Building dependencies layer… [DEBUG] TIMING Building dependencies layer [DEBUG] TIMED Building dependencies layer : 16.0 ms [INFO] Building resources layer… [DEBUG] TIMING Building resources layer [DEBUG] Building resources layer built sha256:40230baf97c5b3d6f026fc9f734a12b9b485c7337d15c5c35e8a3e3ba7eab64f [DEBUG] TIMED Building resources layer : 56.0 ms [INFO] Building classes layer… [DEBUG] TIMING Building classes layer [DEBUG] TIMED Building classes layer : 3.0 ms [DEBUG] TIMING Building container configuration [INFO] [INFO] Container entrypoint set to [java, -Xms512m, -XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -Dspring.profiles.active=production, -cp, /app/resources:/app/classes:/app/libs/*, com.zl.wzgp.Launcher] [DEBUG] TIMED Building container configuration : 1.0 ms [DEBUG] TIMING Preparing layer pushers [DEBUG] TIMED Preparing layer pushers : 0.0 ms [DEBUG] TIMING Preparing layer pushers [DEBUG] TIMED Preparing layer pushers : 0.0 ms [DEBUG] TIMING Pushing BLOB digest: sha256:8b37076acdbeb0264388f03aefe4a023d47fc59c426df22c6a4268ae07b5d280, size: 39479382 [DEBUG] BLOB : digest: sha256:8b37076acdbeb0264388f03aefe4a023d47fc59c426df22c6a4268ae07b5d280, size: 39479382 already exists on registry [DEBUG] TIMED Pushing BLOB digest: sha256:8b37076acdbeb0264388f03aefe4a023d47fc59c426df22c6a4268ae07b5d280, size: 39479382 : 1143.0 ms [DEBUG] TIMING Pushing BLOB digest: sha256:40230baf97c5b3d6f026fc9f734a12b9b485c7337d15c5c35e8a3e3ba7eab22f, size: 4250 [ERROR] I/O error for image [us.gcr.io/zl-ln-wz/zl-wzgp]: [ERROR] Operation timed out (Read failed) [DEBUG] TIMED Pushing BLOB digest: sha256:40230baf97c5b3d6f026fc9f734a12b9b485c7337d15c5c35e8a3e3ba7eab22f, size: 4250 : 43745.0 ms [DEBUG] TIMING Pushing BLOB digest: sha256:c6c66cab87735c35fa5845823954ed52b01d1df7c7a064a655ae98eb1f2a11c6, size: 45543 [DEBUG] TIMING pushBlob [DEBUG] TIMING pushBlob POST sha256:c6c66cab87735c35fa5845823954ed52b01d1df7c7a064a655ae98eb1f2a11c6 [DEBUG] TIMED pushBlob PATCH sha256:c6c66cab87735c35fa5845823954ed52b01d1df7c7a064a655ae98eb1f2a11c6 : 427.0 ms [DEBUG] TIMED pushBlob PUT sha256:c6c66cab87735c35fa5845823954ed52b01d1df7c7a064a655ae98eb1f2a11c6 : 3646.0 ms [DEBUG] TIMED pushBlob POST sha256:c6c66cab87735c35fa5845823954ed52b01d1df7c7a064a655ae98eb1f2a11c6 : 8499.0 ms [DEBUG] TIMED pushBlob : 12572.0 ms [DEBUG] TIMED Pushing BLOB digest: sha256:c6c66cab87735c35fa5845823954ed52b01d1df7c7a064a655ae98eb1f2a99c6, size: 45543 : 54100.0 ms [DEBUG] TIMING Pushing container configuration [DEBUG] TIMING Pushing BLOB digest: sha256:dff9b703c0343d28d384747d74159f1a57cd947bd0abebc09d138ab6f448602f, size: 1736 [DEBUG] BLOB : digest: sha256:dff9b703c0343d28d384747d74159f1a57cd947bd0abebc09d138ab6f448602f, size: 1736 already exists on registry [DEBUG] TIMED Pushing BLOB digest: sha256:dff9b703c0343d28d384747d74159f1a57cd947bd0abebc09d138ab6f448602f, size: 1736 : 678.0 ms [DEBUG] TIMED Pushing container configuration : 715.0 ms [DEBUG] TIMED Building and pushing image : 112051.0 ms [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:54 min [INFO] Finished at: 2019-09-21T22:51:52+05:30 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.6.1:build (default-cli) on project us.gcr.io/zl-ln-wz/zl-wzgp: Operation timed out (Read failed) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.6.1:build (default-cli) on project us.gcr.io/zl-ln-wz/zl-wzgp: Operation timed out (Read failed) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288) at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: org.apache.maven.plugin.MojoExecutionException: Operation timed out (Read failed) at com.google.cloud.tools.jib.maven.BuildImageMojo.execute (BuildImageMojo.java:158) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288) at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: java.net.SocketException: Operation timed out (Read failed) at java.net.SocketInputStream.socketRead0 (Native Method) at java.net.SocketInputStream.socketRead (SocketInputStream.java:116) at java.net.SocketInputStream.read (SocketInputStream.java:171) at java.net.SocketInputStream.read (SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully (InputRecord.java:465) at sun.security.ssl.InputRecord.read (InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord (SSLSocketImpl.java:975) at sun.security.ssl.SSLSocketImpl.readDataRecord (SSLSocketImpl.java:933) at sun.security.ssl.AppInputStream.read (AppInputStream.java:105) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer (AbstractSessionInputBuffer.java:161) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer (SocketInputBuffer.java:82) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine (AbstractSessionInputBuffer.java:278) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead (DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead (DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse (AbstractMessageParser.java:259) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader (AbstractHttpClientConnection.java:286) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader (DefaultClientConnection.java:257) at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader (AbstractClientConnAdapter.java:230) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse (HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute (HttpRequestExecutor.java:125) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute (DefaultRequestDirector.java:684) at org.apache.http.impl.client.DefaultRequestDirector.execute (DefaultRequestDirector.java:486) at org.apache.http.impl.client.AbstractHttpClient.doExecute (AbstractHttpClient.java:835) at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:108) at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:56) at com.google.api.client.http.apache.ApacheHttpRequest.execute (ApacheHttpRequest.java:65) at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:1011) at com.google.cloud.tools.jib.http.Connection.send (Connection.java:197) at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call (RegistryEndpointCaller.java:270) at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.callWithAllowInsecureRegistryHandling (RegistryEndpointCaller.java:186) at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call (RegistryEndpointCaller.java:163) at com.google.cloud.tools.jib.registry.RegistryClient.callRegistryEndpoint (RegistryClient.java:474) at com.google.cloud.tools.jib.registry.RegistryClient.checkBlob (RegistryClient.java:330) at com.google.cloud.tools.jib.builder.steps.PushBlobStep.call (PushBlobStep.java:81) at com.google.cloud.tools.jib.builder.steps.PushLayerStep.call (PushLayerStep.java:95) at com.google.cloud.tools.jib.builder.steps.PushLayerStep.call (PushLayerStep.java:34) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:78) at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute (MoreExecutors.java:322) at java.util.concurrent.AbstractExecutorService.submit (AbstractExecutorService.java:134) at com.google.common.util.concurrent.AbstractListeningExecutorService.submit (AbstractListeningExecutorService.java:66) at com.google.common.util.concurrent.AbstractListeningExecutorService.submit (AbstractListeningExecutorService.java:36) at java.util.stream.ReferencePipeline$3$1.accept (ReferencePipeline.java:193) at java.util.Spliterators$ArraySpliterator.forEachRemaining (Spliterators.java:948) at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto (AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential (ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate (AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect (ReferencePipeline.java:566) at com.google.cloud.tools.jib.builder.steps.StepsRunner.scheduleCallables (StepsRunner.java:423) at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$pushApplicationLayers$11 (StepsRunner.java:360) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:69) at com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:78) at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute (MoreExecutors.java:322) at java.util.concurrent.AbstractExecutorService.submit (AbstractExecutorService.java:134) at com.google.common.util.concurrent.AbstractListeningExecutorService.submit (AbstractListeningExecutorService.java:66) at com.google.common.util.concurrent.AbstractListeningExecutorService.submit (AbstractListeningExecutorService.java:36) at com.google.cloud.tools.jib.builder.steps.StepsRunner.pushApplicationLayers (StepsRunner.java:358) at java.util.ArrayList.forEach (ArrayList.java:1257) at com.google.cloud.tools.jib.builder.steps.StepsRunner.run (StepsRunner.java:175) at com.google.cloud.tools.jib.api.Containerizer.run (Containerizer.java:324) at com.google.cloud.tools.jib.api.JibContainerBuilder.containerize (JibContainerBuilder.java:496) at com.google.cloud.tools.jib.api.JibContainerBuilder.containerize (JibContainerBuilder.java:477) at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild (JibBuildRunner.java:226) at com.google.cloud.tools.jib.maven.BuildImageMojo.execute (BuildImageMojo.java:108) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288) at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) [ERROR] [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[praveen4463]

@chanseokoh

It is unclear to me from which part of the log above or by what kind of reasoning you came to the conclusion that Jib got some kind of “response”

Sorry, I think its clear now that it failed before a chunk was filled into buffer, I read readLine/fillBuffer and came to conclusion that a response was received but it seems those several method calls during socket read operation that were on stack before the exception.

The blob check HTTP GET is only for checking if a layer exists on the registry. The server does not return any content but rather just returns an HTTP code. There isn’t a layer or something else to read. Jib just checks the return code.

Oh, I got you, didn’t know it checks just the return code, but I was also not specifically expecting it to read a returned layer, I thought it matches the hash of it that was returned by the server to see whether its stale or latest.

Thanks for those insights, I’ll try some more build and let you guys know if anything is reproducible given I get a similar issue.

Thanks for the workarounds as well, I’d implement the first one to invoke jib in a loop.

As I said, often you need to lift the timeouts imposed by the kernel. On my machine for example, I found the connection timeout is set to 30 seconds, so setting the timeout over 30 seconds at the JVM level is meaningless.

Yeah I remember you said that in last post, that’s a good piece of information. I’ll bump that too.

Thanks again!

[briandealwis]

I think @chanseokoh specifically means network unreliability.