Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Images created by Jib randomly causes 500 Internal server error in Red Hat Quay

See original GitHub issue


  • Jib version: 1.5.1
  • Build tool: Maven
  • OS: Linux and Windows

Description of the issue: Images created by Jib randomly causes 500 Internal server error in Red Hat Quay. This has been reproduced with the HelloWorld example from Jib, but does not happen with images created in any other way than with Jib.

Expected behavior: Images created by Jib should consistently be accepted by Quay.

Steps to reproduce:

  1. Download
  2. Change <image></image> to <image>QUAY-REGISTRY/image-built-with-jib</image>
  3. Run mvn compile jib:build

jib-maven-plugin Configuration:


Log output:

> [2019-09-13T07:50:47.986Z] [ERROR] Failed to execute goal (build_docker_image) on project xx: 500 Internal Server Error
> [2019-09-13T07:50:47.986Z] [ERROR] <html>
> [2019-09-13T07:50:47.986Z] [ERROR]   <head>
> [2019-09-13T07:50:47.986Z] [ERROR]     <title>Internal Server Error</title>
> [2019-09-13T07:50:47.986Z] [ERROR]   </head>
> [2019-09-13T07:50:47.986Z] [ERROR]   <body>
> [2019-09-13T07:50:47.986Z] [ERROR]     <h1>Internal Server Error</h1>
> [2019-09-13T07:50:47.986Z] [ERROR]
> [2019-09-13T07:50:47.986Z] [ERROR]   </body>
> [2019-09-13T07:50:47.986Z] [ERROR] </html>

Additional Information: Red Hat’s response is that this only happens with Jib-created images and therefore points the finger at Jib.

Is it possible to get Google and Red Hat to cooperate in finding a solution to this? Red Hat Case #: 02456747

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:42 (25 by maintainers)

github_iconTop GitHub Comments

kurtismullinscommented, Sep 19, 2019

Quay developer here; I just wanted to take a moment to chime in.


Probably because jib creates schema 2 images? Has quay rolled out schema 2 support fully yet?

Great catch! The service does not (yet) support Schema 2 images, yet. To the best of my knowledge, some on-premise installations may have this enabled depending on their configuration.

Our initial struggle was being able to reproduce this issue locally. There’s a ton of really great information in here that will get us one step closer. I, personally, have never heard of jib until this issue popped up so it’s all new ground for me.

The HTTP 500 response sounds like an issue on our end. I can’t promise when it will be fixed, but regardless of the payload received by Quay, it should respond with an appropriate status. I will do my best to provide updates if anyone is interested.

Please don’t hesitate to @ me if anyone has questions I might be able to answer.

kurtismullinscommented, Sep 23, 2019


I’ve been able to create the “Hello World” example using jib and push it to my local (development) Quay environment. Thanks to all for the reproduction steps!

Here’s some initial observations:

  • If Schema 2_2 is not enabled, I always get the (expected) “Manifest Invalid” error
  • On a new/fresh image build+push using jib, I am seeing an exception related to a chunk missing a newline terminator.

It’s still pretty premature but I believe that exception is not caught and handled correctly (hence the 5xx) and I’m not sure which chunk (if any?) is malformed. On my local system and example, the request which generates this error is:

PATCH /v2/devtable/image-built-with-jib-test/blobs/uploads/bcab81cf-b0d9-4972-b4a6-63ba87d13da5 HTTP/1.1

Moving forward, I’m digging through the raw requests using Wireshark to see if I can find a malformed chunk. My goal is to produce a testable scenario to trigger this exception so that we can at least get it handled properly on Quay’s end. If I find a malformed request being generated by jib and am able to reproduce it, I’ll share that information as well.

For reference purposes, here’s essentially my current test-case:

  • Clone jib (working directly out of master)
  • Point the helloWorld example to my Quay instance
  • Execute mvn -Djib.console=plain -Djava.util.logging.config.file=./ -DsendCredentialsOverHttp=true compile jib:build
  • Modify and the repository’s name in pom.xml for each fresh run.

Note: After encountering the HTTP 500 error initially for a given image, a subsequent retry succeeds. Hence the purpose of my changes to pom.xml and in my test-case.

I am far from a Java, maven, or jib expert so please don’t hesitate to share suggestions if you have them. I had hoped I could use mitmproxy but neither the command-line arguments (e.g. -Dhttp.proxyHost [...]) or the maven settings file seemed to have worked. It is probably related to my local configuration and not using TLS/HTTPs.

Read more comments on GitHub >

github_iconTop Results From Across the Web

500 Internal Server Error while copying multi-arch image into ...
It works well on a local registry but not on the Openshift cluster internal registry. Environment. Red Hat OpenShift Container Platform (RHOCP).
Read more >
Jib & Quay: 500s, 400s, and 401s - Google Groups
The essential issues: - On Quay v3.0.5 with the Jib "helloworld" example, Quay returns HTTP 500 responses. One theory is that this is...
Read more >
OpenShift Maven Plugin - JKube - Eclipse
How to configure image name generated by Eclipse JKube? 17. Appendix ... to build and deploy your Java application with Red Hat OpenShift...
Read more >
Home of Quarkus Cheat-Sheet - GitHub Pages
application , if container image is created (JiB, Docker extensions) is ... There is a Quarkus Test Resource that starts and stops H2...
Read more >
Tag Latest Not Found In Repository Docker.Io/Library/Centos ...
Io/Library/Centos On Redhat Enterprise Linux Behind Proxy Server ... In RHEL 8.0 and 8.1, is listed before in the similar images...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found