jib often fails with "imagestreammappings: the object has been modified"

Environment:

• Jib version: 2.7.1
• Build tool: Gradle 6.7.1
• OS: CentOS

Description of the issue:

Very often (around every third build) gradlew jib fails with:

> Task :productservice:service:jib

Containerizing application to openshiftregistry.scoop-gmbh.de/xxxxx/product-service, openshiftregistry.scoop-gmbh.de/xxxxx/product-service:prod...
Base image 'adoptopenjdk/openjdk15:centos' does not use a specific image digest - build may not be reproducible
Using credentials from to.auth for openshiftregistry.scoop-gmbh.de/xxxxx/product-service
Getting manifest for base image adoptopenjdk/openjdk15:centos...
Building dependencies layer...
Building resources layer...
Building classes layer...
Building project dependencies layer...
The base image requires auth. Trying again for adoptopenjdk/openjdk15:centos...
Using base image with digest: sha256:065a8148eec0bf34dc093da7a8e50828c416e1838cd2f52f3d080564832ee56a

Container entrypoint set to [java, -Dfile.encoding=UTF-8, -XX:-OmitStackTraceInFastThrow, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/tmp, -XX:MaxRAMPercentage=90, --enable-preview, -XX:+UseShenandoahGC, -cp, /app/resources:/app/classes:/app/libs/*, de.scoopsoftware.product_service.Main]

> Task :productservice:service:jib FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':productservice:service:jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: 500 Internal Server Error
  PUT https://openshiftregistry.scoop-gmbh.de/v2/xxxxx/product-service/manifests/prod
  {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"ErrStatus":{"metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on imagestreammappings.image.openshift.io \"product-service\": the object has been modified; please apply your changes to the latest version and try again","reason":"Conflict","details":{"name":"product-service","group":"image.openshift.io","kind":"imagestreammappings"},"code":409}}}]}

Expected behavior:

Jib is the only process pushing to our registry, so this shouldn’t fail with “the object has been modified”.

jib-gradle-plugin Configuration:

    String now = ISO_OFFSET_DATE_TIME.format(now())
    String buildNumber = ofNullable(getenv("BUILD_NUMBER")).orElse("local")
    String gitCommit = getenv("BUILD_NUMBER") != null ? "git rev-parse HEAD".execute().text.trim() : "unknown"
    jib {
        container {
            mainClass = 'de.scoopsoftware.product_service.Main'
            ports = ['8180', '8181']
            jvmFlags = ['-Dfile.encoding=UTF-8', '-XX:-OmitStackTraceInFastThrow', '-XX:+HeapDumpOnOutOfMemoryError', '-XX:HeapDumpPath=/tmp', '-XX:MaxRAMPercentage=90', '--enable-preview', '-XX:+UseShenandoahGC']
            filesModificationTime = now
            environment = [
                    APP_STORAGE: '/data',
            ]
            labels = [
                    'de.scoopgmbh.proavo.build-date': now,
                    'de.scoopgmbh.proavo.build-number': buildNumber,
                    'de.scoopgmbh.proavo.git-commit': gitCommit,
            ]
        }
        from {
            image = 'adoptopenjdk/openjdk15:centos'
        }
        to {
            image = 'openshiftregistry.scoop-gmbh.de/xxxxx/product-service'
            tags = [getenv('DOCKER_IMAGE_TAG') ?: 'latest']
            auth {
                username = getenv('DOCKER_USERNAME')
                password = getenv('DOCKER_PASSWORD')
            }
        }
    }

[josephschorr]

@chanseokoh I’m no longer affiliated with Red Hat, so my (wildly uninformed) guess is that the OpenShift Registry (which last I checked was backed by Kubernetes itself) might have tried to update the underlying Image Kub object after it had already been changed, which would definitely be a bug in that implementation.

I may be wildly off though 😃

[dmoebius]

omg, you’re right! No, it’s not intended, that “:latest” is implied here. I thought Jib would push to only to the specified tags, which is either getenv('DOCKER_IMAGE_TAG') (if DOCKER_IMAGE_TAG is defined) or “latest”. I wasn’t aware that the “tags” property only allows to specify additional tags. (The docs say so; I have overlooked this.) Many thanks for the hint!

So, the correct solution imho is not to specify “tags” at all, but:

image = `openshiftregistry.scoop-gmbh.de/xxxxx/product-service:${getenv('DOCKER_IMAGE_TAG') ?: 'latest'}`

This also explains why pushing to latest never fails; it only fails when DOCKER_IMAGE_TAG != 'latest'.