Jib uses wrong tag latest and gets error BLOB_UPLOAD_INVALID Repository does not allow updating assets

Environment:

• Jib version: 2.1.0
• Build tool: Maven 3.6.1
• OS: Linux

Description of the issue: Jib tries to update tag latest in non redeploy repository and catches exception {“errors”:[{“code”:“BLOB_UPLOAD_INVALID”,“message”:“blob upload invalid”,“detail”:“Repository does not allow updating assets: docker_default”

But I didn’t specify tag latest in my configuration.

Expected behavior: Upload success with tag v.0.0.5.5

Steps to reproduce:

1. Run mvn compile jib:build

jib-maven-plugin Configuration:

<plugin>
  <groupId>com.google.cloud.tools</groupId>
  <artifactId>jib-maven-plugin</artifactId>
  <version>2.1.0</version>
  <configuration>
      <from>
          <image>private.registry/openjdk18</image>
          <auth>
              <username>${env.PRIVATE_REGISTRY_USR}</username>
              <password>${env.PRIVATE_REGISTRY_PSW}</password>
          </auth>
      </from>
      <to>
          <image>private.registry/myapp</image>
          <tags>
              <tag>v.0.0.5.5</tag>
          </tags>
          <auth>
              <username>${env.PRIVATE_REGISTRY_USR}</username>
              <password>${env.PRIVATE_REGISTRY_PSW}</password>
          </auth>
      </to>
  </configuration>
</plugin>

Log output: Note that output log with wrong tag latest:

[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.1.0:build (default-cli) on project myapp: Tried to push image manifest for private.registry/myapp:latest but failed because: other: blob upload invalid | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new: 400 Bad Request
[ERROR] {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"blob upload invalid","detail":"Repository does not allow updating assets: docker_default"}]}

Additional Information: On version Jib 1.6.1 log output with normal tag but the same error:

[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.6.1:build (default-cli) on project myapp: Tried to push image manifest for private.registry/myapp:v.0.0.5.5 but failed because: other: blob upload invalid | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new: 400 Bad Request
[ERROR] {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"blob upload invalid","detail":"Repository does not allow updating assets: docker_default"}]}

registry - sonatype nexus, repository with “Disable redeploy” option

docker cli uploads this image correctly

When admin of registry deletes on his side tag latest after that jib can push image but only once because tag latest comes back and it can’t be redeployed.

[gavriluk]

I really appreciate you are answering on open source issues!!! Thank you for your help!

[chanseokoh]

This makes sense and was actually what I suspected.

I tested the “disable redeploy” option with Docker CLI, and I convinced myself that Sonatype Nexus should resolve this issue. For the Docker Registry V2 API, I believe the only way to “add a new tag to an existing manifest” is to put the same manifest (but with a different tag) through PUT /v2/<name>/manifests/<reference>. (Or, you could argue that the Registry API could add a new API for just adding a new tag.) That is, Sonatype Nexus should not return BLOB_UPLOAD_INVALID but proceed to add a new tag, because the operation is not modifying the manifest but merely having another tag.

However, I did notice that Docker CLI works. But the way it works is very unusual.

$ docker tag ubuntu:18.04 localhost:5000/ubuntu:tag1
$ docker tag ubuntu:18.04 localhost:5000/ubuntu:tag2
$ docker tag ubuntu:18.04 localhost:5000/ubuntu:tag3
$ docker push localhost:5000/ubuntu:tag1
The push refers to repository [localhost:5000/ubuntu]
e0b3afb09dc3: Pushed 
6c01b5a53aac: Pushed 
2c6ac8e5063e: Pushed 
cc967c529ced: Pushed 
tag1: digest: sha256:134c7fe821b9d359490cd009ce7ca322453f4f2d018623f849e580a89a685e5d size: 1152
$ docker push localhost:5000/ubuntu:tag2
The push refers to repository [localhost:5000/ubuntu]
e0b3afb09dc3: Layer already exists 
6c01b5a53aac: Layer already exists 
2c6ac8e5063e: Layer already exists 
cc967c529ced: Layer already exists 
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the localhost:5000 registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
tag2: digest: sha256:c1719faa7caaca0273424beb7e316d474cc072193046e186a396f17edf034ebf size: 4925
$ docker push localhost:5000/ubuntu:tag3
The push refers to repository [localhost:5000/ubuntu]
e0b3afb09dc3: Layer already exists 
6c01b5a53aac: Layer already exists 
2c6ac8e5063e: Layer already exists 
cc967c529ced: Layer already exists 
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the localhost:5000 registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
tag3: digest: sha256:a8c06373ac1fd2556ee1620e74026b57827902612ea920de2c2570da9c5ce92c size: 4925

As observed, the second tag2 and third tag3 pushes return the following deprecation warning: [DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the localhost:5000 registry NOW to avoid future disruption. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/

It led to me think that Docker CLI side-stepped the Sonatype Nexus issue by pushing a deprecated schema1 manifest (UPDATE: as of now, Docker CLI finally removed schema1 support, so Docker CLI now fails too.) that is different from the schema2 manifest first pushed with tag1. I looked into the registry, and indeed, it had created three manifests:

The first one is the schema2 manifest of size 1.1KB:

The other two are schema1 manifests whose size is 5 times of the first one:

Next, I enabled redeploying on Nexus and pushed yet another tag. As I expected, it did not print the schema1 deprecation warning.

$ docker tag ubuntu:18.04 localhost:5000/ubuntu:tag4
$ docker push localhost:5000/ubuntu:tag4
The push refers to repository [localhost:5000/ubuntu]
e0b3afb09dc3: Layer already exists 
6c01b5a53aac: Layer already exists 
2c6ac8e5063e: Layer already exists 
cc967c529ced: Layer already exists 
tag4: digest: sha256:134c7fe821b9d359490cd009ce7ca322453f4f2d018623f849e580a89a685e5d size: 1152

Checking the manifest list on the server, I verified that it did not create a new manifest this time, as expected. The registry already has the schema2 manifest, so the server just had to add the new tag to it. Now I see three manifests and four tags:

FWIW, the next Docker CLI major version plans to drop the support for the deprecated schema1, so I expect Docker CLI will also stop working eventually.

Given all these, I think Sonatype Nexus should fix this issue so that the “disable redeploy” option works when adding a new tag. They could argue that adding a new tag is a kind of “redeploying” and preventing it is “working as intended.” However, given that Docker CLI works at the moment (UPDATE: as of now, Docker CLI finally removed schema1 support, so Docker CLI now fails too.)