GAR error "cannot reuse body, request must be retried"

Troubleshooting

Before submitting a bug report please read the Troubleshooting doc.

• I have read TROUBLESHOOTING.md.

I wasn’t sure from the “Cannot push to a registry” section whether your preference is for issues to be opened here or in the linked tools’ repos.

Behaviour

Pushing image to GAR sometimes fails with error:

ERROR: failed commit on ref "manifest-sha256:0f17d0ea56a824cc2efa0ad1ded3e1dc540cdb3d0468feebb0b5506853c90f82": cannot reuse body, request must be retried

Steps to reproduce this issue

1. Make and push a commit
2. See build stage of build-push-action@v2 succeed
3. See push stage of build-push-action@v2 fail with above error

It seems only to happen for commits that only touch files excluded from the Docker build context. However, the action does occasionally succeed this type of commit.

Expected behaviour

Push stage should succeed.

Actual behaviour

Push stage fails with error.

Configuration

• Repository URL (if public): https://github.com/politics-rewired/fly-shortener
• Build URL (if public): https://github.com/politics-rewired/fly-shortener/runs/3461355947

name: CI/CD Pipeline

on:
  push:

jobs:
  publish-docker-image:
    if: github.ref == 'refs/heads/master'
    name: Publish Docker image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          version: latest
          buildkitd-flags: --debug

      # Caching strategy from: https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md#github-cache
      - name: Cache Docker layers for ${{ matrix.service }}
        uses: actions/cache@v2
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-fly-shortener-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-fly-shortener-

      - name: Get image tags
        id: image-tags
        run: |
          export GIT_SHA="${{ github.sha }}"
          export GIT_SHA_SHORT=${GIT_SHA:0:7}
          echo "::set-output name=sha::$GIT_SHA_SHORT"

      - name: Login to GAR
        uses: docker/login-action@v1
        with:
          registry: us-east4-docker.pkg.dev
          username: _json_key
          password: ${{ secrets.GCR_JSON_KEY }}

      - name: Build and push ${{ matrix.service }}
        id: docker_build
        uses: docker/build-push-action@v2
        with:
          push: true
          tags: |
            us-east4-docker.pkg.dev/assemble-services/apps/link-shortener:latest
            us-east4-docker.pkg.dev/assemble-services/apps/link-shortener:${{ steps.image-tags.outputs.sha }}
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new

      # Temp fix
      # https://github.com/docker/build-push-action/issues/252
      # https://github.com/moby/buildkit/issues/1896
      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

Logs

1_Set up job.txt 2_Checkout.txt 3_Set up Docker Buildx.txt 4_Cache Docker layers for.txt 5_Get image tags.txt 6_Login to GAR.txt 7_Build and push.txt 12_Post Build and push.txt 13_Post Login to GAR.txt 15_Post Set up Docker Buildx.txt 16_Post Checkout.txt 17_Complete job.txt

[sanchezpaco]

Hi, I’ve been struggling with this problem and the only solution I’ve found is to add a label in docker buildx command. Example:

docker --context $DOCKER_CONTEXT buildx build  --platform linux/amd64,linux/arm64 -t $CONTAINER_BASE_IMAGE --push --label "git-commit=$CI_COMMIT_SHA" .

In my case I was having the problem in 2 different pipelines that didn’t touch the docker build context. (first one worked and second one failed)

Maybe you’ll need to change CI_COMMIT_SHA to something like a timestamp.

Hope this helps!

[Sleepful]

Happened just now, “Re run all jobs” is good enough