Error: buildx call failed with: failed to solve: rpc error: code = Unknown desc = unexpected status: 403 Forbidden

Behaviour

A gha using buildx with push:true reports unexpected status: 403 Forbidden, after login to ghcr.io (which was ok).

Final lines in log before error were:

------
failed to solve: rpc error: code = Unknown desc = unexpected status: 403 Forbidden
Error: buildx call failed with: failed to solve: rpc error: code = Unknown desc = unexpected status: 403 Forbidden

Steps to reproduce this issue

1. Follow this tutorial to set up the gha (using build-push-action and buildx). Use the “push: true” setting.
2. Use a repository with a Dockerfile which generates a fairly large image (perhaps image size comes into play?)
3. Run a gha like this one: https://github.com/KTH-Library/kontarion/blob/master/.github/workflows/push-kontarion.yml#L63-L86 (but use “push:true” instead of that run command which uses “docker push” after exporting to a tarball and loading into the local registry).

Expected behaviour

No 403 Forbidden error? I’m not sure what causes that, I don’t think it is the CR_PAT token since the login to ghcr.io works.

Actual behaviour

I’m getting that 403. Perhaps some time out kicks in? Or maybe the docker image is too large for the runner? Not sure where to begin to investigate.

Configuration

• Repository URL (if public): https://github.com/KTH-Library/kontarion
• Build URL (if public): https://github.com/KTH-Library/kontarion/runs/1440993889?check_suite_focus=true

name: kontarion push

on:
  workflow_dispatch:
  push:
    paths:
    - '1.7.0/**'
    - '.github/workflows/push-kontarion.yml'

jobs:
  kontarion-push:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    -
      name: Set Environment Variables
      run: |
        IMG=kontarion
        OWNER="$(echo "${{ github.repository_owner }}" | tr '[:upper:]' '[:lower:]')"
        echo "BUILD_VER=1.0.$GITHUB_RUN_NUMBER" >> $GITHUB_ENV
        echo "IMG=${IMG}" >> $GITHUB_ENV
        echo "IMAGE=ghcr.io/${OWNER}/${IMG}" >> $GITHUB_ENV
        echo "BUILD_DATE=$(date +'%Y-%m-%d %H:%M:%S')" >> $GITHUB_ENV
        echo "GIT_SHA=$(echo ${{ github.sha }} | cut -c1-7)" >> $GITHUB_ENV
        echo "GIT_REF=$(git symbolic-ref -q --short HEAD || git describe --tags --exact-match)" >> $GITHUB_ENV
 
    -
      name: Set up QEMU
      id: qemu
      uses: docker/setup-qemu-action@v1
      with:
        image: tonistiigi/binfmt:latest
        platforms: all

    -
      name: Set up Docker Buildx
      id: buildx
      uses: docker/setup-buildx-action@master
      with:
        version: latest
        install: true

    -
      name: Cache Docker layers
      uses: actions/cache@v2
      with:
        path: /tmp/.buildx-cache
        key: ${{ runner.os }}-buildx-${{ github.sha }}
        restore-keys: |
          ${{ runner.os }}-buildx-

    -
      name: Login to Container Registry
      uses: docker/login-action@v1
      with:
        registry: ghcr.io
        username: ${{ github.repository_owner }}
        password: ${{ secrets.CR_PAT }}

    -
      name: Docker build and push
      uses: docker/build-push-action@v2
      with:
        context: ./1.7.0/
        file: ./1.7.0/Dockerfile
        builder: ${{ steps.buildx.outputs.name }}
        labels: |
          org.opencontainers.image.authors=${{ github.repository_owner }}
          org.opencontainers.image.created=${{ env.BUILD_DATE }}
          org.opencontainers.image.description=Created from commit ${{ env.GIT_SHA }} and ref ${{ env.GIT_REF }}
          org.opencontainers.image.ref.name=${{ env.GIT_REF }}
          org.opencontainers.image.revision=${{ github.sha }}
          org.opencontainers.image.source=https://github.com/${{ github.repository }}
          org.opencontainers.image.version=${{ env.BUILD_VER }}
        tags: |
          ${{ env.IMAGE }}:latest
          ${{ env.IMAGE }}:${{ env.GIT_REF }}
          ${{ env.IMAGE }}:${{ env.GIT_SHA }}
          ${{ env.IMAGE }}:${{ env.BUILD_VER }}
        push: true
        platforms: linux/amd64
        cache-from: type=local,src=/tmp/.buildx-cache
        cache-to: type=local,dest=/tmp/.buildx-cache

Workaround

I’m attempting to use a workflow file which instead sets “load: true” and then runs a docker push command, here is a section from the end of that workflow file:

    -
      name: Docker build and push
      uses: docker/build-push-action@v2
      with:
        context: ./1.7.0/
        file: ./1.7.0/Dockerfile
        builder: ${{ steps.buildx.outputs.name }}
        labels: |
          org.opencontainers.image.authors=${{ github.repository_owner }}
          org.opencontainers.image.created=${{ env.BUILD_DATE }}
          org.opencontainers.image.description=Created from commit ${{ env.GIT_SHA }} and ref ${{ env.GIT_REF }}
          org.opencontainers.image.ref.name=${{ env.GIT_REF }}
          org.opencontainers.image.revision=${{ github.sha }}
          org.opencontainers.image.source=https://github.com/${{ github.repository }}
          org.opencontainers.image.version=${{ env.BUILD_VER }}
        tags: |
          ${{ env.IMAGE }}:latest
          ${{ env.IMAGE }}:${{ env.GIT_REF }}
          ${{ env.IMAGE }}:${{ env.GIT_SHA }}
          ${{ env.IMAGE }}:${{ env.BUILD_VER }}
        load: true
        platforms: linux/amd64
        cache-from: type=local,src=/tmp/.buildx-cache
        cache-to: type=local,dest=/tmp/.buildx-cache
    - run: docker push ${{ env.IMAGE }}:latest

    -
      name: Image digest
      run: echo ${{ steps.docker_build.outputs.digest }}

This progresses a bit further with these messages in the log:

#22 exporting layers 539.5s done
#22 exporting manifest sha256:42674974463151fb0e3148dc84ed9af0bd2e792dd2aec3353da6767f70f242aa done
#22 exporting config sha256:9c3a0bc0b6bc49f64ccd6f8f38d3b91a5e004d66977d62dfcf1193c8fbbdb2f7 done
#22 sending tarball
#22 ...

#23 importing to docker
#23 DONE 0.0s

#22 exporting to oci image format
#22 sending tarball 136.4s done
#22 DONE 675.9s

#24 exporting cache
#24 preparing build cache for export 0.0s done
#24 writing layer sha256:08462c4da0eadbace59b3f09dc207e2ecf4e5f70c2c9b820082201ef98710eec
#24 writing layer sha256:08462c4da0eadbace59b3f09dc207e2ecf4e5f70c2c9b820082201ef98710eec 0.2s done
#24 writing layer sha256:0ff0f048790af67aa460f80f3232314f83f9c86ed19c8599cd53d9bbb1eb1103
#24 writing layer sha256:0ff0f048790af67aa460f80f3232314f83f9c86ed19c8599cd53d9bbb1eb1103 5.7s done
#24 writing layer sha256:127c9761dcbaa288abc58fc56437c2f2ffbe611b9f7f30e0b5b43cd348bb2094 done
#24 writing layer sha256:14409b438e8b0f4cd5b1110245de7d35a8c061b8c8f223884d99c6bd65f26e1f 0.0s done
#24 writing layer sha256:1f2fe70d116b95b311b68e88fd11aeccc245a4c69fb36479b52b1c031f0db62d
#24 writing layer sha256:1f2fe70d116b95b311b68e88fd11aeccc245a4c69fb36479b52b1c031f0db62d 11.8s done
#24 writing layer sha256:207bc9dc5200f94a1d8ef2a5b5a725f928be54d44f1ecdf51dbc82113ccaa598
#24 writing layer sha256:207bc9dc5200f94a1d8ef2a5b5a725f928be54d44f1ecdf51dbc82113ccaa598 0.0s done
#24 writing layer sha256:283f88d94097c44b33b48a39c27803d9f952f9087460fc0270b0991d5d8ad867
#24 writing layer sha256:283f88d94097c44b33b48a39c27803d9f952f9087460fc0270b0991d5d8ad867 0.6s done
#24 writing layer sha256:2f54329a6711f1e325c7d17e843ca546b01c1f504c546f23678c0a71e5147f91
#24 writing layer sha256:2f54329a6711f1e325c7d17e843ca546b01c1f504c546f23678c0a71e5147f91 17.3s done
#24 writing layer sha256:39ad6d9967d356670f59b4c1397613b5b1840bd915be68dab372844e5d00cb94
#24 writing layer sha256:39ad6d9967d356670f59b4c1397613b5b1840bd915be68dab372844e5d00cb94 36.4s done
#24 writing layer sha256:4039240d2e0b4bcb42ccbce75bc54570e471ad81457478de35fbeef63536e9c0
#24 writing layer sha256:4039240d2e0b4bcb42ccbce75bc54570e471ad81457478de35fbeef63536e9c0 done
#24 writing layer sha256:4a41b77c8e3fee3711347ffe7a4872ed41b6d91a2655b048a3c5ddb81597ddf9
#24 writing layer sha256:4a41b77c8e3fee3711347ffe7a4872ed41b6d91a2655b048a3c5ddb81597ddf9 2.2s done
#24 writing layer sha256:5025ef4ebbe82547d13bae1ce5db63aea6b496b1a44dfb443012fe49b52ae003
#24 writing layer sha256:5025ef4ebbe82547d13bae1ce5db63aea6b496b1a44dfb443012fe49b52ae003 done
#24 writing layer sha256:57346c02ff626ad57ac7c89f0087ae813c35777012a069e63e9598b6bfa25802 done
#24 writing layer sha256:722348be19cc015bddc73213b78bffb242533b76a41f8c1fa3908db2c90daf8e

But either it is stuck there or takes a long time to complete.

Logs

https://github.com/KTH-Library/kontarion/runs/1441299541?check_suite_focus=true

[atorosyan]

@mskyttner have you “Enabling improved container support” for your account and organisation?
https://docs.github.com/en/free-pro-team@latest/packages/guides/enabling-improved-container-support I had the same issue but it started to work after I followed the steps from the link.

[mskyttner]

@crazy-max thanks so much for the help on this, my “big build” now passes after I made two changes a) made it smaller by making a part into an optional install (conda) which effectively reduced the image size to 10GB uncompressed and b) did the settings you recommended with regards to registry cache. No other changes and I’m not sure which one kicked in, but now it pulled through. Travis build failed though, but that is another story. Merry xmas!