Action is encountering internal issue and sits in a retry loop(?), chews up action minutes

This was working fine 7 days ago and there have been no changes to our workflows however now this action is having some issue occur internally which results in it sitting in what appears to be a retry loop or its hung. This has cost us a few hours total of paid action minutes.

Behaviour

Steps to reproduce this issue

1. Repo has 2 dockerfiles (lets call them A and B), CI workflow has 6 jobs.
2. Workflow (that last passed on Nov 7th)
   1. Code is linted then tested
   2. If lint/test is successful, container A builds a dev and test tag, push
   3. If lint/test is successful, container B builds a dev and test tag, push
3. Note that 2.2 and 2.3 are occurring in parallel in the workflow

Expected behavior

This was originally (as of the 7th of Nov) working fine - the workflow would complete as expected and containers A and B would be published to the GHCR, both tagged dev and test correctly.

Actual behavior

Workflow runs today are now getting stuck in what appears to be a retry loop(?) and only one of the two parallel jobs will pass, the other gets stuck in a loop(?) and need to be cancelled.

When the jobs are cancelled, the job log has the following:

2021-11-13T02:04:53.2663646Z #19 exporting to image
2021-11-13T02:04:53.7166729Z #19 5.856 error: failed to copy: failed to do request: Put "https://ghcr.io/v2/[redacted]/blobs/upload/[uuid]?digest=sha256...": write tcp 172.17.0.2:42834->140.82.112.34:443: write: connection reset by peer
2021-11-13T02:04:53.7170084Z #19 5.856 retrying in 1s
2021-11-13T02:07:44.4163623Z ##[error]The operation was canceled.

Given the timestamps, it’s clearly not retrying in on second … or at all for that matter.

Run examples:

• A passed, B ran for 13 min before cancelled
• Retried the above workflow which resulting in B passing while A ran for 5 min before cancelled.

Configuration

• Repository URL (if public): not public
• Build URL (if public): not public

YAML workflow file

name: Push CI

on:
  push:
    branches:
      - "*"
    tags-ignore:
      - "*.*.*"

jobs:
  lint:
    name: Lint
    runs-on: ubuntu-latest
    if: github.event_name == 'push'
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Setup Node
        uses: actions/setup-node@v2
        with:
          cache: "npm"

      - name: Install dependencies
        run: npm ci

      - name: Lint
        run: npm run lint

  test:
    name: Test
    runs-on: ubuntu-latest
    if: github.event_name == 'push'
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Setup Node
        uses: actions/setup-node@v2
        with:
          cache: "npm"

      - name: Install dependencies
        run: npm ci

      - name: Test
        run: npm run test
        env:
          CI: true

  build-api-docker-development:
    name: Build and push development API docker image
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/master'
    needs:
      - lint
      - test
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v3
        with:
          images: ghcr.io/[redacted]
          tags: |
            type=raw,value=dev
          flavor: |
            latest=false

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_PACKAGES_TOKEN }}

      - name: Docker build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          file: "./api.Dockerfile"
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          build-args: |
            APP_VERSION_ENV=dev
            APP_VERSION_SHA=${{ github.sha }}

  build-bootstrapper-docker-development:
    name: Build and push development Bootstrapper docker image
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/master'
    needs:
      - lint
      - test
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v3
        with:
          images: ghcr.io/[redacted]
          tags: |
            type=raw,value=dev
          flavor: |
            latest=false

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_PACKAGES_TOKEN }}

      - name: Docker build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          file: "./bootstrapper.Dockerfile"
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

  build-api-docker-test:
    name: Build and push test API docker image
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/master'
    needs:
      - build-api-docker-development
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v3
        with:
          images: ghcr.io/[redacted]
          tags: |
            type=raw,value=test
          flavor: |
            latest=false

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_PACKAGES_TOKEN }}

      - name: Docker build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          file: "./api.Dockerfile"
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          build-args: |
            APP_VERSION_ENV=test
            APP_VERSION_SHA=${{ github.sha }}

  build-bootstrapper-docker-test:
    name: Build and push test Bootstrapper docker image
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/master'
    needs:
      - build-bootstrapper-docker-development
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v3
        with:
          images: ghcr.io/[redacted]
          tags: |
            type=raw,value=test
          flavor: |
            latest=false

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_PACKAGES_TOKEN }}

      - name: Docker build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          file: "./bootstrapper.Dockerfile"
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

[South-Paw]

Cool, thanks @crazy-max

Puts things a bit more at ease having it confirmed to be getting handled upstream 😃

[crazy-max]

Possibly related: containerd/containerd#6242

Yes that’s it, tracked in https://github.com/docker/buildx/issues/834 and PR opened on BuildKit repo: https://github.com/moby/buildkit/pull/2461. In the meantime use the solution from https://github.com/docker/buildx/issues/834#issuecomment-965730742.