question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

setup-buildx-action hangs with 99% CPU when running on latest Github Action Runner v2.285.0

See original GitHub issue

Behaviour

Our runners were auto-updated this morning from v2.284.0 to v2.285.0 and began to hang on the Download and Install Buildx step:

Screen Shot 2021-12-01 at 5 24 52 PM

Steps to reproduce this issue

  1. Upgrade your runner to version v2.285.0
  2. Run the docker/setup-buildx-action@v1 in your workflow
  3. You should see heavy CPU utilization that did not exist in v2.284.0 image

Expected behaviour

The step should complete in under 10s per our typical experience

Actual behaviour

The step is hanging indefinitely

Configuration

  • Repository URL (if public): private
  • Build URL (if public): private

Note: all steps prior to the failing step pass as expected

name: Test Environment

on:
  pull_request:
    types: [labeled, reopened, synchronize, ready_for_review]
  workflow_dispatch:

env:
  HEAD_SHA: ${{ github.event.pull_request.head.sha }}

jobs:
  build:
    name: Build Image
    runs-on: self-hosted
    steps:
      - name: Set outputs
        id: vars
        run: |
          echo "::set-output name=head_sha::$(echo $PR_HEAD_SHA | cut -c1-7)"
        env:
          PR_HEAD_SHA: ${{ env.HEAD_SHA }}

      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKER_HUB_USERNAME }}
          password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: redacted
          role-to-assume: redacted

      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1

      - name: Set up Docker Context for Buildx
        id: buildx-context
        run: |
          docker context create test-builder

      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@v1
        with:
          endpoint: test-builder

      - name: Build and push Docker images
        uses: docker/build-push-action@v2.7.0

Logs

Download the log file of your build and attach it to this issue.

Our logs are littered with private, implementation details. Could you help me isolate which part of the logs you need for diagnosis?

2021-12-01T19:25:54.2902593Z Found online and idle self-hosted runner in the  current repository's enterprise account that matches the required labels: 'self-hosted'
2021-12-01T19:25:54.4048239Z Waiting for the runner to pick up this job...
2021-12-01T19:26:54.7363687Z Job was not picked up by the preferred runner, try requesting additional runners to run this job
2021-12-01T19:26:54.8282602Z Waiting for a self-hosted runner to pickup this job...
2021-12-01T19:32:40.3175224Z Job is about to start running on the runner: self-hosted-runner-group-6zjgq-67xpv (enterprise)
2021-12-01T19:32:44.8744015Z Current runner version: '2.285.0'
2021-12-01T19:32:44.8753670Z Runner name: 'self-hosted-runner-group-6zjgq-67xpv'
2021-12-01T19:32:44.8755541Z Runner group name: 'self-hosted'
2021-12-01T19:32:44.8758083Z Machine name: 'self-hosted-runner-group-6zjgq-67xpv'
...
2021-12-01T19:32:49.5211515Z ##[group]Download and install buildx
2021-12-01T19:32:49.5825561Z Downloading https://github.com/docker/buildx/releases/download/v0.7.1/buildx-v0.7.1.linux-amd64
2021-12-01T20:12:24.6445002Z ##[error]The operation was canceled.
2021-12-01T20:12:24.6564749Z Post job cleanup.
2021-12-01T20:12:24.7568406Z Post job cleanup.
2021-12-01T20:12:24.9055991Z Post job cleanup.
2021-12-01T20:12:24.9780442Z Post job cleanup.
2021-12-01T20:12:25.0473555Z [command]/usr/local/bin/docker logout 
2021-12-01T20:12:25.1428510Z Removing login credentials for https://index.docker.io/v1/
2021-12-01T20:12:25.1518891Z Cleaning up orphan processes

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:12
  • Comments:61 (19 by maintainers)

github_iconTop GitHub Comments

4reactions
afritzlercommented, Mar 7, 2022

OK, what solved the issue for us was pinning the version to 287.1 and disabling the auto update of the runner:

image: ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:v2.287.1-ubuntu-20.04
env:
  # Disables automatic runner updates
  - name: DISABLE_RUNNER_UPDATE
    value: "true"
4reactions
jnsvdcommented, Dec 2, 2021

@crazy-max If we add the extra Install Docker Buildx step the pipeline finished successfully.

Read more comments on GitHub >

github_iconTop Results From Across the Web

fs.copyFileSync hangs for file created within same ... - GitHub
setup-buildx-action hangs with 99 % CPU when running on latest Github Action Runner v2.285.0 docker/setup-buildx-action#117.
Read more >
GitHub Actions step is hanging until timeout · Issue #1326
Describe the bug At some point, a GitHub Actions step stops producing output and hangs until the workflow is terminated by timeout.
Read more >
fs.copyFileSync hangs for file created within same ... - GitHub
setup-buildx-action hangs with 99 % CPU when running on latest Github Action Runner v2.285.0 docker/setup-buildx-action#117.
Read more >
The runner always in busy state even the job is finished. #700
I have self-hosted deployment in kubernetes. I had a deployment job ran failed, but the runner can't pickup new job, checked the runner...
Read more >
A process hangs on Ubuntu sometimes #979 - GitHub
The problem is that Hugo hangs on runs-on: ubuntu-18.04 and 20.04 ... Does the Ubuntu VM of GitHub Actions runner has any limitation?...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found