setup-buildx-action hangs with 99% CPU when running on latest Github Action Runner v2.285.0
See original GitHub issueBehaviour
Our runners were auto-updated this morning from v2.284.0 to v2.285.0 and began to hang on the Download and Install Buildx
step:
Steps to reproduce this issue
- Upgrade your runner to version v2.285.0
- Run the
docker/setup-buildx-action@v1
in your workflow - You should see heavy CPU utilization that did not exist in v2.284.0
Expected behaviour
The step should complete in under 10s per our typical experience
Actual behaviour
The step is hanging indefinitely
Configuration
- Repository URL (if public): private
- Build URL (if public): private
Note: all steps prior to the failing step pass as expected
name: Test Environment
on:
pull_request:
types: [labeled, reopened, synchronize, ready_for_review]
workflow_dispatch:
env:
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
jobs:
build:
name: Build Image
runs-on: self-hosted
steps:
- name: Set outputs
id: vars
run: |
echo "::set-output name=head_sha::$(echo $PR_HEAD_SHA | cut -c1-7)"
env:
PR_HEAD_SHA: ${{ env.HEAD_SHA }}
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKER_HUB_USERNAME }}
password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: redacted
role-to-assume: redacted
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Set up Docker Context for Buildx
id: buildx-context
run: |
docker context create test-builder
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1
with:
endpoint: test-builder
- name: Build and push Docker images
uses: docker/build-push-action@v2.7.0
Logs
Download the log file of your build and attach it to this issue.
Our logs are littered with private, implementation details. Could you help me isolate which part of the logs you need for diagnosis?
2021-12-01T19:25:54.2902593Z Found online and idle self-hosted runner in the current repository's enterprise account that matches the required labels: 'self-hosted'
2021-12-01T19:25:54.4048239Z Waiting for the runner to pick up this job...
2021-12-01T19:26:54.7363687Z Job was not picked up by the preferred runner, try requesting additional runners to run this job
2021-12-01T19:26:54.8282602Z Waiting for a self-hosted runner to pickup this job...
2021-12-01T19:32:40.3175224Z Job is about to start running on the runner: self-hosted-runner-group-6zjgq-67xpv (enterprise)
2021-12-01T19:32:44.8744015Z Current runner version: '2.285.0'
2021-12-01T19:32:44.8753670Z Runner name: 'self-hosted-runner-group-6zjgq-67xpv'
2021-12-01T19:32:44.8755541Z Runner group name: 'self-hosted'
2021-12-01T19:32:44.8758083Z Machine name: 'self-hosted-runner-group-6zjgq-67xpv'
...
2021-12-01T19:32:49.5211515Z ##[group]Download and install buildx
2021-12-01T19:32:49.5825561Z Downloading https://github.com/docker/buildx/releases/download/v0.7.1/buildx-v0.7.1.linux-amd64
2021-12-01T20:12:24.6445002Z ##[error]The operation was canceled.
2021-12-01T20:12:24.6564749Z Post job cleanup.
2021-12-01T20:12:24.7568406Z Post job cleanup.
2021-12-01T20:12:24.9055991Z Post job cleanup.
2021-12-01T20:12:24.9780442Z Post job cleanup.
2021-12-01T20:12:25.0473555Z [command]/usr/local/bin/docker logout
2021-12-01T20:12:25.1428510Z Removing login credentials for https://index.docker.io/v1/
2021-12-01T20:12:25.1518891Z Cleaning up orphan processes
Issue Analytics
- State:
- Created 2 years ago
- Reactions:12
- Comments:61 (19 by maintainers)
Top Results From Across the Web
fs.copyFileSync hangs for file created within same ... - GitHub
setup-buildx-action hangs with 99 % CPU when running on latest Github Action Runner v2.285.0 docker/setup-buildx-action#117.
Read more >GitHub Actions step is hanging until timeout · Issue #1326
Describe the bug At some point, a GitHub Actions step stops producing output and hangs until the workflow is terminated by timeout.
Read more >fs.copyFileSync hangs for file created within same ... - GitHub
setup-buildx-action hangs with 99 % CPU when running on latest Github Action Runner v2.285.0 docker/setup-buildx-action#117.
Read more >The runner always in busy state even the job is finished. #700
I have self-hosted deployment in kubernetes. I had a deployment job ran failed, but the runner can't pickup new job, checked the runner...
Read more >A process hangs on Ubuntu sometimes #979 - GitHub
The problem is that Hugo hangs on runs-on: ubuntu-18.04 and 20.04 ... Does the Ubuntu VM of GitHub Actions runner has any limitation?...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
OK, what solved the issue for us was pinning the version to 287.1 and disabling the auto update of the runner:
@crazy-max If we add the extra
Install Docker Buildx
step the pipeline finished successfully.