Self-hosted runner with Docker step creates files that trip up the checkout step

Describe the bug When using self-hosted runners, git checkouts are cached between runs (this is nice, because it greatly speeds up our builds).

However, if a docker-based step writes a file to the workspace it will (possibly) be owned by the root user. If the permissions don’t give the (non-root) action runner user +w permission then a checkout step in a future workflow run will fail to remove this file. The first time, the error will look like this:

##[group]Cleaning the repository
[command]/usr/bin/git clean -ffdx
warning: could not open directory 'foo/': Permission denied
warning: failed to remove foo/: Directory not empty
##[endgroup]
##[warning]Unable to clean or reset the repository. The repository will be recreated instead.
Deleting the contents of '/home/jparker/actions-runner/_work/self-hosted-runner-permissions-issue-repro/self-hosted-runner-permissions-issue-repro'
##[error]Command failed: rm -rf "/home/jparker/actions-runner/_work/self-hosted-runner-permissions-issue-repro/self-hosted-runner-permissions-issue-repro/foo"
rm: cannot remove '/home/jparker/actions-runner/_work/self-hosted-runner-permissions-issue-repro/self-hosted-runner-permissions-issue-repro/foo': Permission denied

So git clean -ffdx tried to stat() this foo/ directory (created via a container in a previous build) but failed. It was then unable to remove the directory because it wasn’t empty. It tried to fall back to rm -rf which failed for the same reasons.

In future builds it goes straight to rm -rf because the .git folder did get cleaned up. It continues to fail in the same way for all future builds. Here’s a screenshot:

To Reproduce

I’ve created a repo that reproduces the error: https://github.com/Brightspace/self-hosted-runner-permissions-issue-repro

Here’s an example of a workflow failing: https://github.com/Brightspace/self-hosted-runner-permissions-issue-repro/runs/596011452?check_suite_focus=true

Expected behavior

I guess I’d expect all the files to be owned by the runner user… in a perfect world. Maybe that can be done with user namespace maps? Documentation. Not sure what that would entail though or if it makes sense for what the runner is doing.

I think this is not an issue with the checkout action because I don’t think there is anything they could do about it - it’d impact other actions too, checkout was just the first one I hit the issue with.

Runner Version and Platform

Ubuntu 18.04, runner version 2.168.0

These are org-level runners but I imagine it’s not specific to that.

[jsmartt]

Everyone’s comments above are appreciated greatly. I prefer to run everything in a container to keep the build server’s environment as clean as possible, and I ended up with the following at the start of my jobs section:

jobs:
  build_and_test:
    runs-on: [ self-hosted ]
    container:
      image: ubuntu

    steps:
      - name: Clean the workspace
        run: rm -rf $GITHUB_WORKSPACE/*

      - uses: actions/checkout@v2

      ...

If a job container is not specified, you can use this as a cleanup step:

- name: Clean the workspace
  uses: docker://alpine
  with:
    args: /bin/sh -c "rm -rf /github/workspace/.* || rm -rf /github/workspace/*"

A few additional thoughts:

1. Coming from drone.io, it’s a bit disappointing that environment pollution is even something that needs solved manually with GH actions (with containers), but I don’t think that starting the runner as root is the right solution; this likely isn’t even an option in many enterprise environments.
2. The checkout action really ought to handle the cleanup. If it’s running in a container because of a job container specification, it should have the same permissions to modify/delete the files as the commands that created them.
3. None of this would be an issue if a temporary docker volume was used instead of volume-mounting the workspace dir on the runner host. Any additional mounted volumes can be specified manually, but if you’re running everything in a container, I’m not sure why you’d want them. The whole reason I want to run everything in a container is to start with a clean slate, not a workspace contaminated by other builds.
   • The checkout action could even run in a container itself, preventing the need to have git 2.18+ installed on the runner. (The ability to specify a cert bundle would be critical for GHES customers though.)

[MuchToKnow]

Also, I don’t think we can say must run runner as root since many folks run as systemd service and I don’t think that will allow us to run as root.

Ran into this myself today due to running python in docker containers during a test step, creating root-owned pycache files. These files then break the next build when the runner attempts removal during the checkout step like OP.

Are there any issues with running the service as root utilizing the [user] param for install? I’m hosting a runner on ubuntu 20.04.1 and running:

sudo ./svc.sh install root
sudo ./svc.sh start

works well for me as a workaround for now.