The linux runner does not gracefully shutdown on SIGINT

Describe the bug According to https://github.com/actions/runner/issues/2190#issuecomment-1273302389, the runner should wait for the job is finished and then stop. But it doesn’t happen, and new tasks are assigned to the runner with an old PID.

To Reproduce Steps to reproduce the behavior:

1. Go to a runner with a running job and execute pgrep -af Runner.Listener
2. Remember it and execute pkill -INT Runner.Listener
3. Wait for a one task finished, and another one is started on the same runner.
4. The pgrep -af Runner.Listener does not change/is still there
5. The pkill -INT run.sh does not work as well

Expected behavior The runner should gracefully shutdown. If it does not by SIGINT, there must be another way to stop it right after the job is finished, before assigning a new one

Runner Version and Platform

Linux 2.298.2 amd64

What’s not working?

Please include error messages and screenshots.

[Felixoid]

The situation with ephemeral runners is a bit better than with normal. See the discussion in https://github.com/ClickHouse/ClickHouse/pull/49283. It’s not necessary to shut down the host, the process could restart.

It still doesn’t guarantee that you could tear down the runner process if there are no jobs assigned there for long, unfortunately. Imagine, we have a pool of 30 runners, and only 24 of them have running jobs. After some period, 60 seconds in our case, we shut down each one of them that still don’t have things to do. And at this moment, GH reports there was a job assigned to one of these poor runners.

It looks like GH assigns the jobs to runners, and not a runner is assigning a job by connecting to the API. If so, no matter what, there will be killed jobs.

All described above is my own conclusion. It’s based on the long time playing left and right with different schemes to get runners working reliably, but desperately failing again and again.

[asos-tommycouzens]

Would also love this!

We are in the process of setting up self hosted runners, and would like a safe way to scale down runners without causing running jobs to fail. Without the SIGTERM functionality as requested in this issue this is not possible without us building a quite complex and fragile orchestrator.

We similarly do not use ephermeal runners because we want to make use of docker caching. The availability of docker caching was the primary reason we chose github actions over azure devops.