How to handle upgrade with ephemeral

To support ephemeral runners as docker containers, we created an init script which runs the following:

/config.sh <arguments>
/run.sh

We’ve noticed that if the actions runner version is old, the runner will self update then exit without actually running a job or de-registering. When there is no upgrade the runner works correctly.

We are using the latest ubuntu docker image as a base.

Here is the log from the container:

# Authentication
√ Connected to GitHub
# Runner Registration
√ Runner successfully added
√ Runner connection is good
# Runner settings
√ Settings Saved.
√ Connected to GitHub
2021-10-01 20:50:54Z: Listening for Jobs
Runner update in progress, do not shutdown runner.
Downloading 2.283.2 runner
Waiting for current job finish running.
Generate and execute update script.
Runner will exit shortly for update, should be back online within 10 seconds.
√ Removed .credentials
√ Removed .runner

Is there a way to either skip the upgrade or have the runner process a job?

[ethomson]

We’re adding an option to allow self-hosted ephemeral runners to opt-out of automatic updates so that you can manage updates yourself.

Some background: we consider the runner software and the hosted Actions software as a cohesive whole. Many times when we add a new feature to GitHub Actions, these changes need to be made both on the hosted service and in the runner - for example, when we added conditional steps to composite actions. This is why we’ve always required runner updates, so that we can be sure that the runner is compatible with the service version.

Obviously this is a painful requirement for many ephemeral users. So we’ll add an opt-out mechanism for ephemeral, where the runner will not try to do a self-update. This flag will allow you to control when you update your runners.

Because the runner versions are so tightly coupled to the overall service, you’ll be required to update within a month of a new runner version being released. After a month, your runners will no longer be able to connect to GitHub, so you will need to perform updates regularly. Immediately upon a new release, the runner will begin notifying you when an update is available on stdout and stderr. We’ll also start adding annotations to workflow runs on outdated runners.

This is in development now and we plan to have it generally available in the new year.

[cgeers]

@TingluoHuang #1384 seems relevant, but I think the crux of this problem is that the auto update procedure creates a new process before the old one waits around for a few seconds and exits. The old process does not appear to check for success on upgrade, it just waits a fixed time and exits. In a containerized runner, when this happens all processes in the container are killed, whether or not the upgrade actually had time to complete. I see this routinely.

This changing pid throughout the upgrade procedure doesn’t play well with containerized runners that don’t also embed their own service manager (systemd). This is why I question whether ephemeral runners should be subject to auto upgrade at all.

People are expecting the —ephemeral option to finally bring with it basic support for orchestrating containerized runners and I’m not sure it does just yet. The design of the upgrade process seems to be a blocker.

personally, I’d be fine with a —autoupdate=false option being available either with or without —ephemeral.