Feature request: provide a signal to other programs when the runner has started a job

It would be helpful if the runner could signal that it has picked up a job to other programs.

We intend to use ephemeral runners (#510). We want to have an auto-scale-group in AWS that holds spare/idle capacity. When a job starts we want to remove the runner from that ASG so that it gets replaced in parallel with the job. The alternative is to scale up only after the job ends (and the machine terminates) but this adds latency, and if we get a burst of jobs (that uses up all of our capacity) we would get excessive queueing (and if the jobs are long-running it could get really bad.)

(You don’t care about these details) when we detect a job has started we trigger a Lambda function that looks at the caller identity and removes the caller from its ASG (we do it this way so that the runner VM has very narrow AWS permissions).

How we’ve worked around this: with --once (which we tried using before noticing #510) we were looking at the runner stdout for the “Job started” message. This isn’t the classiest thing to do (but it at least gracefully degrades – if the message were to change then we would just have more build queueing). Any design you come up with would be fine for us.

[thboop]

We’ve shipped a beta of this functionality in 2.289.1, please try it out and provide any feedback you have on the adr!

[bryanmacfarlane]

In the meantime one easy way to get a signal is the runner has two processes. One that listens to the queue (long running) and one that spawns a worker process (one per job) so you could ideally hook into process start / stop and at worst poll (ps aux | grep …)