Pipes appear to not pass on EOF on macOS runners

Describe the bug

(first reported at https://github.com/actions/virtual-environments/discussions/2352)

Using yes | some_script anywhere in a workflow on macOS hangs forever. some_script will terminate but yes will keep going, blocking the rest of the workflow.

To Reproduce

I’ve made a minimal test case here: https://github.com/kousu/hanging-actions/. It basically just tests:

yes | head -n 7

plus collecting some context for debugging.

To reproduce, just fork my repo and watch its Actions tab. Linux (and Travis!) will pass easily, but Actions on macOS hangs until cancelled.

Expected behavior All platforms should succeed in approximately the same time and way.

Runner Version and Platform

From my logs:

Request a runner to run this job

• Current runner version: ‘2.275.1’

Operating System

• Mac OS X
• 10.15.7
• 19H15

Virtual Environment

• Environment: macos-10.15
• Version: 20201212.1
• Included Software: https://github.com/actions/virtual-environments/blob/macOS-10.15/20201212.1/images/macos/macos-10.15-Readme.md

Thanks to @maxim-lobanov we know that it only happens under actions/runner, and not under https://github.com/microsoft/azure-pipelines-agent (even though azure pipelines runs the same macOS images) or when connected over VNC or ssh.

What’s not working?

The symptom is that the Linux builds finish immediately while the macOS build hangs in yellow forever:

In fact even Travis on nearly (but not exactly the same) version of macOS works fine.

For example, consider this Actions Workflow and this this Travis script for comparison, everything finishes in about a minute except for Actions-macOS.

This should only take a moment to finish, and on Travis it does:

but on Actions it’s at 3 minutes and counting. I’ve had jobs hung much longer too – up to their 6 hour limit – before I noticed what was going on:

The only way to stop the job is to cancel it. It never undeadlocks.

Job Log Output

• Actions Ubuntu: https://github.com/kousu/hanging-actions/runs/1608143472 actions-ubuntu.txt
• Actions macOS: https://github.com/kousu/hanging-actions/runs/1608143479 (hung) actions-macos.txt
• Actions CentOS: https://github.com/kousu/hanging-actions/runs/1608143484 actions-centos.txt
• Travis Ubuntu: https://travis-ci.com/github/kousu/hanging-actions/jobs/464777666 travis-ubuntu.txt
• Travis macOS: https://travis-ci.com/github/kousu/hanging-actions/jobs/464777667 travis-macos.txt

Runner and Worker’s Diagnostic Logs

I don’t have access to these! If I install a runner locally and reproduce there I’ll update this.

[tavianator]

genv --default-signal=PIPE yes | head -n7 is the only workaround I could figure out. Needs GNU coreutils installed from e.g. Homebrew.

[tavianator]

I am running into this here: https://github.com/tavianator/homebrew-tap/runs/4874104560?check_suite_focus=true. I think I found the issue: macOS uses node to spawn the shell to work around an SIP issue: https://github.com/actions/runner/blob/c95d5eae3092358e0027f4d26b9962eedf2af93c/src/Runner.Worker/Handlers/ScriptHandler.cs#L270-L279

macos-run-invoker.js calls the shell with spawn(): https://github.com/actions/runner/blob/c95d5eae3092358e0027f4d26b9962eedf2af93c/src/Misc/layoutbin/macos-run-invoker.js#L1-L13

Old versions of node have a bug in spawn() that runs the child with SIGPIPE ignored: https://github.com/nodejs/node/issues/13662. Apparently macOS yes doesn’t check for EPIPE, it just expects to die from SIGPIPE, so it hangs forever writing y to a broken pipe.

I suspect using Node 16 instead of Node 12 would fix it, but I can’t confirm easily because I don’t have hands-on access to macOS.