Pipes appear to not pass on EOF on macOS runners
See original GitHub issueDescribe the bug
(first reported at https://github.com/actions/virtual-environments/discussions/2352)
Using yes | some_script
anywhere in a workflow on macOS hangs forever. some_script
will terminate but yes
will keep going, blocking the rest of the workflow.
To Reproduce
I’ve made a minimal test case here: https://github.com/kousu/hanging-actions/. It basically just tests:
yes | head -n 7
plus collecting some context for debugging.
To reproduce, just fork my repo and watch its Actions tab. Linux (and Travis!) will pass easily, but Actions on macOS hangs until cancelled.
Expected behavior All platforms should succeed in approximately the same time and way.
Runner Version and Platform
From my logs:
Request a runner to run this job
- Current runner version: ‘2.275.1’
Operating System
- Mac OS X
- 10.15.7
- 19H15
Virtual Environment
- Environment: macos-10.15
- Version: 20201212.1
- Included Software: https://github.com/actions/virtual-environments/blob/macOS-10.15/20201212.1/images/macos/macos-10.15-Readme.md
Thanks to @maxim-lobanov we know that it only happens under actions/runner, and not under https://github.com/microsoft/azure-pipelines-agent (even though azure pipelines runs the same macOS images) or when connected over VNC or ssh.
What’s not working?
The symptom is that the Linux builds finish immediately while the macOS build hangs in yellow forever:
In fact even Travis on nearly (but not exactly the same) version of macOS works fine.
For example, consider this Actions Workflow and this this Travis script for comparison, everything finishes in about a minute except for Actions-macOS.
This should only take a moment to finish, and on Travis it does:
but on Actions it’s at 3 minutes and counting. I’ve had jobs hung much longer too – up to their 6 hour limit – before I noticed what was going on:
The only way to stop the job is to cancel it. It never undeadlocks.
Job Log Output
- Actions Ubuntu: https://github.com/kousu/hanging-actions/runs/1608143472 actions-ubuntu.txt
- Actions macOS: https://github.com/kousu/hanging-actions/runs/1608143479 (hung) actions-macos.txt
- Actions CentOS: https://github.com/kousu/hanging-actions/runs/1608143484 actions-centos.txt
- Travis Ubuntu: https://travis-ci.com/github/kousu/hanging-actions/jobs/464777666 travis-ubuntu.txt
- Travis macOS: https://travis-ci.com/github/kousu/hanging-actions/jobs/464777667 travis-macos.txt
Runner and Worker’s Diagnostic Logs
I don’t have access to these! If I install a runner locally and reproduce there I’ll update this.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:5
- Comments:12 (2 by maintainers)
genv --default-signal=PIPE yes | head -n7
is the only workaround I could figure out. Needs GNU coreutils installed from e.g. Homebrew.I am running into this here: https://github.com/tavianator/homebrew-tap/runs/4874104560?check_suite_focus=true. I think I found the issue: macOS uses
node
to spawn the shell to work around an SIP issue: https://github.com/actions/runner/blob/c95d5eae3092358e0027f4d26b9962eedf2af93c/src/Runner.Worker/Handlers/ScriptHandler.cs#L270-L279macos-run-invoker.js
calls the shell withspawn()
: https://github.com/actions/runner/blob/c95d5eae3092358e0027f4d26b9962eedf2af93c/src/Misc/layoutbin/macos-run-invoker.js#L1-L13Old versions of
node
have a bug inspawn()
that runs the child withSIGPIPE
ignored: https://github.com/nodejs/node/issues/13662. Apparently macOSyes
doesn’t check forEPIPE
, it just expects to die fromSIGPIPE
, so it hangs forever writingy
to a broken pipe.I suspect using Node 16 instead of Node 12 would fix it, but I can’t confirm easily because I don’t have hands-on access to macOS.