Jobs getting cancelled with `runner status Job <job name> completed with result: Canceled`
See original GitHub issueStarting today all cml jobs (using github actions and AWS) are failing with The self-hosted runner: cml-yfmmfzugyh lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
(even when re-running jobs that previously succeeded - and the shutdown cause is Client.UserInitiatedShutdown: User initiated shutdown
).
Checking the logs it seems like the job gets cancelled for some reason: runner status Job <job name> completed with result: Canceled
Whether this is due to cml or other github runner or aws issues I’m not sure, but there are various confusing messages in the logs that may be related:
runner status An error occurred Access denied. System:Serviceldentity;DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD needs View permissions to perform the action.
\tFailed: Cannot destructure property 'id' of '(intermediate value)' as it is undefined.
Issue Analytics
- State:
- Created 2 years ago
- Comments:6 (5 by maintainers)
Top GitHub Comments
Closed as we are unable to reproduce it again
Hello! So it was still failing earlier today, and I just tried a run to properly collect all the logs and now the issue isn’t happening! So unfortunately the only logs I have are the screenshot above (taken quickly with ec2-instance-connect and
cat /var/log/syslog | grep cml
before the instance shut down).One thing I did find is that the
Cannot destructure property 'id' of '(intermediate value)'
message appears even when it succeeds, so that seems to be unrelated. So it looks likeAn error occurred Access denied. System:Serviceldentity;DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD
is the only unusual one.Not sure if there’s much that can be done if it is transient and fixed now, so if you want to close the issue I can report if it comes back or if I find any other more clues.