question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Jobs getting cancelled with `runner status Job <job name> completed with result: Canceled`

See original GitHub issue

Starting today all cml jobs (using github actions and AWS) are failing with The self-hosted runner: cml-yfmmfzugyh lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error. (even when re-running jobs that previously succeeded - and the shutdown cause is Client.UserInitiatedShutdown: User initiated shutdown).

Checking the logs it seems like the job gets cancelled for some reason: runner status Job <job name> completed with result: Canceled

Screenshot 2021-12-09 at 20 25 23

Whether this is due to cml or other github runner or aws issues I’m not sure, but there are various confusing messages in the logs that may be related:

runner status An error occurred Access denied. System:Serviceldentity;DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD needs View permissions to perform the action.
\tFailed: Cannot destructure property 'id' of '(intermediate value)' as it is undefined.

Similar messages were reported in #676 and #345

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
DavidGOrtegacommented, Dec 23, 2021

Closed as we are unable to reproduce it again

1reaction
jamt9000commented, Dec 10, 2021

Hello! So it was still failing earlier today, and I just tried a run to properly collect all the logs and now the issue isn’t happening! So unfortunately the only logs I have are the screenshot above (taken quickly with ec2-instance-connect and cat /var/log/syslog | grep cml before the instance shut down).

One thing I did find is that the Cannot destructure property 'id' of '(intermediate value)' message appears even when it succeeds, so that seems to be unrelated. So it looks like An error occurred Access denied. System:Serviceldentity;DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD is the only unusual one.

Not sure if there’s much that can be done if it is transient and fixed now, so if you want to close the issue I can report if it comes back or if I find any other more clues.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Ensure after_script is called for cancelled and timed out ...
If a job is in either created or pending , it should transition directly to canceled as no execution has happened yet. Regarding...
Read more >
How to run a script at the end of a job, even if ... - Stack Overflow
always : Always returns true , even when canceled. A job or step will not run when a critical failure prevents the task...
Read more >
Jobs in Azure Pipelines and TFS - Microsoft Learn
The timeout period begins when the job starts running. It does not include the time the job is queued or is waiting for...
Read more >
Cancel TSM Jobs - Tableau Help
Job is canceled when Tableau Server processes are stopping: The job is canceled, but the services will try to get to a stopped...
Read more >
Cannot cancel a build if a task is running
You'll experience this issue if the condition that's configured in the stage doesn't include a job status check function. To resolve the issue, ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found