Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

`cml-runner` and `cml-send-comment` commands take ages to execute

See original GitHub issue

Hey everyone! This is something that (as far as I can tell) has started happening after the update to 0.14.1 last week. The cml-runner and cml-send-comment commands take a very long time to run now.

Up to 1h for the cml runner command
Up to 30 minutes for the send comment command.

I’m attaching logs for each of these here. Not a lot of info, apart from the fact that it seems to be waiting for a really long time and then just retrying? Very weird.

1_Deploy Cloud Instances.txt 4_Post comment.txt

Issue Analytics

State:
Created a year ago
Comments:16 (12 by maintainers)

Top GitHub Comments

1reaction

dacbdcommented, May 6, 2022

@thatGreekGuy96 on having a safety check for training instances: https://github.com/marketplace/actions/gcpcc (at some point Intend to make a AWS version)

1reaction

dacbdcommented, Apr 19, 2022

I had issues back in December where cml runner instances did not self-terminate in GCP for about 2-3 weeks / and wasn’t keeping a close eye out on that account. Then I made a thing which I used in an action to create an issue if there were more instances than there should be in said account.

Edit: there were no helpful logs that could decern what went wrong. @thatGreekGuy96 if the above sounds helpful I can slap together an example you could use.