stop not working when start was cancelled by concurrency cancelation rule
See original GitHub issueTo ensure that no more than one instance is running on several consequent pushes, the idea is to cancel the previous still running workflows on a new push, so we added a global setting:
concurrency: # cancel previous build on a new push
group: ${{ github.ref }} # https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#github-context
cancel-in-progress: true
now when I’m testing this, quite often stop fails to stop leaving the instance running, which is very expensive!
Run machulav/ec2-github-runner@v2
Error: Error: Not all the required inputs are provided for the 'stop' mode
Error: Not all the required inputs are provided for the 'stop' mode
Error: TypeError: Cannot read property 'mode' of undefined
Error: Cannot read property 'mode' of undefined
Here is the log from the start job:
with:
mode: start
github-token: ***
ec2-image-id: ami-03540b272db1624b7
ec2-instance-type: p3.8xlarge
security-group-id: sg-f2a4e2fc
subnet-id: subnet-b7533b96
aws-resource-tags: [
{"Key": "Name", "Value": "ec2-github-runner"},
{"Key": "GitHubRepository", "Value": "bigscience-workshop/Megatron-DeepSpeed"}
]
env:
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ACCESS_KEY_ID: ***
AWS_SECRET_ACCESS_KEY: ***
GitHub Registration Token is received
AWS EC2 instance i-038eeed014c994b48 is started
Error: The operation was canceled.
so it looks like it didn’t set the vars it was supposed to set because it was cancelled.
Here is the full workflow for context: https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/7c636d7555e915f1f426984172f73840b2168313/.github/workflows/main.yml
If there are other solutions I’m all ears.
Thank you!
Issue Analytics
- State:
- Created 2 years ago
- Comments:12
Top Results From Across the Web
How to cancel previous runs in the PR when you push new ...
You can use Concurrency: Concurrency ensures that only a single job or workflow using the same concurrency group will run at a time....
Read more >Cancellation in the PPL | Microsoft Learn
Cancellation does not occur immediately. Although new work is not started if a task or task group is canceled, active work must check...
Read more >Chapter 9. Cancellation and Timeouts - O'Reilly
In that case, cancel throws the ThreadKilled exception to the thread, so waitCatch will return Left ThreadKilled . Starts the downloads as before....
Read more >Cancellation and timeouts | Kotlin Documentation
The launch function returns a Job that can be used to cancel the running ... the job job.join() // waits for job's completion...
Read more >Structured Concurrency (async let) - Cancellation - Swift Forums
This is why, in your first scenario, the computeA task will run until completion without being cancelled, since computeB has not been asked...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Would it help if you used a combination of workflow-level concurrency control and job-level concurrency control? Then you could use
cancel-in-progress
for the test job, which I presume would be safe (since it would only cut the test job short), while still being able to limit concurrent workflow runs to just one as well?Nice idea 👍 Not sure how complex/brittle the implementation would get, though.
Yep, the stop job is in fact annotated with that (see the
if
condition), but the problem is that the input to that job is now lacking (because the start job was cut short).Understood. One option could then be to leverage PR comments, e.g. verbatim
/test
to trigger tests at a suitable spot. You can use theissue_comment
event (comparing the comment text) to trigger a workflow when a PR comment is added. Or perhaps there are some ready-made github apps/integrations that could help with this?That was just an example, static concurrency group name.
All right, looks like you are in fact using
${{ github.ref }}
for the concurrency group already 👍