[Build] Cancelling workflows with airflow-cancel-workflow-runs action within each workflow doesn't seem to work

Cancelling workflows with airflow-cancel-workflow-runs within each workflow doesn’t seem to work. That change was made in #9357 .

However there’s something that is cancelling workflows. Cancelling happened for example for PR #9443 . All of it’s checks are now in cancelled state. I checked also a few other cases. It looks like someone is manually maintaining the build queue and cancelling jobs. Another example I looked at was #9476 . The remaining jobs from the previous commit had been already cancelled even though the new jobs hadn’t yet started. Currently the automated cancellation will only happen after the next workflow run has started.

This issue report is about errors seen for in the logs of this workflow run:

https://github.com/apache/pulsar/runs/1804639066#step:3:119

2021-02-01T09:09:48.6152871Z Cancelling run: 527073250.
2021-02-01T09:09:48.6153130Z 
2021-02-01T09:09:48.7047449Z ##[warning]
Could not cancel run 527073250: [403] Resource not accessible by integration

2021-02-01T09:09:48.7062237Z 
2021-02-01T09:09:48.7062636Z Cancelling run: 125656974.
2021-02-01T09:09:48.7062888Z 
2021-02-01T09:09:48.8020647Z ##[warning]
Could not cancel run 125656974: [403] Resource not accessible by integration

2021-02-01T09:09:48.8022156Z 
2021-02-01T09:09:48.8022528Z Cancelling run: 124625158.
2021-02-01T09:09:48.8022809Z 
2021-02-01T09:09:48.8864785Z ##[warning]
Could not cancel run 124625158: [403] Resource not accessible by integration

Full logs for the cancellation step

2021-02-01T09:09:47.0409051Z ##[group]Run apache/airflow-cancel-workflow-runs@953e057dc81d3458935a18d1184c386b0f6b5738
2021-02-01T09:09:47.0410182Z with:
2021-02-01T09:09:47.0411786Z   token: ***
2021-02-01T09:09:47.0412340Z env:
2021-02-01T09:09:47.0413416Z   MAVEN_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.count=3
2021-02-01T09:09:47.0414822Z ##[endgroup]
2021-02-01T09:09:47.4123083Z [@octokit/rest] `const Octokit = require("@octokit/rest")` is deprecated. Use `const { Octokit } = require("@octokit/rest")` instead
2021-02-01T09:09:47.4379824Z [@octokit/rest] `const Octokit = require("@octokit/rest")` is deprecated. Use `const { Octokit } = require("@octokit/rest")` instead
2021-02-01T09:09:47.4451574Z 
2021-02-01T09:09:47.4452863Z Getting workflow id for source run id: 526558278, owner: apache, repo: pulsar, skipEventTypes: 
2021-02-01T09:09:47.4453418Z 
2021-02-01T09:09:47.5954282Z The source run 526558278 is in https://api.github.com/repos/apache/pulsar/actions/workflows/955 workflow
2021-02-01T09:09:47.5955439Z 
2021-02-01T09:09:47.5956145Z Finding runs for my own workflow 955
2021-02-01T09:09:47.5956718Z 
2021-02-01T09:09:47.5957960Z Repository: apache/pulsar, Owner: apache, Repo: pulsar, Event name: pull_request, CancelMode: duplicates, sourceWorkflowId: 955, sourceRunId: 526558278, selfRunId: 526558278, jobNames: 
2021-02-01T09:09:47.7072108Z Source workflow: Head repo: aloyszhang/pulsar, Head branch: fix Event: pull_request, Head sha: dd34a5b547f194d3d03fdc6d798ed7dba27d5124, url: https://api.github.com/repos/apache/pulsar/actions/runs/526558278
2021-02-01T09:09:47.7073359Z 
2021-02-01T09:09:47.7074041Z Finding PR request id for: owner: apache, Repo:pulsar, Head:aloyszhang:fix.
2021-02-01T09:09:47.7074592Z 
2021-02-01T09:09:47.8817175Z 
2021-02-01T09:09:47.8818545Z Comparing: 9334 sha: dd34a5b547f194d3d03fdc6d798ed7dba27d5124 with expected: dd34a5b547f194d3d03fdc6d798ed7dba27d5124.
2021-02-01T09:09:47.8825915Z 
2021-02-01T09:09:47.8826135Z 
2021-02-01T09:09:47.8826514Z Found PR: 9334
2021-02-01T09:09:47.8826742Z 
2021-02-01T09:09:47.8827256Z Setting output: sourceHeadRepo: aloyszhang/pulsar
2021-02-01T09:09:47.8838946Z Setting output: sourceHeadBranch: fix
2021-02-01T09:09:47.8841018Z Setting output: sourceHeadSha: dd34a5b547f194d3d03fdc6d798ed7dba27d5124
2021-02-01T09:09:47.8842236Z Setting output: sourceEvent: pull_request
2021-02-01T09:09:47.8843303Z Setting output: pullRequestNumber: 9334
2021-02-01T09:09:47.8844500Z Setting output: mergeCommitSha: 7da506c8190ed5aebbd8f78cacdf357865a9d3f6
2021-02-01T09:09:47.8846276Z Setting output: targetCommitSha: 7da506c8190ed5aebbd8f78cacdf357865a9d3f6
2021-02-01T09:09:47.8847247Z 
2021-02-01T09:09:47.8847619Z ###################################################################################
2021-02-01T09:09:47.8847889Z 
2021-02-01T09:09:47.8849102Z All parameters: owner: apache, repo: pulsar, run id: 526558278, Source workflow id: 955, head repo aloyszhang/pulsar, headBranch: fix, sourceEventName: pull_request, cancelMode: duplicates, jobNames: 
2021-02-01T09:09:47.8849991Z 
2021-02-01T09:09:47.8850879Z ###################################################################################
2021-02-01T09:09:47.8851138Z 
2021-02-01T09:09:47.8851850Z # Cancel duplicate runs for workflow 955 for same triggering branch as own run Id.
2021-02-01T09:09:47.8852321Z 
2021-02-01T09:09:47.8853069Z ###################################################################################
2021-02-01T09:09:47.8853414Z 
2021-02-01T09:09:47.8853620Z 
2021-02-01T09:09:47.8854350Z Finding duplicate runs: Owner: apache, Repo: pulsar, Status: queued Workflow ID:955, Head Branch: fix,Event name: pull_request
2021-02-01T09:09:47.8854985Z 
2021-02-01T09:09:48.4512396Z 
2021-02-01T09:09:48.4513555Z Finding duplicate runs: Owner: apache, Repo: pulsar, Status: in_progress Workflow ID:955, Head Branch: fix,Event name: pull_request
2021-02-01T09:09:48.4514183Z 
2021-02-01T09:09:48.6103631Z 
2021-02-01T09:09:48.6104310Z Found runs: 7212,7211,2654,2610,7192
2021-02-01T09:09:48.6111163Z 
2021-02-01T09:09:48.6111472Z 
2021-02-01T09:09:48.6113228Z Checking run number: 7212 RunId: 527117669 Url: https://api.github.com/repos/apache/pulsar/actions/runs/527117669 Status queued Created at 2021-02-01T08:34:21Z
2021-02-01T09:09:48.6113940Z 
2021-02-01T09:09:48.6114212Z 
2021-02-01T09:09:48.6115042Z The run 527117669 is from a different repo: eolivelli/pulsar (expected aloyszhang/pulsar). Not adding as candidate to cancel.
2021-02-01T09:09:48.6115935Z 
2021-02-01T09:09:48.6116140Z 
2021-02-01T09:09:48.6116711Z Cancel Future Duplicates: Returning run id that might be duplicate or my own run: 527117669.
2021-02-01T09:09:48.6117207Z 
2021-02-01T09:09:48.6117410Z 
2021-02-01T09:09:48.6118359Z Adding the run: 527117669 to candidates with :955:aloyszhang/pulsar:fix:pull_request key.
2021-02-01T09:09:48.6118845Z 
2021-02-01T09:09:48.6119177Z 
2021-02-01T09:09:48.6120496Z Checking run number: 7211 RunId: 527073250 Url: https://api.github.com/repos/apache/pulsar/actions/runs/527073250 Status queued Created at 2021-02-01T08:14:22Z
2021-02-01T09:09:48.6121122Z 
2021-02-01T09:09:48.6121303Z 
2021-02-01T09:09:48.6121954Z The run 527073250 is from a different repo: BewareMyPower/pulsar (expected aloyszhang/pulsar). Not adding as candidate to cancel.
2021-02-01T09:09:48.6122535Z 
2021-02-01T09:09:48.6122713Z 
2021-02-01T09:09:48.6123675Z Cancel Future Duplicates: Returning run id that might be duplicate or my own run: 527073250.
2021-02-01T09:09:48.6124372Z 
2021-02-01T09:09:48.6124696Z 
2021-02-01T09:09:48.6125452Z Adding the run: 527073250 to candidates with :955:aloyszhang/pulsar:fix:pull_request key.
2021-02-01T09:09:48.6125937Z 
2021-02-01T09:09:48.6126130Z 
2021-02-01T09:09:48.6127304Z Checking run number: 2654 RunId: 125656974 Url: https://api.github.com/repos/apache/pulsar/actions/runs/125656974 Status queued Created at 2020-06-05T06:35:30Z
2021-02-01T09:09:48.6128261Z 
2021-02-01T09:09:48.6128451Z 
2021-02-01T09:09:48.6129166Z The run 125656974 is from a different repo: magrain/pulsar (expected aloyszhang/pulsar). Not adding as candidate to cancel.
2021-02-01T09:09:48.6129776Z 
2021-02-01T09:09:48.6130077Z 
2021-02-01T09:09:48.6130657Z Cancel Future Duplicates: Returning run id that might be duplicate or my own run: 125656974.
2021-02-01T09:09:48.6131333Z 
2021-02-01T09:09:48.6131542Z 
2021-02-01T09:09:48.6132105Z Adding the run: 125656974 to candidates with :955:aloyszhang/pulsar:fix:pull_request key.
2021-02-01T09:09:48.6132593Z 
2021-02-01T09:09:48.6132783Z 
2021-02-01T09:09:48.6133956Z Checking run number: 2610 RunId: 124625158 Url: https://api.github.com/repos/apache/pulsar/actions/runs/124625158 Status queued Created at 2020-06-04T09:04:51Z
2021-02-01T09:09:48.6134672Z 
2021-02-01T09:09:48.6134864Z 
2021-02-01T09:09:48.6135571Z The run 124625158 is from a different repo: magrain/pulsar (expected aloyszhang/pulsar). Not adding as candidate to cancel.
2021-02-01T09:09:48.6136185Z 
2021-02-01T09:09:48.6136374Z 
2021-02-01T09:09:48.6136959Z Cancel Future Duplicates: Returning run id that might be duplicate or my own run: 124625158.
2021-02-01T09:09:48.6137602Z 
2021-02-01T09:09:48.6137797Z 
2021-02-01T09:09:48.6138370Z Adding the run: 124625158 to candidates with :955:aloyszhang/pulsar:fix:pull_request key.
2021-02-01T09:09:48.6138853Z 
2021-02-01T09:09:48.6139042Z 
2021-02-01T09:09:48.6140199Z Checking run number: 7192 RunId: 526558278 Url: https://api.github.com/repos/apache/pulsar/actions/runs/526558278 Status in_progress Created at 2021-02-01T03:35:40Z
2021-02-01T09:09:48.6140944Z 
2021-02-01T09:09:48.6141132Z 
2021-02-01T09:09:48.6142212Z I have self-preservation built in. I refuse to cancel myself :)
2021-02-01T09:09:48.6142811Z 
2021-02-01T09:09:48.6148088Z 
2021-02-01T09:09:48.6149541Z Skipping the first run (527117669) of all the matching duplicates for ':955:aloyszhang/pulsar:fix:pull_request'. This one we are going to leave in peace!
2021-02-01T09:09:48.6150278Z 
2021-02-01T09:09:48.6150459Z 
2021-02-01T09:09:48.6151242Z ###### Cancelling runs for :955:aloyszhang/pulsar:fix:pull_request starting from the most recent  ##########
2021-02-01T09:09:48.6151746Z 
2021-02-01T09:09:48.6152095Z      Number of runs to cancel: 3
2021-02-01T09:09:48.6152366Z 
2021-02-01T09:09:48.6152542Z 
2021-02-01T09:09:48.6152871Z Cancelling run: 527073250.
2021-02-01T09:09:48.6153130Z 
2021-02-01T09:09:48.7047449Z ##[warning]
Could not cancel run 527073250: [403] Resource not accessible by integration
2021-02-01T09:09:48.7062237Z
2021-02-01T09:09:48.7062636Z Cancelling run: 125656974.
2021-02-01T09:09:48.7062888Z
2021-02-01T09:09:48.8020647Z ##[warning]
Could not cancel run 125656974: [403] Resource not accessible by integration
2021-02-01T09:09:48.8022156Z
2021-02-01T09:09:48.8022528Z Cancelling run: 124625158.
2021-02-01T09:09:48.8022809Z
2021-02-01T09:09:48.8864785Z ##[warning]
Could not cancel run 124625158: [403] Resource not accessible by integration
2021-02-01T09:09:48.8866064Z
2021-02-01T09:09:48.8866581Z ######  Finished cancelling runs for :955:aloyszhang/pulsar:fix:pull_request ##########
2021-02-01T09:09:48.8867216Z
2021-02-01T09:09:48.8867667Z Setting output: cancelledRuns: [527073250,125656974,124625158]
2021-02-01T09:09:48.8868918Z
2021-02-01T09:09:48.8869475Z ############### Cancel complete ##################

It seems that airflow-cancel-workflow-runs is not supposed to be used in the way that we use it.

There are actions that support per-workflow cancelling, for example https://github.com/marketplace/actions/cancel-workflow-action . That was previously used, until it was possibly unintentionally removed by #9069 .

It would have been possible to continue to use cancel-workflow-action, as described in https://github.com/apache/pulsar/issues/9154#issuecomment-756984731

An alternative option is to raise INFRA ticket to allow styfle/cancel-workflow-action (and still use SHA-based action ids like styfle/cancel-workflow-action@123123123…123123)

[potiuk]

You are inded usign the action wrongly.

The problem is that the workflow that runs from fork does not have permission to write in your repo:

This is what Could not cancel run 527073250: [403] Resource not accessible by integration message means (thought it is not obvious).

The actual “cancel” action shoudl run in a separate worklfow (of workflow_run type): https://docs.github.com/en/actions/reference/events-that-trigger-workflows#workflow_run - this way it will have WRITE permission necessary to cancel workflows.

If you start many workflows at the same time, it will be enough to start single “workflow_run” with multiple cancel-workflow steps (one per-workflow).

I think also you misuderstand how allDuplicates action works. It does not kill “one branch” duplicates" - it is far more aggressive, It kills ALL duplicates in ALL branches for a given workflow.

I can - tomorrow- draft a PR fixing your setup if you would like my help with that 😃

[lhotari]

Fixed by @potiuk 's PR #9503 which has been merged.

@potiuk Thank you for the great help!