Github actions: "Re-run failed jobs" will run the entire test suite

We have the e2e tests configure to run on cypress dashboard parellely. I was following this thread to add the custom-build-id to the command to distinguish different run based on different build id. Every thing works fine until github actions roll out the ability to Re-run failed jobs. If i just set the custom-build-id to ${{ github.run_id }}, the second attempt will always marks tests as passing with ‘Run Finished’ but tests are not triggered at all. So I set set the custom-build-id to ${{ github.run_id }}-${{ github.run_attempt }}, now it will run the entire test suite instead of running the originally allocated subset of tests.

 E2E_tests:
    runs-on: ubuntu-latest
    name: E2E tests
    strategy:
      fail-fast: false
      matrix:
        ci_node_total: [6]
        ci_node_index: [0, 1, 2, 3, 4, 5]
    timeout-minutes: 45
    steps:
        - uses: actions/checkout@v2

        - name: Use Node.js
          uses: actions/setup-node@v2

        - name: Install Dependencies
          run: npm ci

        - name: Start app
          run: make start-app-for-e2e
          timeout-minutes: 5

        - name: Cypress Dashboard - Cypress run
          run: |
              npm run cypress

[admah]

I was able to get some more clarity on this from our Cloud team. Issue #574 also has some additional context.

Here is the current status:

• Before, there was an issue where all re-runs got a PASS, regardless of actual status. This issue has been fixed.
• Currently, if a re-run is initiated, all specs get run on the machines available. That is not optimal. The Cloud team is looking into the connection between GH Actions and Cypress in order to set up re-runs to be accurate and efficient.

I will be updating this issue as new information is available.

[tebeco]

That’s definitly something critical given how the billing works (Cypress and Github included), it sounds like we’re getting billed for something that already passed