Inconsistent cypress runs between github action and cypress dashboard

Hi 👋

I’ve recently updated my cypress github workflow to run only on pull_requests against the master branch instead of push events and I’m seeing a weird side effect when opening a new Pull Requests and re-running the cypress job:

• Cypress will run the test suite of the master branch (not the head branch that is being submitted as a PR to merge on top of master). On the github status check for the cypress dashboard, I can see it’s called default-group (merge) and the title of the run contains the commit hash of my head branch and the HEAD commit hash of the master branch we’re merging into.

• If I rerun the same job, Cypress will run the test suite of the head branch (as expected). On the github status check for the cypress dashboard, I can see it’s simply called default-group and the title of the run

It looks like Cypress Dashboard is somehow capturing 2 different events on the same github ci run (from the same workflow file). I’m pretty convinced that my update to run cypress only on pull_request events triggered.

Am I doing something wrong?

Here’s a screenshot of the pull requests checks:

Here’s my Github Workflow file:

name: Run End-to-End Test Suite

on:
  pull_request:
    branches:
      - master

jobs:
  build-and-e2e-test:
    runs-on: ubuntu-16.04
    strategy:
      matrix:
        node-version: [11.x]
    steps:
      - name: Checkout Commit
        uses: actions/checkout@v1
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}
      - name: Cache NPM Dependencies
        id: cache-dependencies
        uses: actions/cache@v1
        with:
          path: node_modules
          key: node-modules-${{ runner.OS }}-npm-cache-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            node-modules-${{ runner.OS }}-npm-cache
      - name: Cache Cypress Binaries
        id: cache-cypress-binaries
        uses: actions/cache@v1
        with:
          path: ~/.cache/Cypress
          key: cypress-binaries-${{ runner.OS }}-npm-cache-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            cypress-binaries-${{ runner.OS }}-npm-cache
      - name: Install Dependencies
        if: steps.cache-dependencies.outputs.cache-hit != 'true' || steps.cache-cypress-binaries.outputs.cache-hit != 'true'
        run: |
          yarn install --force --non-interactive
      - name: Build Application
        run: yarn build
        env:
          CI: true
      - name: Run E2E Tests without recording (if not on master branch)
        uses: cypress-io/github-action@v1
        with:
          browser: chrome
          record: true
          headless: true
          start: yarn serve-built-files
          wait-on: "http://localhost:8081"
          config: defaultCommandTimeout=100000,pageLoadTimeout=100000,watchForFileChanges=false,video=false
        env:
          NODE_ENV: "test"
          CI: true
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

[bahmutov]

I have looked at what is going on by opening https://github.com/bahmutov/cypress-gh-action-example/pull/14

The workflow is running both on push (commit) and on pull_request events

name: merge
# run this action on Push and on Pull Request
# to demonstrate the difference in commit SHA
on: [push, pull_request]

You can see the Dashboard at https://dashboard.cypress.io/projects/yz8qku and filter by tags since I pass the event name as a tag

- name: Cypress run on ${{ github.event_name }}
  uses: cypress-io/github-action@v2
  with:
    record: true
    group: merge
    tag: ${{ github.event_name }}
  env:
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

Every commit with open pull request thus triggers 2 events = 2 action runs

For example here in GH Actions we see merge #3 which is a push event with the current commit title “print”, and merge #4 which is the pull request that uses the original PR title “add pr workflow”.

Here is what our Dashboard shows in this case

Build 164 is the recording of the “push” event, has the expected title from the commit “print” and is ok.

Build 162 is the recording of the “pull_request” event, and it uses the default commit title created by GitHub (merge commit A into B), and this is what we would like to change.

When we look closely at the build 162, its commit also is not as nice as it could be

which leads to the merge commit and not to the pull request

Solution

To me the largest problem is the commit message for pull request. Thus a good solution is to set it for pull requests as an environment variable to overwrite the Git information. You can find the file at https://github.com/bahmutov/cypress-gh-action-example/blob/master/.github/workflows/merge.yml and here is the relevant portion

- name: Cypress run on ${{ github.event_name }}
  uses: cypress-io/github-action@v2
  with:
    record: true
    group: merge
    # tag will be either "push" or "pull_request"
    tag: ${{ github.event_name }}
  env:
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
    # - if the event is push, the title will be undefined
    # and Cypress will get the commit message from Git information
    # - if the event is pull_request, then we set the commit
    # message to the pull request title
    COMMIT_INFO_MESSAGE: ${{ github.event.pull_request.title }}

The Dashboard shows then:

“push” event shows the commit’s subject line. “pull_request” event shows the PR title.

[eplatek-safyr]

@bahmutov Thanks for this solution that fixes the dashboard labels, but it breaks the PR comments on github. After applying this solution to my project I don’t have cypress bot comments on my PR, after removing the COMMIT_INFO_MESSAGE variable the comments are back.

Any idea on how to keep both of those features working ?