option to enable CI/CD checks

Hi!

I have a scheduled job for model re-training and integrated CML to simplify running a DVC pipeline and creating a PR within Github Actions. For example, once a week a job is triggered and in the end I see a new PR with model results. I would like to run some pre-configured checks for the newly created PR, however, they do not run as per docs. I thought that the checks are disabled for the first created with cml pr commit, but even a manual second commit does not trigger jobs. Is it because of CML? If so, can this behavior be toggle-able?

[RadionBik]

Hi!

After many trials and errors we have managed to retrieve and integrate github application token within our CML workflow. Here is an excerpt from the yaml:

container: docker://ghcr.io/iterative/cml:0-dvc2-base1
    steps:
      - uses: navikt/github-app-token-generator@v1
        id: get-token
        with:
          private-key: ${{ secrets.CML_GITHUB_APP_PEM }}
          app-id: ${{ secrets.CML_GITHUB_APP_ID }}

      - uses: actions/checkout@v2
        with:
          fetch-depth: 0
          token: ${{ steps.get-token.outputs.token }}


        ...

A potential pitfall is setting proper permissions for the app, once it is done, the rest part is easy.

P.S. I think it would be great to have the information from this issue in the docs

[DavidGOrtega]

@RadionBik as @dacbd states cml pr avoids the CI run to avoid infinite CI executions (one PR would generate another PR)

A solution might be trigger it with

on:
  issue_comment:
    types: [created]

jobs:
  pr-checks:
    if: startsWith(github.event.comment.body, '/pr-checks')

Then after generating the pr

echo '/pr-checks' > action.md
cml send-comment --commit-sha HEAD --pr action.md