Deploy waits for 'existing' runner which doesn't exist

I’ve been having this issue where if i set the --reuse flag, subsequent jobs will fail to start because they are waiting for a runner that has already terminated.

Run cml-runner \
Reusing existing runners with the cml-runner labels...

No further CML runs will launch unless I cancel them and then restart them. This is my cml.yaml:

name: Run DVC repro
on: [push]
jobs:
  deploy-runner:
    runs-on: [ubuntu-latest]
    steps:
      - uses: actions/checkout@v2
      - uses: iterative/setup-cml@v1

      - name: deploy
        shell: bash
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          cml-runner \
          --cloud aws \
          --cloud-region eu-west \
          --cloud-type=g4dn.xlarge \
          --cloud-spot true \
          --reuse \
          --labels=cml-runner,reviews-labeler \
          --idle-timeout 300
  run:
    needs: deploy-runner
    runs-on: [self-hosted,cml-runner]
    container: docker://dvcorg/cml:0-dvc2-base1-gpu
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: '3.8.5'
      - name: cml_run
        shell: bash
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          # Install dependencies
          pip install -r requirements.txt
          pip install .

          dvc repro

          # Report metrics
          echo "## Metrics" > report.md
          git fetch --prune

          dvc metrics diff main --show-md | grep "Change\|\-\-\-" >> report.md
          dvc metrics diff main --show-md | grep -v "threshold-" | grep "weighted" | sort >> report.md
          sed "s/results\///g" -i report.md

          cml-send-comment report.md
 
          dvc push          

      - uses: EndBug/add-and-commit@v7
        with:
           add: 'dvc.lock --force'
           pull_strategy: 'NO-PULL'
           message: 'chg: dvc repro'

[0x2b3bfa0]

You’re welcome, @ivyleavedtoadflax!

Then, it’s not a bug, but a feature! 😉

[ivyleavedtoadflax]

ahhh ok, that makes sense. Actually that is very helpful because AWS seem to have rather tight limits on how many vCPUs are available in the G and P ranges. So, queuing up jobs in this way will prevent a job from failing because an instance type is not available.

Thanks @0x2b3bfa0 !