GCP cloud runner not terminating

This is a repeat of #661, which was supposedly fixed in #653. Unfortunately, I’m not seeing any changes in the shutdown behavior of my GCP compute instances. That is, they keep running past the timeout interval.

I’m using the same workflow as before (in #661):

name: 'Train-in-the-cloud-GCP'
on: 
  workflow_dispatch:

jobs:
  deploy-runner:
    runs-on: [ubuntu-latest]
    steps:
      - uses: iterative/setup-cml@v1
      - uses: actions/checkout@v2
      - name: 'Deploy runner on GCP'
        shell: bash
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
          # Notice use of `GOOGLE_APPLICATION_CREDENTIALS_DATA` instead of
          # `GOOGLE_APPLICATION_CREDENTIALS`. Contrary to what docs suggest, the
          # latter causes problems for terraform.
          GOOGLE_APPLICATION_CREDENTIALS_DATA: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
        run: |
          cml-runner \
          --cloud gcp \
          --cloud-region europe-west1-b	 \
          --cloud-type=n1-standard-1 \
          --labels=cml-runner
          
  model-training:
    needs: deploy-runner
    runs-on: [self-hosted, cml-runner]
    container: docker://dvcorg/cml-py3:latest
    steps:
      - uses: actions/checkout@v2
      - name: 'Train my dummy model'
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
        run: |
          echo "Training a super awesome model"
          sleep 5
          echo "Training complete"

Anyway, this seems to contradict the tests, as @DavidGOrtega explains in the comments under #653:

[…] tests with TPI indicates that the instances are disposed after the expected time.

Any idea what I might be doing wrong?

[lemontheme]

Hi @dacbd, sorry to keep you waiting. Been a while since I looked at this.

Anyway, I’m happy to confirm that instances are now indeed stopped and deleted as expected! 😃 That’s using the exact same workflow as above. Great to see you’ve made progress with this. Thanks!

[dacbd]

@lemontheme I believe this issue is resolved, can you confirm your workflow is functional without any workarounds?