Deploying multiple runners fails with github rate limit after version `0.5.0`

Hey everyone, Noticed the following error appearing from version 0.5.0 onwards. We have a stage in our github actions pipeline that deploys multiple runners, as such:

  deploy_runners:
    name: Deploy Cloud Instances
    needs: [setup_config]

    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - uses: iterative/setup-cml@v1
        with:
          version:  ${{ env.cml_version }}

      - name: "Deploy runner on EC2"
        shell: bash
        env:
          repo_token: ${{ secrets.ACCESS_TOKEN_CML }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID_TESTING }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY_TESTING }}
          CASE_NAME: ${{ matrix.case_name }}
          N_RUNNERS: ${{ fromJson(needs.setup_config.outputs.json_string).n_runners }}

        run: |
          for (( i=1; i<=N_RUNNERS; i++ ))
          do
            echo "Deploying runner ${i}"
            cml-runner \
            --cloud aws \
            --cloud-region eu-west-2 \
            --cloud-type=m \
            --cloud-hdd-size 100 \
            --cloud-spot \
            --labels=cml-runner &
          done
          wait
          echo "Deployed ${N_RUNNERS} runners."

I noticed that the following error has started appearing after version 0.5.0 of cml was released:

2021-07-11T12:25:17.9903383Z RequestError [HttpError]: You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.
2021-07-11T12:25:17.9905919Z     at /usr/local/lib/node_modules/@dvcorg/cml/node_modules/@octokit/request/dist-node/index.js:86:21
2021-07-11T12:25:17.9907296Z     at processTicksAndRejections (internal/process/task_queues.js:95:5)
2021-07-11T12:25:17.9909090Z     at async Object.next (/usr/local/lib/node_modules/@dvcorg/cml/node_modules/@octokit/plugin-paginate-rest/dist-node/index.js:124:28)
2021-07-11T12:25:17.9910621Z     at async Github.getRunners (/usr/local/lib/node_modules/@dvcorg/cml/src/drivers/github.js:233:17)
2021-07-11T12:25:17.9911889Z     at async CML.getRunners (/usr/local/lib/node_modules/@dvcorg/cml/src/cml.js:221:12)
2021-07-11T12:25:17.9913066Z     at async CML.runnerByName (/usr/local/lib/node_modules/@dvcorg/cml/src/cml.js:226:21)
2021-07-11T12:25:17.9914381Z     at async run (/usr/local/lib/node_modules/@dvcorg/cml/bin/cml-runner.js:326:7) {
2021-07-11T12:25:17.9915289Z   status: 403,
2021-07-11T12:25:17.9916315Z   response: {
2021-07-11T12:25:17.9917781Z     url: 'https://api.github.com/repositories/263006174/actions/runners?page=8',
2021-07-11T12:25:17.9918908Z     status: 403,
2021-07-11T12:25:17.9919585Z     headers: {
2021-07-11T12:25:17.9920423Z       'access-control-allow-origin': '*',
2021-07-11T12:25:17.9923366Z       'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
2021-07-11T12:25:17.9926135Z       connection: 'close',
2021-07-11T12:25:17.9926949Z       'content-encoding': 'gzip',
2021-07-11T12:25:17.9937883Z       'content-security-policy': "default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events translator.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com render-temp.githubusercontent.com viewscreen.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com secured-user-images.githubusercontent.com/; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com",
2021-07-11T12:25:17.9947806Z       'content-type': 'application/json; charset=utf-8',
2021-07-11T12:25:17.9948751Z       date: 'Sun, 11 Jul 2021 12:25:17 GMT',
2021-07-11T12:25:17.9950048Z       'expect-ct': 'max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"',
2021-07-11T12:25:17.9951813Z       'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
2021-07-11T12:25:17.9953126Z       'retry-after': '60',
2021-07-11T12:25:17.9953893Z       server: 'GitHub.com',
2021-07-11T12:25:17.9955102Z       'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
2021-07-11T12:25:17.9956369Z       'transfer-encoding': 'chunked',
2021-07-11T12:25:17.9957547Z       vary: 'Accept-Encoding, Accept, X-Requested-With',
2021-07-11T12:25:17.9958535Z       'x-content-type-options': 'nosniff',
2021-07-11T12:25:17.9959415Z       'x-frame-options': 'deny',
2021-07-11T12:25:17.9960321Z       'x-github-media-type': 'github.v3; format=json',
2021-07-11T12:25:17.9961382Z       'x-github-request-id': '04FC:02AD:4BF0BFD:907BDD4:60EAE32D',
2021-07-11T12:25:17.9962208Z       'x-xss-protection': '0'
2021-07-11T12:25:17.9962722Z     },
2021-07-11T12:25:17.9963157Z     data: {
2021-07-11T12:25:17.9964543Z       documentation_url: 'https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#abuse-rate-limits',
2021-07-11T12:25:17.9966598Z       message: 'You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.'
2021-07-11T12:25:17.9967480Z     }
2021-07-11T12:25:17.9967921Z   },
2021-07-11T12:25:17.9968390Z   request: {
2021-07-11T12:25:17.9969032Z     method: 'GET',
2021-07-11T12:25:17.9970129Z     url: 'https://api.github.com/repositories/263006174/actions/runners?page=8',
2021-07-11T12:25:17.9970995Z     headers: {
2021-07-11T12:25:17.9971876Z       accept: 'application/vnd.github.v3+json',
2021-07-11T12:25:17.9973977Z       'user-agent': 'octokit-core.js/3.5.1 Node.js/14.17.1 (linux; x64)',
2021-07-11T12:25:17.9975171Z       authorization: 'token [REDACTED]'
2021-07-11T12:25:17.9977395Z     },
2021-07-11T12:25:17.9978105Z     request: { agent: [Agent], hook: [Function: bound bound register] }
2021-07-11T12:25:17.9978736Z   }
2021-07-11T12:25:17.9979202Z }

Any clues as to what would be going on? Reverting to 0.4.7 fixed the problem.

[DavidGOrtega]

Im finishing the PR adding throttling and being smarter when consuming it

[thatGreekGuy96]

I’ll give that a try, it shouldn’t have too big of an impact.