Intermittent rate limit errors cause CI/CD pipelines to fail when using the azure cli

Describe the bug We are running the azure cli in a pipeline on Bitbucket to deploy Bicep templates to our development environment, and soon enough to production as well. Intermittently, our pipelines fail due to rate limits when the Azure CLI performs a version check for Bicep. Running az deployment group create ... sometimes fails with:

DEBUG: cli.azure.cli.core.util: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/azure/cli/command_modules/resource/_bicep.py", line 157, in get_bicep_latest_release_tag
    response.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: rate limit exceeded for url: https://api.github.com/repos/Azure/bicep/releases/latest
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/usr/local/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/usr/local/lib/python3.9/site-packages/azure/cli/command_modules/resource/custom.py", line 542, in deploy_arm_template_at_resource_group
    return _deploy_arm_template_at_resource_group(cmd=cmd,
  File "/usr/local/lib/python3.9/site-packages/azure/cli/command_modules/resource/custom.py", line 571, in _deploy_arm_template_at_resource_group
    deployment_properties = _prepare_deployment_properties_unmodified(cmd, 'resourceGroup', template_file=template_file,
  File "/usr/local/lib/python3.9/site-packages/azure/cli/command_modules/resource/custom.py", line 982, in _prepare_deployment_properties_unmodified
    run_bicep_command(["build", "--stdout", template_file])
  File "/usr/local/lib/python3.9/site-packages/azure/cli/command_modules/resource/_bicep.py", line 63, in run_bicep_command
    ensure_bicep_installation(stdout=False)
  File "/usr/local/lib/python3.9/site-packages/azure/cli/command_modules/resource/_bicep.py", line 106, in ensure_bicep_installation
    release_tag = release_tag if release_tag else get_bicep_latest_release_tag()
  File "/usr/local/lib/python3.9/site-packages/azure/cli/command_modules/resource/_bicep.py", line 160, in get_bicep_latest_release_tag
    raise ClientRequestError(f"Error while attempting to retrieve the latest Bicep version: {err}.")
azure.cli.core.azclierror.ClientRequestError: Error while attempting to retrieve the latest Bicep version: 403 Client Error: rate limit exceeded for url: https://api.github.com/repos/Azure/bicep/releases/latest.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/azure/cli/core/commands/arm.py", line 109, in handle_template_based_exception
    raise CLIError(ex.inner_exception.error.message)
AttributeError: 'ClientRequestError' object has no attribute 'inner_exception'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/usr/local/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/usr/local/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 713, in _run_job
    return cmd_copy.exception_handler(ex)
  File "/usr/local/lib/python3.9/site-packages/azure/cli/core/commands/arm.py", line 114, in handle_template_based_exception
    raise CLIError(ex)
knack.util.CLIError: Error while attempting to retrieve the latest Bicep version: 403 Client Error: rate limit exceeded for url: https://api.github.com/repos/Azure/bicep/releases/latest.
ERROR: cli.azure.cli.core.azclierror: Error while attempting to retrieve the latest Bicep version: 403 Client Error: rate limit exceeded for url: https://api.github.com/repos/Azure/bicep/releases/latest.
ERROR: az_command_data_logger: Error while attempting to retrieve the latest Bicep version: 403 Client Error: rate limit exceeded for url: https://api.github.com/repos/Azure/bicep/releases/latest.
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f458d57df70>]
INFO: az_command_data_logger: exit code: 1
INFO: cli.__main__: Command ran in 0.642 seconds (init: 0.137, invoke: 0.505)
INFO: telemetry.save: Save telemetry record of length 3236 in cache
WARNING: telemetry.check: Negative: The /root/.azure/telemetry.txt was modified at 2022-02-17 09:03:42.859799, which in less than 600.000000 s

To Reproduce Run az deployment group create ... in a Bitbucket pipeline, and hope for the worst I guess.

Expected behavior We expect that our az deployment group create ... behaves consistently, and is not failing due to rate limits in version checks.

Environment summary Bitbucket pipelines using the image:

mcr.microsoft.com/azure-cli@sha256:d2a53ad219b28567e256693eba1b985bbebea4687fddf293b0ab6d75ab9f728b

Additional context We are worried that this behavior may be caused by multiple clients sharing the same IP address in Bitbucket pipelines, which thus increases the risk of hitting this API limit. In any event, we can think of two solutions that seem reasonable:

• Do not return an erroneous exit code just because the version check fails, but rather log a warning and carry on with the command
• Add a flag to disable version checks entirely, which could be useful in CI/CD environments

If this is simply because we are using the tool incorrectly, please let us know how to do better!

[alex-frankel]

Just as FYI - we plan to host bicep on a CDN in the near future to avoid hitting the rate limit: https://github.com/Azure/bicep/issues/3689

[GordonBeeming]

@alex-frankel / @navba-MSFT Any ETA on this moving to a CDN? 7 months feels a little excessive to just let the tooling break.

bicep #3689