Deployments slow to locations that differ from the host machine

Describe the bug When using the az deployment command, it appears that if the location of the physical machine on which the command is executed differs from the target location of the deployment, there is severe degradation in the speed of the deployment.

I encountered this issue initially when using GitHub Actions to deploy to the North Europe location. Locally, my deployment would execute in about 8 minutes. When running on GitHub Actions, it would frequently time out after an hour when running exactly the same command on exactly the same version of the CLI.

GitHub Actions was running an earlier version of Python than I was locally, so in order to discount that as a possibility, I created a Docker container using the mcr.microsoft.com/azure-cli image, in which I baked my deployment command. This ran in around the same time locally, but when GitHub actions executed the command, the deployment was unstable.

I was at a loss. The only difference was the hosting environment, so I enabled the --debug switch on the az deployment command, and I noticed that the x-ms-routing-request-id header was frequently pointing to one of the Azure US regions (geographically closest to the GitHub Actions hosted runners), whereas locally it was being routed to North Europe (I’m based in the UK).

In order to verify that the location of the physical machine on which the command was run was indeed a factor, I set the location of my VPN to the US, and low and behold, my deployment times skyrocketed, and experienced the same degradation as I observed with GitHub Actions. I am not able to influence the physical location of the GitHub Actions runner unless I use the self-hosted variety, which I’m not prepared to do. Altering the --location parameter of the command has no effect.

Expected behavior The --location parameter of the az deployment command should be used to route requests to the Azure deployment service, rather than the geographical location of the machine on which the command is executed, since this is semantically a reference to the primary region into which you’d like to deploy. There is something about the way Azure deploys across regions that appears to be causing unacceptable delays for anything other than the simplest of ARM templates. I would have expected the service to hand off the deployment template to the most appropriate Azure region as specified in the parameters, rather than attempting to deploy across regions, which I assume is what is happening. Any insight into this matter would be greatly appreciated, as this is effectively preventing us from using continuous integration to deploy infrastructure changes, at least on GitHub Actions hosted runners.

[source-studio]

Thanks for your response. In my case, the deployment was to the tenant level, as I’m trying to automate the creation of Management Groups and Subscriptions, but the latency issue happens regardless of the target scope of the deployment.

I’ll open a support request, as you suggested but I fail to believe I’m the only one experiencing this issue, so I was hoping the issue could be tracked in an open forum. I’ll report back with any pertinent findings.

[msftbot[bot]]

Hi, we’re sending this friendly reminder because we haven’t heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don’t hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!