Avoid hanging when Azure CLI has autoupgrade enabled and upgrade is available
See original GitHub issueIs your feature request related to a problem? Please describe. Azure SDK hangs quietly when the azure cli has autoupgrade on, and there is an upgrade available. This is because the subprocess hangs waiting for user input:
subprocess.TimeoutExpired: Command '['cmd', '/c', 'az account get-access-token --output json --resource https://vault.azure.net']' timed out after 10 seconds
Describe the solution you’d like
First help push the az cli team to support --no-auto-upgrade
or similar mechanism (https://github.com/Azure/azure-cli/issues/21249), then use this mechanism to avoid hanging.
To reproduce Pre request from @xiangyan99. This is a bit theoretical for now, since I now have the latest CLI, but something like this:
- Install the Azure CLI of an older version
- Enable auto upgrade using
az config set auto-upgrade.enable=yes
- Run some SDK call that calls out to the cli. In my case, the code is essentially this:
client = SecretClient(vault_url=vault_url, credential=self.azure_credential)
secret = client.get_secret(secret_name)
Sample problem output
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 507, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1134, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1510, in _communicate
raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['cmd', '/c', 'az account get-access-token --output json --resource https://vault.azure.net']' timed out after 10 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "redacted.py", line 39, in get_secret
secret = client.get_secret(secret_name)
File "redacted\.venv\lib\site-packages\azure\core\tracing\decorator.py", line 83, in wrapper_use_tracer
return func(*args, **kwargs)
File "redacted\.venv\lib\site-packages\azure\keyvault\secrets\_client.py", line 67, in get_secret
bundle = self._client.get_secret(
File "redacted\.venv\lib\site-packages\azure\keyvault\secrets\_generated\_operations_mixin.py", line 1515, in get_secret
return mixin_instance.get_secret(vault_base_url, secret_name, secret_version, **kwargs)
File "redacted\.venv\lib\site-packages\azure\keyvault\secrets\_generated\v7_2\operations\_key_vault_client_operations.py", line 286, in get_secret
pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
File "redacted\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 211, in run
return first_node.send(pipeline_request) # type: ignore
File "redacted\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
response = self.next.send(request)
File "redacted\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
response = self.next.send(request)
File "redacted\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
response = self.next.send(request)
[Previous line repeated 2 more times]
File "redacted\.venv\lib\site-packages\azure\core\pipeline\policies\_redirect.py", line 158, in send
response = self.next.send(request)
File "redacted\.venv\lib\site-packages\azure\core\pipeline\policies\_retry.py", line 445, in send
response = self.next.send(request)
File "redacted\.venv\lib\site-packages\azure\keyvault\secrets\_shared\challenge_auth_policy.py", line 111, in send
self._handle_challenge(request, challenge)
File "redacted\.venv\lib\site-packages\azure\keyvault\secrets\_shared\challenge_auth_policy.py", line 137, in _handle_challenge
self._token = self._credential.get_token(scope)
File "redacted\.venv\lib\site-packages\azure\identity\_internal\decorators.py", line 30, in wrapper
token = fn(*args, **kwargs)
File "redacted\.venv\lib\site-packages\azure\identity\_credentials\azure_cli.py", line 73, in get_token
output = _run_command(command)
File "redacted\.venv\lib\site-packages\azure\identity\_credentials\azure_cli.py", line 141, in _run_command
return subprocess.check_output(args, **kwargs)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 516, in run
exc.stdout, exc.stderr = process.communicate()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1134, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1508, in _communicate
self.stdout_thread.join(self._remaining_time(endtime))
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\threading.py", line 1053, in join
self._wait_for_tstate_lock()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\threading.py", line 1073, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
KeyboardInterrupt
Underlying az cli behaviour This is what is in fact happening behind the scenes, when the Python SDK calls out to the cli:
> az account get-access-token --output json --resource https://vault.azure.net
{
...
}
New Azure CLI version available. Running 'az upgrade' to update automatically.
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Your current Azure CLI version is 2.32.0. Latest version available is 2.33.0.
Please check the release notes first: https://docs.microsoft.com/cli/azure/release-notes-azure-cli
Do you want to continue? (Y/n):
Issue Analytics
- State:
- Created 2 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
AKS upgrade 1.19.11 to latest one - Microsoft Q&A
Automatically completed upgrades are functionally the same as manual upgrades. The timing of upgrades is determined by the selected channel.
Read more >How to update the Azure CLI | Microsoft Learn
Learn how to update the Azure Command-Line Interface (CLI) by performing a manual update or enabling auto-upgrade for the CLI.
Read more >Release notes & updates – Azure CLI - Microsoft Learn
[BREAKING CHANGE] az mysql flexible-server create/update : --high-availability available parameter is changed from 'Enabled' to 'ZoneRedundant' and 'SameZone' .
Read more >Sign in with Azure CLI — Login and Authentication
Learn the different authentication types for your Azure CLI login — sign in with Azure CLI automatically, locally, or interactively using ...
Read more >Troubleshooting Azure Windows VM extension failures
If you're running scripts on the VM using Custom Script Extension, you could sometimes run into an error where VM was created successfully...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@xiangyan99 I added more information, though perhaps the key issue to be understood is this: Once the azure cli has autoupgrade enabled, it will ask for input which it will never get. When this happens in an invisible subprocess from the python SDK, the SDK appears to hang:
Given https://github.com/Azure/azure-cli/issues/21249 is closed, close this one as well.