question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Avoid hanging when Azure CLI has autoupgrade enabled and upgrade is available

See original GitHub issue

Is your feature request related to a problem? Please describe. Azure SDK hangs quietly when the azure cli has autoupgrade on, and there is an upgrade available. This is because the subprocess hangs waiting for user input:

subprocess.TimeoutExpired: Command '['cmd', '/c', 'az account get-access-token --output json --resource https://vault.azure.net']' timed out after 10 seconds

Describe the solution you’d like First help push the az cli team to support --no-auto-upgrade or similar mechanism (https://github.com/Azure/azure-cli/issues/21249), then use this mechanism to avoid hanging.

To reproduce Pre request from @xiangyan99. This is a bit theoretical for now, since I now have the latest CLI, but something like this:

  • Install the Azure CLI of an older version
  • Enable auto upgrade using az config set auto-upgrade.enable=yes
  • Run some SDK call that calls out to the cli. In my case, the code is essentially this:
        client = SecretClient(vault_url=vault_url, credential=self.azure_credential)
        secret = client.get_secret(secret_name)

Sample problem output

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 507, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1134, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1510, in _communicate
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['cmd', '/c', 'az account get-access-token --output json --resource https://vault.azure.net']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "redacted.py", line 39, in get_secret
    secret = client.get_secret(secret_name)
  File "redacted\.venv\lib\site-packages\azure\core\tracing\decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "redacted\.venv\lib\site-packages\azure\keyvault\secrets\_client.py", line 67, in get_secret
    bundle = self._client.get_secret(
  File "redacted\.venv\lib\site-packages\azure\keyvault\secrets\_generated\_operations_mixin.py", line 1515, in get_secret
    return mixin_instance.get_secret(vault_base_url, secret_name, secret_version, **kwargs)
  File "redacted\.venv\lib\site-packages\azure\keyvault\secrets\_generated\v7_2\operations\_key_vault_client_operations.py", line 286, in get_secret
    pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
  File "redacted\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 211, in run
    return first_node.send(pipeline_request)  # type: ignore
  File "redacted\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "redacted\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "redacted\.venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  [Previous line repeated 2 more times]
  File "redacted\.venv\lib\site-packages\azure\core\pipeline\policies\_redirect.py", line 158, in send
    response = self.next.send(request)
  File "redacted\.venv\lib\site-packages\azure\core\pipeline\policies\_retry.py", line 445, in send
    response = self.next.send(request)
  File "redacted\.venv\lib\site-packages\azure\keyvault\secrets\_shared\challenge_auth_policy.py", line 111, in send
    self._handle_challenge(request, challenge)
  File "redacted\.venv\lib\site-packages\azure\keyvault\secrets\_shared\challenge_auth_policy.py", line 137, in _handle_challenge
    self._token = self._credential.get_token(scope)
  File "redacted\.venv\lib\site-packages\azure\identity\_internal\decorators.py", line 30, in wrapper
    token = fn(*args, **kwargs)
  File "redacted\.venv\lib\site-packages\azure\identity\_credentials\azure_cli.py", line 73, in get_token
    output = _run_command(command)
  File "redacted\.venv\lib\site-packages\azure\identity\_credentials\azure_cli.py", line 141, in _run_command
    return subprocess.check_output(args, **kwargs)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 516, in run
    exc.stdout, exc.stderr = process.communicate()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1134, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1508, in _communicate
    self.stdout_thread.join(self._remaining_time(endtime))
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\threading.py", line 1053, in join
    self._wait_for_tstate_lock()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\threading.py", line 1073, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
KeyboardInterrupt

Underlying az cli behaviour This is what is in fact happening behind the scenes, when the Python SDK calls out to the cli:

> az account get-access-token --output json --resource https://vault.azure.net
{
    ...
}
New Azure CLI version available. Running 'az upgrade' to update automatically.
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Your current Azure CLI version is 2.32.0. Latest version available is 2.33.0.
Please check the release notes first: https://docs.microsoft.com/cli/azure/release-notes-azure-cli
Do you want to continue? (Y/n): 

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
elygrecommented, Feb 10, 2022

@xiangyan99 I added more information, though perhaps the key issue to be understood is this: Once the azure cli has autoupgrade enabled, it will ask for input which it will never get. When this happens in an invisible subprocess from the python SDK, the SDK appears to hang:

> az account get-access-token --output json --resource https://vault.azure.net
{
    ...
}
New Azure CLI version available. Running 'az upgrade' to update automatically.
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Your current Azure CLI version is 2.32.0. Latest version available is 2.33.0.
Please check the release notes first: https://docs.microsoft.com/cli/azure/release-notes-azure-cli
Do you want to continue? (Y/n): 
0reactions
xiangyan99commented, Mar 28, 2022

Given https://github.com/Azure/azure-cli/issues/21249 is closed, close this one as well.

Read more comments on GitHub >

github_iconTop Results From Across the Web

AKS upgrade 1.19.11 to latest one - Microsoft Q&A
Automatically completed upgrades are functionally the same as manual upgrades. The timing of upgrades is determined by the selected channel.
Read more >
How to update the Azure CLI | Microsoft Learn
Learn how to update the Azure Command-Line Interface (CLI) by performing a manual update or enabling auto-upgrade for the CLI.
Read more >
Release notes & updates – Azure CLI - Microsoft Learn
[BREAKING CHANGE] az mysql flexible-server create/update : --high-availability available parameter is changed from 'Enabled' to 'ZoneRedundant' and 'SameZone' .
Read more >
Sign in with Azure CLI — Login and Authentication
Learn the different authentication types for your Azure CLI login — sign in with Azure CLI automatically, locally, or interactively using ...
Read more >
Troubleshooting Azure Windows VM extension failures
If you're running scripts on the VM using Custom Script Extension, you could sometimes run into an error where VM was created successfully...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found