[BUG] Azure.CLI timeout when running in AzureHosted Agents

Library name and version

Azure.identity 1.7.0

Describe the bug

I’ve been running into this the last couple of months in our Azure DevOps pipelines.

In the pipeline, we use an Azure Powershell task to switch to the Service Principal account when running our C# code. When trying to get credentials to connect to the keyvault, it seems to always timeout now. This works for developers without issues (pulling credentials from Visual Studio or local cache), but when running in the pipeline (using Azure Hosted Build agents), the request times out 99 times out of a hundred now:

Loading secrets from https://mykeyvault.vault.azure.net

There was an issue with keyvault values
Azure PowerShell authentication timed out.

Before, this same procedure (been in use for over 2 years now) worked reliably, but as of late, it basically always fails. Every 1/100 requests works, so I know that it is possible for it to connect successfully still.

I’ve tried to up the timeout the DefaultAzureCredentials, but it doesn’t seem to actually change anything, or even wait longer before failing.

Console.WriteLine($"Loading secrets from ${KeyVaultUrl}");
                
SecretClientOptions secretOptions = new SecretClientOptions();
DefaultAzureCredentialOptions credentialOptions = new DefaultAzureCredentialOptions();
credentialOptions.Retry.NetworkTimeout = TimeSpan.FromSeconds(20);
DefaultAzureCredential credentials = new DefaultAzureCredential(credentialOptions);

this.Client = new SecretClient(new Uri(KeyVaultUrl), credentials, secretOptions);

I’ve updated to latest Azure.Identity (1.7) and Azure.Security.KeyVaults.Secrets (4.4.0).

Expected behavior

Azure PowerShell token auth shouldn’t timeout.

Actual behavior

Loading secrets from https://mykeyvault.vault.azure.net

There was an issue with keyvault values
Azure PowerShell authentication timed out.

Reproduction Steps

1. Create a C# program that makes a call to AzureKeyVault:

Console.WriteLine($"Loading secrets from ${KeyVaultUrl}");
                
SecretClientOptions secretOptions = new SecretClientOptions();
DefaultAzureCredentialOptions credentialOptions = new DefaultAzureCredentialOptions();
credentialOptions.Retry.NetworkTimeout = TimeSpan.FromSeconds(20);
DefaultAzureCredential credentials = new DefaultAzureCredential(credentialOptions);

this.Client = new SecretClient(new Uri(KeyVaultUrl), credentials, secretOptions);
var keys = this.Client.GetPropertiesOfSecrets();

1. Create a Build in AzureDevOps that uses AzureHosted Agents, windows-latest
2. Create an AzurePowershell task that authenticates with your resource group.
3. Call the C# program in the AzurePowershell Task.
4. Notice the AzureCLI timeout when returning the token sometimes
5. Increasing the NetworkTimeout has no effect.

Environment

Azure Powershell Output:

2022-10-18T22:01:22.9631030Z ##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\418a992a-3b46-4a1c-bd77-650312f89f54.ps1'"
2022-10-18T22:01:24.7635896Z ##[command]Import-Module -Name C:\Modules\az_8.3.0\Az.Accounts\2.10.0\Az.Accounts.psd1 -Global
2022-10-18T22:01:26.7303071Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2022-10-18T22:01:27.4586165Z ##[command]Clear-AzContext -Scope Process
2022-10-18T22:01:27.5988331Z ##[command]Connect-AzAccount -ServicePrincipal -Tenant TenantID -Credential System.Management.Automation.PSCredential -Environment AzureCloud @processScope
2022-10-18T22:01:29.7357522Z ##[command] Set-AzContext -SubscriptionId SubscriptionID -TenantId TenantID

[KrisSodroski]

Filed here:

https://github.com/Azure/azure-powershell/issues/19955

[christothes]

Closing this one unless we discover there is something related to the behavior of Azure.Identity itself.