aks command invoke failing due to init container failure

Related command

az aks command invoke

Describe the bug

When running az aks command invoke to run any command eg kubectl get pods -A, the command returns with:

(KubernetesOperationError) Failed to run command due to cluster perf issue, container command-4a5fb8583ae44b069516eb2d9db9fc88 in aks-command namespace did not start within 30s on your cluster, retry may helps. If issue persist, you may need to tune your cluster with better performance (larger node/paid tier). Code: KubernetesOperationError Message: Failed to run command due to cluster perf issue, container command-4a5fb8583ae44b069516eb2d9db9fc88 in aks-command namespace did not start within 30s on your cluster, retry may helps. If issue persist, you may need to tune your cluster with better performance (larger node/paid tier).

Actual cause of error is a failure in the init container:

copying default SA wait for AAD token inotifywait: error while loading shared libraries: libinotifytools.so.0: cannot open shared object file: No such file or directory Error from server (BadRequest): container “user-command” in pod “command-4a5fb8583ae44b069516eb2d9db9fc88” is waiting to start: PodInitializing

To Reproduce

az aks command invoke --resource-group <my-rg> --name <my-aks-cluster> --command ‘kubectl get pods -A’

Expected behavior

Command to run successfully

Environment summary

azure-cli 2.37.0 Tested on Windows 11, Azure DevOps pipeline (Ubuntu), WSL (Ubuntu)

Additional context

[rcgokhale]

This is affecting us as well. All our CI/CD pipelines are broken and kind of P1 for us. Issue seems to appear only when “AKS-managed Azure Active Directory” option is enabled and “az aks command invoke” is used.

We use “az aks command invoke” on Microsoft hosted agents on azure devops with private cluster enabled.

kubectl get pods -n aks-command command-527f7f3d04ab440da4fb36229fe57d39 0/1 Init:Error 0 5m59s

kubectl logs command-ed4909772b794cd68bcc1e7ded55cdad -n aks-command Defaulted container “user-command” out of: user-command, init-command (init) Error from server (BadRequest): container “user-command” in pod “command-ed4909772b794cd68bcc1e7ded55cdad” is terminated.

[datadot]

@navba-MSFT please find sanitised debug output. If you require a non sanitised version, please let me know a secure mechanism to send. Thanks.

az aks command invoke --resource-group <resource_group> --name <cluster_name> --command “kubectl get pods -A” --debug cli.knack.cli: Command arguments: [‘aks’, ‘command’, ‘invoke’, ‘–resource-group’, ‘<resource_group>’, ‘–name’, ‘<cluster_name>’, ‘–command’, ‘kubectl get pods -A’, ‘–debug’] cli.knack.cli: init debug log: Enable color in terminal. cli.knack.cli: Event: Cli.PreExecute [] cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f31a1529280>, <function OutputProducer.on_global_arguments at 0x7f31a14cd160>, <function CLIQuery.on_global_arguments at 0x7f31a145f670>] cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate [] cli.azure.cli.core: Modules found from index for ‘aks’: [‘azure.cli.command_modules.acs’] cli.azure.cli.core: Loading command modules: cli.azure.cli.core: Name Load Time Groups Commands cli.azure.cli.core: acs 0.108 10 64 cli.azure.cli.core: Total (1) 0.108 10 64 cli.azure.cli.core: Loaded 10 groups, 64 commands. cli.azure.cli.core: Found a match in the command table. cli.azure.cli.core: Raw command : aks command invoke cli.azure.cli.core: Command table: aks command invoke cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f31a0c415e0>] cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to ‘2022-06-07.08-13-59.aks_command_invoke.102628.log’. az_command_data_logger: command args: aks command invoke --resource-group {} --name {} --command {} --debug cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f31a0b89700>] cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad [] cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f31a0af9d30>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f31a0af9e50>] cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded [] cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [] cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f31a14cd1f0>, <function CLIQuery.handle_query_parameter at 0x7f31a145f700>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f31a0af9dc0>] cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ContainerServiceClient cli.azure.cli.core.auth.persistence: build_persistence: location=‘.azure/msal_token_cache.json’, encrypt=False cli.azure.cli.core.auth.binary_cache: load: .azure/msal_http_cache.bin urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) msal.authority: openid_config = {‘token_endpoint’: ‘https://login.microsoftonline.com/<subscription_id>/oauth2/v2.0/token’, ‘token_endpoint_auth_methods_supported’: [‘client_secret_post’, ‘private_key_jwt’, ‘client_secret_basic’], ‘jwks_uri’: ‘https://login.microsoftonline.com/<subscription_id>/discovery/v2.0/keys’, ‘response_modes_supported’: [‘query’, ‘fragment’, ‘form_post’], ‘subject_types_supported’: [‘pairwise’], ‘id_token_signing_alg_values_supported’: [‘RS256’], ‘response_types_supported’: [‘code’, ‘id_token’, ‘code id_token’, ‘id_token token’], ‘scopes_supported’: [‘openid’, ‘profile’, ‘email’, ‘offline_access’], ‘issuer’: ‘https://login.microsoftonline.com/<subscription_id>/v2.0’, ‘request_uri_parameter_supported’: False, ‘userinfo_endpoint’: ‘https://graph.microsoft.com/oidc/userinfo’, ‘authorization_endpoint’: ‘https://login.microsoftonline.com/<subscription_id>/oauth2/v2.0/authorize’, ‘device_authorization_endpoint’: ‘https://login.microsoftonline.com/<subscription_id>/oauth2/v2.0/devicecode’, ‘http_logout_supported’: True, ‘frontchannel_logout_supported’: True, ‘end_session_endpoint’: ‘https://login.microsoftonline.com/<subscription_id>/oauth2/v2.0/logout’, ‘claims_supported’: [‘sub’, ‘iss’, ‘cloud_instance_name’, ‘cloud_instance_host_name’, ‘cloud_graph_host_name’, ‘msgraph_host’, ‘aud’, ‘exp’, ‘iat’, ‘auth_time’, ‘acr’, ‘nonce’, ‘preferred_username’, ‘name’, ‘tid’, ‘ver’, ‘at_hash’, ‘c_hash’, ‘email’], ‘kerberos_endpoint’: ‘https://login.microsoftonline.com/<subscription_id>/kerberos’, ‘tenant_region_scope’: ‘EU’, ‘cloud_instance_name’: ‘microsoftonline.com’, ‘cloud_graph_host_name’: ‘graph.windows.net’, ‘msgraph_host’: ‘graph.microsoft.com’, ‘rbac_url’: ‘https://pas.windows.net’} cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=(‘https://management.core.windows.net//.default’,), kwargs={} cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=(‘https://management.core.windows.net//.default’,), kwargs={} msal.application: Found 1 RTs matching {‘environment’: ‘login.microsoftonline.com’, ‘home_account_id’: '', ‘family_id’: ‘1’} msal.telemetry: Generate or reuse correlation_id: d5a9d321-2917-4c2f-85c5-260b130eb06b msal.application: Cache attempts an RT urllib3.connectionpool: Starting new HTTPS connection (1): login.microsoftonline.com:443 urllib3.connectionpool: https://login.microsoftonline.com:443 “POST /<subscription_id>/oauth2/v2.0/token HTTP/1.1” 200 5049 msal.token_cache: event={ “client_id”: "", “data”: { “claims”: null, “refresh_token”: "", “scope”: [ “openid”, “profile”, “https://management.core.windows.net//.default”, “offline_access” ] }, “environment”: “login.microsoftonline.com”, “grant_type”: “refresh_token”, “params”: null, “response”: { “access_token”: "", “client_info”: "", “expires_in”: 4319, “ext_expires_in”: 4319, “foci”: “1”, “id_token”: "", “scope”: “https://management.core.windows.net//user_impersonation https://management.core.windows.net//.default”, “token_type”: “Bearer” }, “scope”: [ “https://management.core.windows.net//user_impersonation”, “https://management.core.windows.net//.default” ], “skip_account_creation”: true, “token_endpoint”: “https://login.microsoftonline.com/<subscription_id>/oauth2/v2.0/token” } cli.azure.cli.core.sdk.policies: Request URL: ‘https://management.azure.com/subscriptions/*****/resourceGroups/<resource_group>/providers/Microsoft.ContainerService/managedClusters/<cluster_name>?api-version=2022-03-01’ cli.azure.cli.core.sdk.policies: Request method: ‘GET’ cli.azure.cli.core.sdk.policies: Request headers: cli.azure.cli.core.sdk.policies: ‘Accept’: ‘application/json’ cli.azure.cli.core.sdk.policies: ‘x-ms-client-request-id’: '’ cli.azure.cli.core.sdk.policies: ‘CommandName’: ‘aks command invoke’ cli.azure.cli.core.sdk.policies: ‘ParameterSetName’: ‘–resource-group --name --command --debug’ cli.azure.cli.core.sdk.policies: ‘User-Agent’: ‘AZURECLI/2.36.0 (DEB) azsdk-python-azure-mgmt-containerservice/19.0.0 Python/3.8.13 (Linux-5.13.0-1023-azure-x86_64-with-glibc2.29)’ cli.azure.cli.core.sdk.policies: ‘Authorization’: '’ cli.azure.cli.core.sdk.policies: Request body: cli.azure.cli.core.sdk.policies: This request has no body urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions//resourceGroups/<resource_group>/providers/Microsoft.ContainerService/managedClusters/<cluster_name>?api-version=2022-03-01 HTTP/1.1" 200 None cli.azure.cli.core.sdk.policies: Response status: 200 cli.azure.cli.core.sdk.policies: Response headers: cli.azure.cli.core.sdk.policies: ‘Cache-Control’: ‘no-cache’ cli.azure.cli.core.sdk.policies: ‘Pragma’: ‘no-cache’ cli.azure.cli.core.sdk.policies: ‘Transfer-Encoding’: ‘chunked’ cli.azure.cli.core.sdk.policies: ‘Content-Type’: ‘application/json’ cli.azure.cli.core.sdk.policies: ‘Content-Encoding’: ‘gzip’ cli.azure.cli.core.sdk.policies: ‘Expires’: ‘-1’ cli.azure.cli.core.sdk.policies: ‘Vary’: ‘Accept-Encoding’ cli.azure.cli.core.sdk.policies: ‘x-ms-routing-request-id’: '’ cli.azure.cli.core.sdk.policies: ‘x-ms-ratelimit-remaining-subscription-reads’: ‘11999’ cli.azure.cli.core.sdk.policies: ‘x-ms-correlation-request-id’: '’ cli.azure.cli.core.sdk.policies: ‘x-ms-request-id’: '’ cli.azure.cli.core.sdk.policies: ‘Strict-Transport-Security’: ‘max-age=31536000; includeSubDomains’ cli.azure.cli.core.sdk.policies: ‘X-Content-Type-Options’: ‘nosniff’ cli.azure.cli.core.sdk.policies: ‘Server’: ‘nginx’ cli.azure.cli.core.sdk.policies: ‘Date’: ‘Tue, 07 Jun 2022 08:14:00 GMT’ cli.azure.cli.core.sdk.policies: Response content: cli.azure.cli.core.sdk.policies: { “id”: "/subscriptions//resourcegroups/<resource_group>/providers/Microsoft.ContainerService/managedClusters/<cluster_name>", “location”: “uksouth”, “name”: “<cluster_name>”, “type”: “Microsoft.ContainerService/ManagedClusters”, “properties”: { “provisioningState”: “Succeeded”, “powerState”: { “code”: “Running” }, “kubernetesVersion”: “1.21.9”, “dnsPrefix”: "", “fqdn”: ".hcp.uksouth.azmk8s.io", “azurePortalFQDN”: "", “privateFQDN”: ".privatelink.uksouth.azmk8s.io", “agentPoolProfiles”: [ { “name”: “frontend”, “count”: 2, “vmSize”: “Standard_D2a_v4”, “osDiskSizeGB”: 256, “osDiskType”: “Managed”, “kubeletDiskType”: “OS”, “workloadRuntime”: “OCIContainer”, “vnetSubnetID”: "/subscriptions//resourceGroups//providers/Microsoft.Network/virtualNetworks//subnets/", “maxPods”: 110, “type”: “VirtualMachineScaleSets”, “scaleDownMode”: “Delete”, “provisioningState”: “Succeeded”, “powerState”: { “code”: “Running” }, “orchestratorVersion”: “1.21.9”, “enableNodePublicIP”: false, “nodeTaints”: [ “access=frontend:NoSchedule” ], “mode”: “User”, “enableEncryptionAtHost”: true, “enableUltraSSD”: false, “osType”: “Linux”, “osSKU”: “Ubuntu”, “nodeImageVersion”: “AKSUbuntu-1804gen2containerd-2022.05.04”, “upgradeSettings”: {}, “enableFIPS”: false }, { “name”: “workers”, “count”: 1, “vmSize”: “Standard_E2a_v4”, “osDiskSizeGB”: 256, “osDiskType”: “Managed”, “kubeletDiskType”: “OS”, “workloadRuntime”: “OCIContainer”, “vnetSubnetID”: "/subscriptions//resourceGroups//providers/Microsoft.Network/virtualNetworks//subnets/", “maxPods”: 110, “type”: “VirtualMachineScaleSets”, “maxCount”: 2, “minCount”: 1, “enableAutoScaling”: true, “provisioningState”: “Succeeded”, “powerState”: { “code”: “Running” }, “orchestratorVersion”: “1.21.9”, “enableNodePublicIP”: false, “mode”: “System”, “enableEncryptionAtHost”: true, “enableUltraSSD”: false, “osType”: “Linux”, “osSKU”: “Ubuntu”, “nodeImageVersion”: “AKSUbuntu-1804gen2containerd-2022.05.04”, “enableFIPS”: false } ], “linuxProfile”: { “adminUsername”: "", “ssh”: { “publicKeys”: [ { “keyData”: "" } ] } }, “servicePrincipalProfile”: { “clientId”: “msi” }, “nodeResourceGroup”: “MC_<resource_group>_<cluster_name>_uksouth”, “enableRBAC”: true, “enablePodSecurityPolicy”: false, “networkProfile”: { “networkPlugin”: “kubenet”, “loadBalancerSku”: “Standard”, “loadBalancerProfile”: { “managedOutboundIPs”: { “count”: 1 }, “effectiveOutboundIPs”: [ { “id”: "/subscriptions//resourceGroups/MC_<resource_group>_<cluster_name>_uksouth/providers/Microsoft.Network/publicIPAddresses/" } ] }, “podCidr”: “10.200.0.0/16”, “serviceCidr”: “10.201.0.0/16”, “dnsServiceIP”: “10.201.0.10”, “dockerBridgeCidr”: “172.17.0.1/16”, “outboundType”: “loadBalancer”, “podCidrs”: [ “10.200.0.0/16” ], “serviceCidrs”: [ “10.201.0.0/16” ], “ipFamilies”: [ “IPv4” ] }, “aadProfile”: { “managed”: true, “adminGroupObjectIDs”: [ "" ], “adminUsers”: null, “enableAzureRBAC”: false, “tenantID”: “<subscription_id>” }, “maxAgentPools”: 100, “privateLinkResources”: [ { “id”: "/subscriptions//resourcegroups/<resource_group>/providers/Microsoft.ContainerService/managedClusters/<cluster_name>/privateLinkResources/management", “name”: “management”, “type”: “Microsoft.ContainerService/managedClusters/privateLinkResources”, “groupId”: “management”, “requiredMembers”: [ “management” ] } ], “apiServerAccessProfile”: { “enablePrivateCluster”: true, “privateDNSZone”: “system”, “enablePrivateClusterPublicFQDN”: true, “privateClusterVersion”: “v1” }, “identityProfile”: { “kubeletidentity”: { “resourceId”: "/subscriptions//resourcegroups/<resource_group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/", “clientId”: "", “objectId”: "" } }, “autoScalerProfile”: { “balance-similar-node-groups”: “false”, “expander”: “random”, “max-empty-bulk-delete”: “10”, “max-graceful-termination-sec”: “600”, “max-node-provision-time”: “15m”, “max-total-unready-percentage”: “45”, “new-pod-scale-up-delay”: “0s”, “ok-total-unready-count”: “3”, “scale-down-delay-after-add”: “10m”, “scale-down-delay-after-delete”: “10s”, “scale-down-delay-after-failure”: “3m”, “scale-down-unneeded-time”: “10m”, “scale-down-unready-time”: “20m”, “scale-down-utilization-threshold”: “0.5”, “scan-interval”: “10s”, “skip-nodes-with-local-storage”: “false”, “skip-nodes-with-system-pods”: “true” }, “disableLocalAccounts”: false, “httpProxyConfig”: { “httpProxy”: "", “httpsProxy”: "", “noProxy”: [ “.azurecr.io”, ".privatelink.uksouth.azmk8s.io", “konnectivity” ], “trustedCa”: "" }, “securityProfile”: {}, “storageProfile”: { “diskCSIDriver”: { “enabled”: false }, “fileCSIDriver”: { “enabled”: false }, “snapshotController”: { “enabled”: false } } }, “identity”: { “type”: “UserAssigned”, “userAssignedIdentities”: { "/subscriptions//resourcegroups/<resource_group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/": { “clientId”: "", “principalId”: "" } } }, “sku”: { “name”: “Basic”, “tier”: “Free” } } urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) msal.authority: openid_config = {‘token_endpoint’: ‘https://login.microsoftonline.com/<subscription_id>/oauth2/v2.0/token’, ‘token_endpoint_auth_methods_supported’: [‘client_secret_post’, ‘private_key_jwt’, ‘client_secret_basic’], ‘jwks_uri’: ‘https://login.microsoftonline.com/<subscription_id>/discovery/v2.0/keys’, ‘response_modes_supported’: [‘query’, ‘fragment’, ‘form_post’], ‘subject_types_supported’: [‘pairwise’], ‘id_token_signing_alg_values_supported’: [‘RS256’], ‘response_types_supported’: [‘code’, ‘id_token’, ‘code id_token’, ‘id_token token’], ‘scopes_supported’: [‘openid’, ‘profile’, ‘email’, ‘offline_access’], ‘issuer’: ‘https://login.microsoftonline.com/<subscription_id>/v2.0’, ‘request_uri_parameter_supported’: False, ‘userinfo_endpoint’: ‘https://graph.microsoft.com/oidc/userinfo’, ‘authorization_endpoint’: ‘https://login.microsoftonline.com/<subscription_id>/oauth2/v2.0/authorize’, ‘device_authorization_endpoint’: ‘https://login.microsoftonline.com/<subscription_id>/oauth2/v2.0/devicecode’, ‘http_logout_supported’: True, ‘frontchannel_logout_supported’: True, ‘end_session_endpoint’: ‘https://login.microsoftonline.com/<subscription_id>/oauth2/v2.0/logout’, ‘claims_supported’: [‘sub’, ‘iss’, ‘cloud_instance_name’, ‘cloud_instance_host_name’, ‘cloud_graph_host_name’, ‘msgraph_host’, ‘aud’, ‘exp’, ‘iat’, ‘auth_time’, ‘acr’, ‘nonce’, ‘preferred_username’, ‘name’, ‘tid’, ‘ver’, ‘at_hash’, ‘c_hash’, ‘email’], ‘kerberos_endpoint’: ‘https://login.microsoftonline.com/<subscription_id>/kerberos’, ‘tenant_region_scope’: ‘EU’, ‘cloud_instance_name’: ‘microsoftonline.com’, ‘cloud_graph_host_name’: ‘graph.windows.net’, ‘msgraph_host’: ‘graph.microsoft.com’, ‘rbac_url’: ‘https://pas.windows.net’} cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('/.default’,), kwargs={} msal.application: Found 1 RTs matching {‘environment’: ‘login.microsoftonline.com’, ‘home_account_id’: '', ‘family_id’: ‘1’} msal.telemetry: Generate or reuse correlation_id: ***** msal.application: Cache attempts an RT urllib3.connectionpool: Starting new HTTPS connection (1): login.microsoftonline.com:443 urllib3.connectionpool: https://login.microsoftonline.com:443 “POST /<subscription_id>/oauth2/v2.0/token HTTP/1.1” 200 5071 msal.token_cache: event={ “client_id”: "", “data”: { “claims”: null, “refresh_token”: "", “scope”: [ “openid”, "/.default", “profile”, “offline_access” ] }, “environment”: “login.microsoftonline.com”, “grant_type”: “refresh_token”, “params”: null, “response”: { “access_token”: “", “client_info”: "", “expires_in”: 5205, “ext_expires_in”: 5205, “foci”: “1”, “id_token”: "", “scope”: "”, “token_type”: “Bearer” }, “scope”: [ “", "” ], “skip_account_creation”: true, “token_endpoint”: “https://login.microsoftonline.com/<subscription_id>/oauth2/v2.0/token” } cli.azure.cli.core.sdk.policies: Request URL: ‘https://management.azure.com/subscriptions/*****/resourceGroups/<resource_group>/providers/Microsoft.ContainerService/managedClusters/<cluster_name>/runCommand?api-version=2022-03-01’ cli.azure.cli.core.sdk.policies: Request method: ‘POST’ cli.azure.cli.core.sdk.policies: Request headers: cli.azure.cli.core.sdk.policies: ‘Content-Type’: ‘application/json’ cli.azure.cli.core.sdk.policies: ‘Content-Length’: ‘2277’ cli.azure.cli.core.sdk.policies: ‘Accept’: ‘application/json’ cli.azure.cli.core.sdk.policies: ‘x-ms-client-request-id’: ‘’ cli.azure.cli.core.sdk.policies: ‘CommandName’: ‘aks command invoke’ cli.azure.cli.core.sdk.policies: ‘ParameterSetName’: ‘–resource-group --name --command --debug’ cli.azure.cli.core.sdk.policies: ‘User-Agent’: ‘AZURECLI/2.36.0 (DEB) azsdk-python-azure-mgmt-containerservice/19.0.0 Python/3.8.13 (Linux-5.13.0-1023-azure-x86_64-with-glibc2.29)’ cli.azure.cli.core.sdk.policies: ‘Authorization’: '’ cli.azure.cli.core.sdk.policies: Request body: cli.azure.cli.core.sdk.policies: {“command”: “kubectl get pods -A”, “context”: “”, “clusterToken”: “"} urllib3.connectionpool: https://management.azure.com:443 "POST /subscriptions//resourceGroups/<resource_group>/providers/Microsoft.ContainerService/managedClusters/<cluster_name>/runCommand?api-version=2022-03-01 HTTP/1.1” 500 213 cli.azure.cli.core.sdk.policies: Response status: 500 cli.azure.cli.core.sdk.policies: Response headers: cli.azure.cli.core.sdk.policies: ‘Cache-Control’: ‘no-cache’ cli.azure.cli.core.sdk.policies: ‘Pragma’: ‘no-cache’ cli.azure.cli.core.sdk.policies: ‘Content-Length’: ‘213’ cli.azure.cli.core.sdk.policies: ‘Content-Type’: ‘application/json’ cli.azure.cli.core.sdk.policies: ‘Expires’: ‘-1’ cli.azure.cli.core.sdk.policies: ‘x-ms-failure-cause’: ‘service’ cli.azure.cli.core.sdk.policies: ‘x-ms-correlation-request-id’: ‘’ cli.azure.cli.core.sdk.policies: ‘x-ms-request-id’: '’ cli.azure.cli.core.sdk.policies: ‘Strict-Transport-Security’: ‘max-age=31536000; includeSubDomains’ cli.azure.cli.core.sdk.policies: ‘Server’: ‘nginx’ cli.azure.cli.core.sdk.policies: ‘x-ms-ratelimit-remaining-subscription-writes’: ‘1199’ cli.azure.cli.core.sdk.policies: ‘x-ms-routing-request-id’: ‘*****’ cli.azure.cli.core.sdk.policies: ‘X-Content-Type-Options’: ‘nosniff’ cli.azure.cli.core.sdk.policies: ‘Date’: ‘Tue, 07 Jun 2022 08:14:11 GMT’ cli.azure.cli.core.sdk.policies: ‘Connection’: ‘close’ cli.azure.cli.core.sdk.policies: Response content: cli.azure.cli.core.sdk.policies: { “code”: “KubernetesOperationError”, “message”: "Failed to run command in managed cluster due to kubernetes failure. details: failed to execute on pod after retry: SPDYExecutor.SentAADToken retry failed: " } cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception: cli.azure.cli.core.util: Traceback (most recent call last): File “/opt/az/lib/python3.8/site-packages/knack/cli.py”, line 231, in invoke cmd_result = self.invocation.execute(args) File “/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/init.py”, line 658, in execute raise ex File “/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/init.py”, line 721, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) File “/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/init.py”, line 692, in _run_job result = cmd_copy(params) File “/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/init.py”, line 328, in call return self.handler(*args, **kwargs) File “/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/command_operation.py”, line 121, in handler return op(**command_args) File “/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/acs/custom.py”, line 2488, in aks_runcommand commandResultFuture = client.begin_run_command( File “/opt/az/lib/python3.8/site-packages/azure/core/tracing/decorator.py”, line 83, in wrapper_use_tracer return func(*args, **kwargs) File “/opt/az/lib/python3.8/site-packages/azure/mgmt/containerservice/v2022_03_01/operations/_managed_clusters_operations.py”, line 2447, in begin_run_command raw_result = self._run_command_initial( File “/opt/az/lib/python3.8/site-packages/azure/mgmt/containerservice/v2022_03_01/operations/_managed_clusters_operations.py”, line 2389, in _run_command_initial raise HttpResponseError(response=response, error_format=ARMErrorFormat) azure.core.exceptions.HttpResponseError: (KubernetesOperationError) Failed to run command in managed cluster due to kubernetes failure. details: failed to execute on pod after retry: SPDYExecutor.SentAADToken retry failed: Code: KubernetesOperationError Message: Failed to run command in managed cluster due to kubernetes failure. details: failed to execute on pod after retry: SPDYExecutor.SentAADToken retry failed:

cli.azure.cli.core.azclierror: (KubernetesOperationError) Failed to run command in managed cluster due to kubernetes failure. details: failed to execute on pod after retry: SPDYExecutor.SentAADToken retry failed: Code: KubernetesOperationError Message: Failed to run command in managed cluster due to kubernetes failure. details: failed to execute on pod after retry: SPDYExecutor.SentAADToken retry failed: az_command_data_logger: (KubernetesOperationError) Failed to run command in managed cluster due to kubernetes failure. details: failed to execute on pod after retry: SPDYExecutor.SentAADToken retry failed: Code: KubernetesOperationError Message: Failed to run command in managed cluster due to kubernetes failure. details: failed to execute on pod after retry: SPDYExecutor.SentAADToken retry failed: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f31a0c41820>] az_command_data_logger: exit code: 1 cli.main: Command ran in 12.628 seconds (init: 0.106, invoke: 12.522) telemetry.save: Save telemetry record of length 3460 in cache telemetry.check: Returns Positive. telemetry.main: Begin creating telemetry upload process. telemetry.process: Creating upload process: “/usr/bin/…/…/opt/az/bin/python3 /opt/az/lib/python3.8/site-packages/azure/cli/telemetry/init.py /home//.azure” telemetry.process: Return from creating process telemetry.main: Finish creating telemetry upload process.`