Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Provide a global timeout setting for IMDS probe

See original GitHub issue

Library or service name.

Azure.Identity
Azure.Services.AppAuthentication

Is your feature request related to a problem? Please describe.

We have a job cluster running inside windows containers on top of Azure VMSS. Each job run as a standalone executable. Right after each process launch, the executable authenticate to various azure services using DefaultAzureCredential and AppServiceTokenProvider.

Approximately 5% of jobs failed to authenticate due to IMDS connection timeout. Experiments show that it can sometimes take up to more than 6 seconds to connect to IMDS endpoint in our scenario.

We’ve created a patched version of Azure.Identity and Azure.Services.AppAuthentication to increase the IMDS connection timeout to 30 seconds and the fix solves most of IMDS connection problems.

The IMDS probe timeout is hardcoded as 1s for DefaultAzureCredential, and 3s for AzureServiceTokenProvider. Is it possible to make this timeout configurable? The configuration knob needs to be static because there are multiple components using DefaultAzureCredential and we’d like to configure the value in one place.

Issue Analytics

State:
Created 2 years ago
Comments:9 (5 by maintainers)

Top GitHub Comments

2reactions

christothescommented, Apr 23, 2021

Hi @yufeih - This is a common request and we are considering making some changes to the IMDS endpoint discovery, as described here

In the meantime, there is a workaround pod example which waits for the IMDS endpoint to become available at startup. You can find it here

0reactions

yufeihcommented, Nov 9, 2021

@christothes @schaabs

We are still hitting IMDS connection problems in a windows AKS cluster using the latest Azure.Identity package. It seems https://github.com/Azure/azure-sdk-for-net/pull/24328 added a hardcoded timeout of 1 second to the initial IMDS request, can that timeout be adjusted by an environment variable so we can tweak it to a larger value for apps running inside our cluster?

Top Results From Across the Web

context timeout 500ms for IMDS healthcheck is aggressive

The 500ms request timeout is aggressive. Based on inputs from @rkammara12 (from IMDS team), this timeout should be longer as it can generate ......

Set the Probe Transaction Timeout Value - TechDocs

To avoid the timeout, you can set the timeout to a greater value in the probe configuration file or through the Raw Configure...

Set the Probe Transaction Timeout Value - TechDocs

Open Admin Console and navigate to the device running. discovery_server · Click on the pulldown menu for discovery_server and select the. Raw Configure...

Terminated the probe because the max timeout was e...

Each probe type (WMI, SSH, SNMP, etc) has a default timeout setting so that a discovery job does not run forever if a...

Guidelines for Setting Authentication Server Timeouts

The global timeout is 30 seconds by default (range is 3 to 125). It must be the same as or greater than the...