question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

KeyVault getSecret() hangs when run inside kubernetes

See original GitHub issue

Describe the bug

In versions of Azure-core 1.2x, and 1.3x there seems to be a bug that afflicts the azure-security-keyvault-secrets package while running in Kubernetes. The bug is that while running a java spring application in a Kubernetespod, the getSecret call just hangs and the program locks up and never recovers. I noticed the pod is getting CPU throttled but there is very little diagnostic information available to try and troubleshoot. There is another github thread found below which provide more detail and some suggestions (None of which work) - See additional context for link to thread. The solution for me ended up being downgrading azure-core to 1.1x

Can you please look at correcting this? I am forced to run old version of the Azure-core, Cosmos, and keyvault libraries which are missing a lot of great enhancements.

Exception or Stack Trace No stack trace, the application just hangs.

To Reproduce Run a java application in k8s with java 8, spring, and Azure-core 1.2x and 1.3x, then try to call secretClient.getSecret(secret).getValue()

Code Snippet LOGGER.info(“Initializing Keyvault Client…”);

secretClient = new SecretClientBuilder()
		.vaultUrl(properties.getKeyVaultUri())
		.addPolicy(new TimeoutPolicy(Duration.ofSeconds(100)))
		.credential(
				new ClientSecretCredentialBuilder()
						.clientSecret(properties.getKeyVaultClientSecret())
						.clientId(properties.getKeyVaultClientId())
						.tenantId(properties.getKeyVaultTenantid())
						.build())
		.clientOptions(httpClientOptions)
		.buildClient();

secret = secretClient.getSecret(secret).getValue();

Expected behavior We can retrieve the key without the code freezing.

Setup (please complete the following information):

  • OS: Linux
  • IDE: N/A
  • Library/Libraries: Azure-core 1.2x or 1.3x
  • Java version: 8
  • App Server/Environment: Kubernetes
  • Frameworks: Spring boot

Additional context Heres another thread with more context, the issue was created for the keyvault client but I think that is not where the issue exists given that keyvault version changes didnt fix the issue but Azure-core package version downgrade did. https://github.com/Azure/azure-sdk-for-java/issues/28364

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
joshfreecommented, Aug 17, 2022

Hi @bennydean thank you for reaching out to us via this GitHub issue. @vcolin7 from the KeyVault SDK team will follow up with you shortly.

1reaction
vcolin7commented, Aug 20, 2022

@bennydean, I think @anuchandy and I have a lead to where this might be happening in the stack and I will be looking into this next week.

Read more comments on GitHub >

github_iconTop Results From Across the Web

[BUG] Call to key vault may hangs forever #28364 - GitHub
I use azure-security-keyvault-secrets v.4.3.6 on java17 application to ... KeyVault getSecret() hangs when run inside kubernetes #30466.
Read more >
Troubleshoot Azure Key Vault Provider for Secrets Store CSI ...
The Kubernetes kubectl tool. To install kubectl by using Azure CLI, run the az aks install-cli command. The client URL (curl) tool.
Read more >
KeyVault GetSecretAsync never returns - Stack Overflow
var secret = KeyVaultAccessor.GetSecret("https://superSecretUri").Result;. Unfortunately, the call never returns and the query hangs indefintely ...
Read more >
Mount Vault Secrets through Container Storage Interface (CSI ...
Kubernetes application pods that rely on Vault to manage their secrets can retrieve them ... These clusters are run locally inside Virtual Machines...
Read more >
Add an Azure Key Vault Secrets Manager - Harness.io Docs
In the Configure Secrets Manager dialog's Display Name field, enter an arbitrary name to identify this key vault within Harness Manager.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found