[BUG] SetSecretAsync sometimes fails due to ObjectIsBeingRecovered after StartRecoverDeletedSecretAsync completed.

Library name and version

Azure.Keyvault.Secrets 4.4.0

Describe the bug

I recovered a deleted secret (called testSecret) using SecretClient.StartRecoverDeletedSecretAsync() and waited for the recovery process to finish using WaitForCompletionAsync. When I (immediately) try to set a new secret value using SecretClient.SetSecretAsync() I sometimes get ‘Azure.RequestFailedException’ with a 409 “Secret testSecret is currently being recovered and cannot be re-created; retry later”

Expected behavior

When StartRecoverDeletedSecretAsync() has finished succesfully, I am always able to set a new value for that secret.

Actual behavior

Sometimes SetSecretAsync() returns a 409 with message “Secret testSecret is currently being recovered and cannot be re-created; retry later.” with ‘innererror’: “ObjectIsBeingRecovered”.

For testing purposes, I have been testing the same piece of code (delete secret, recover secret, and set secret value) in a loop. Sometimes it runs without issue, sometimes I get a 409 when setting the secret value. I’ve implemented a Polly retry mechanism as well to retry SetSecretAsyn() upon getting Azure.RequestFailedException, and often it succeeds after 1 or multiple retries.

Reproduction Steps

• Secret “testSecret” in KeyVault in ‘deleted’ state and is ‘recoverable’

string secretName = "testSecret"
string secretValue = "testSecretValue"
bool Success = true;

// Recover deleted secret
RecoverDeletedSecretOperation operation = await _keyVaultSecretClient.StartRecoverDeletedSecretAsync(secretName);
await operation.WaitForCompletionAsync();
Success &= operation.HasCompleted && operation.Value.Enabled == true;
if (!isSuccess){
  return
}

// Set new secret value
KeyVaultSecret newSecretValue = new KeyVaultSecret(secretName, secretValue );
// Error occurs here
Azure.Response<KeyVaultSecret> response = await _keyVaultSecretClient.SetSecretAsync(newSecretValue);
KeyVaultSecret storedSecretValue = response?.Value;
isSuccess = storedSecretValue?.Value == newSecretValue.Value;

// Delete secret again (for next test iteration)
DeleteSecretOperation operation = await _keyVaultSecretClient.StartDeleteSecretAsync(secretName);
await operation.WaitForCompletionAsync();

Environment

• .NET core 3.1
• IDE Visual Studio 17.3
• Keyvault with soft delete enabled

[adarce]

@heaths : After taking a look, I believe that this is a bug in the SDK and not the service. After starting the recovery, the SDK’s WaitForCompletionAsync function polls a Get Object API every 2 seconds until the response changes from 404 NotFound to 200 OK. However, it’s polling the versioned object (e.g.: https://my-vault.vault.azure.net/secrets/my-secret/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa), instead of the current object (e.g.: https://my-vault.vault.azure.net/secrets/my-secret). This is problematic because internally when Azure Key Vault recovers an object, it recovers each versioned blob one-by-one first, then recovers the current blob as the last step. WaitForCompletionAsync should always poll the soft-deletion state of an object using its unversioned URI, and never its versioned URI.

Sample link to the change I’m talking about: https://github.com/Azure/azure-sdk-for-net/blob/71054d50e35c7ebf5eee0a185c48dab702f6fb82/sdk/keyvault/Azure.Security.KeyVault.Secrets/src/RecoverDeletedSecretOperation.cs#L84-L88

Instead of doing this:

await _pipeline.GetResponseAsync(
    RequestMethod.Get,
    cancellationToken,
    SecretClient.SecretsPath,
    _value.Name,
    "/",
    _value.Version).ConfigureAwait(false)

The SDK should do this:

await _pipeline.GetResponseAsync(
    RequestMethod.Get,
    cancellationToken,
    SecretClient.SecretsPath,
    _value.Name).ConfigureAwait(false)

[heaths]

Thanks for the investigation, @adarce. @vcolin7 this is probably a problem in all our SDKs we should address in our upcoming beta.

@oxanaoosterlee is a fix in a beta SDK sufficient to resolve this issue?