[BUG] CryptographicException: Padding is invalid and cannot be removed.

Library name and version

Azure.Storage.Blobs 12.10.0

Describe the bug

An exception is thrown when doing multiple UploadAsync and DownloadAsync in parallel with ClientSideEncryption enabled.

Exception: System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed when doing

Background information: We are migrating from Microsoft.Azure.Storage.Blob to Azure.Storage.Blobs. While doing a stress test on our service, we started seeing these exceptions. I’ve isolated the code to a single Unit Test which should be helpful to reproduce the issue. I’ve also tried several variants resulting in the same exception:

• New BlobClient per request
• Using streams for uploading and downloading

Expected behavior

No exception.

Actual behavior

A System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed when doing is thrown.

Reproduction Steps

[TestMethod]
public async Task TestUploadAsyncScenarios2()
{
    var keyVaultUri = new Uri("https://***.vault.azure.net");
    var keyVaultSecretName = "***";
    var keyVaultTenantId = "***";
    var keyVaultClientId = "***";
    var certificateThumbprint = "***";

    var x509Store = new X509Store(StoreName.My, StoreLocation.CurrentUser);
    x509Store.Open(OpenFlags.ReadOnly);
    var certificates = x509Store.Certificates.Find(X509FindType.FindByThumbprint, certificateThumbprint, false);
    var certificate = certificates[0];
    x509Store.Close();

    var credential = new ClientCertificateCredential(keyVaultTenantId, keyVaultClientId, certificate);
    var secretClient = new SecretClient(keyVaultUri, credential);
    var secret = secretClient.GetSecret(keyVaultSecretName);

    var keyResolver = new KeyResolver(credential);
    var keyEncryptionKey = keyResolver.Resolve(secret.Value.Id);

    var clientSideEncryptionOptions = new ClientSideEncryptionOptions(ClientSideEncryptionVersion.V1_0)
    {
        KeyEncryptionKey = keyEncryptionKey,
        KeyResolver = keyResolver,
        KeyWrapAlgorithm = "A256KW"
    };

    var blobConnectionString = "***";
    var blobContainerName = "***";
    var blobName = "test";
    var blobContents = "test content";

    var blobContainerClient = new BlobContainerClient(blobConnectionString, blobContainerName);
    var blobClient = blobContainerClient.GetBlobClient(blobName);
    var encryptedBlobClient = blobClient.WithClientSideEncryptionOptions(clientSideEncryptionOptions);

    await UploadAsync();

    while (true)
    {
        try
        {
            await Task.WhenAll(
                UploadAsync(),
                DownloadAsync(),
                UploadAsync()
            );
        }
        catch (Exception)
        {
            throw;
        }
    }

    Task UploadAsync() => encryptedBlobClient.UploadAsync(new BinaryData(blobContents), new BlobUploadOptions { });
    Task DownloadAsync() => encryptedBlobClient.DownloadContentAsync();
}

Environment

Windows 10 .NET Standard 2.0 Visual Studio 16.10.3

[bsumter]

@bsumter , Sorry to get back and forth. I just wanted to repro this issue with the same code as yours. The X509StoreAdapter and X509Certificate2Retriever arent a part of the System.Security.Cryptography.X509Certificates namespace. I have checked it for .net standard 2.0 and latest .net core versions (.net 3.1,5,6)

https://docs.microsoft.com/en-us/dotnet/api/system.security.cryptography.x509certificates?view=netstandard-2.0

Please point out if Im missing out something

My Bad. I’ve updated the initial post. Should be using System.Security.Cryptography.X509Certificates now.

[SaurabhSharma-MSFT]

@bsumter We are looking into this and get back to you in case need any information.