question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] CertificateClient.DownloadCertificate leads to SSL Authentication exception when server requires certificate and private key

See original GitHub issue

Library name and version

Azure.Security.KeyVault.Certificates 4.2.0

Describe the bug

I’ve been facing the exception below into my web job deployed in Azure App Service for a while, it was really a strange exception that just happened a few times. It took me a while to track down the root cause, which I’ll be describing here.

Could not establish trust relationship for the SSL/TLS secure channel with authority 'host address'. The SSL connection could not be established, see inner exception. Authentication failed, see inner exception. An unknown error occurred while processing the certificate.

System.ServiceModel.Security.SecurityNegotiationException:
   at Company.Project.RulesPlugins.Auto.Services.VehicleSpecifications.VehicleSpecificationRetriever.GetVehicle (Company.Project.RulesPlugins.Auto, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\RulesPlugins\Company.Project.RulesPlugins.Auto\Services\VehicleSpecifications\VehicleSpecificationRetriever.cs:47)
Inner exception System.Net.Http.HttpRequestException handled at Company.Project.RulesPlugins.Auto.Services.VehicleSpecifications.VehicleSpecificationRetriever.GetVehicle:
   at System.Net.Http.ConnectHelper+<EstablishSslConnectionAsync>d__2.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<ConnectAsync>d__96.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__98.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<AddHttp11ConnectionAsync>d__73.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1+<WaitWithCancellationAsync>d__1.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<GetHttp11ConnectionAsync>d__75.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__83.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.DiagnosticsHandler+<SendAsyncCore>d__8.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.DecompressionHandler+<SendAsync>d__16.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpClient+<<SendAsync>g__Core|83_0>d.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.ServiceModel.Channels.HttpChannelFactory`1+HttpClientRequestChannel+HttpClientChannelAsyncRequest+<SendRequestAsync>d__13.MoveNext (System.Private.ServiceModel, Version=4.9.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.Security.Authentication.AuthenticationException handled at System.Net.Http.ConnectHelper+<EstablishSslConnectionAsync>d__2.MoveNext:
   at System.Net.Security.SslStream+<ForceAuthenticationAsync>d__173`1.MoveNext (System.Net.Security, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.ConnectHelper+<EstablishSslConnectionAsync>d__2.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.ComponentModel.Win32Exception handled at System.Net.Security.SslStream+<ForceAuthenticationAsync>d__173`1.MoveNext:

After a lot of debugging and research and many talks with the Azure support team, I managed to find the root cause of the exception, which you can see in the picture below from the network logs of my app service. image

We can see that the exception that is throwing doesn’t give any information that the real issue it’s a bad certificate, this is already something to be addressed but not my focus on this issue.

Once I discovered that I back the focus on the code downloading the SSL certificate from KeyVault, which it’s basically the following method.

https://github.com/Azure/azure-sdk-for-net/blob/10461aa95a53bcb6dbcb695f494c7a62c0e399d8/sdk/keyvault/Azure.Security.KeyVault.Certificates/src/CertificateClient.cs#L207-L250

As we can see this method will attempt to download a certificate with a private key but in the case that the private key cannot be downloaded, it will return the certificate with the public key only. So, here it’s where I believe is the issue, if you have a scenerio like mine where you always need to send the certificate with the private key, you don’t have a way to force this method to throw an exception if fail to download the private key.

I believe some option should be included as a parameter or maybe this DownloadCertificate method should be split between one that downloads only with public and the other for the private key.

Just to finalize and provide more information, here it’s a comparison of the certificate being sent to the server from App Service and a Local Environment, as you can see the certificate size it’s different due to the lack of the private key into the App Service one. image

Expected behavior

CertificateClient.DownloadCertificate method to expose a parameter to force the download only when containing private key or a new method to download the certificate with private key and in case of fail throw some exception.

Actual behavior

The CertificateClient.DownloadCertificate method always returns the certificate without a private key leading to an Authentication exception when trying to use the returned certificate without a private key into a service that requires the private key.

Reproduction Steps

1 - Add a certificate to the key vault which contains the private key 2 - Create a service that downloads the certificate and attach it to the request to a service that requires the certificate with private key 3 - Deploy to app service that has the Identity ID turned on and authorized to the key vault 4 - You should see any intermittent authorization exception when the app service fails to download the certificate with the private key

Environment

.NET SDK (reflecting any global.json): Version: 6.0.101 Commit: ef49f6213a

Runtime Environment: OS Name: Windows OS Version: 10.0.14393 OS Platform: Windows RID: win10-x86 Base Path: C:\Program Files (x86)\dotnet\sdk\6.0.101\

Host (useful for support): Version: 6.0.1 Commit: 3a25a7f1cc

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:31 (13 by maintainers)

github_iconTop GitHub Comments

1reaction
heathscommented, Mar 4, 2022

I’m having trouble finding the specific thread, but there was one about getting different hosts in App Services that might not allow the private key to be exported. It does so by temporarily saving to disk when constructing an X509Certificate2 instance. There’s this one that is related: https://stackoverflow.com/questions/63884019/import-private-key-certificate-as-exportable-in-azure-app-service

That might explain the seeming randomness. Still, not only would caching the certificate help mitigate this, it should improve performance greatly not to make 2+ calls to Key Vault each time you need the certificate.

1reaction
jsquirecommented, Mar 4, 2022

Thank you for your feedback. Tagging and routing to the team member best able to assist.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Azure Key Vault Certificates does not have the Private ...
Because Cer contains only the public key, this method attempts to download the managed secret that contains the full certificate.
Read more >
Troubleshooting SSL related issues (Server Certificate)
The error message is shown because the SSL handshake failed. ... Check if the server certificate has the private key corresponding to it....
Read more >
Untitled
Certificateclient.downloadcertificate .net 6.0 - Troubleshoot ... returns the certificate without a private key leading to an Authentication exception when ...
Read more >
Generate and configure an SSL certificate for backend ...
Learn how to enable backend SSL authentication of an API using the API Gateway console.
Read more >
Rehash: How to Fix the SSL/TLS Handshake Failed Error
The TLS Handshake Failed error can originate from the client or the server, here's a guide for fixing the problem for both users...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found