[BUG] CertificateClient.DownloadCertificate leads to SSL Authentication exception when server requires certificate and private key
See original GitHub issueLibrary name and version
Azure.Security.KeyVault.Certificates 4.2.0
Describe the bug
I’ve been facing the exception below into my web job deployed in Azure App Service for a while, it was really a strange exception that just happened a few times. It took me a while to track down the root cause, which I’ll be describing here.
Could not establish trust relationship for the SSL/TLS secure channel with authority 'host address'. The SSL connection could not be established, see inner exception. Authentication failed, see inner exception. An unknown error occurred while processing the certificate.
System.ServiceModel.Security.SecurityNegotiationException:
at Company.Project.RulesPlugins.Auto.Services.VehicleSpecifications.VehicleSpecificationRetriever.GetVehicle (Company.Project.RulesPlugins.Auto, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\RulesPlugins\Company.Project.RulesPlugins.Auto\Services\VehicleSpecifications\VehicleSpecificationRetriever.cs:47)
Inner exception System.Net.Http.HttpRequestException handled at Company.Project.RulesPlugins.Auto.Services.VehicleSpecifications.VehicleSpecificationRetriever.GetVehicle:
at System.Net.Http.ConnectHelper+<EstablishSslConnectionAsync>d__2.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<ConnectAsync>d__96.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__98.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<AddHttp11ConnectionAsync>d__73.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1+<WaitWithCancellationAsync>d__1.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<GetHttp11ConnectionAsync>d__75.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__83.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.DiagnosticsHandler+<SendAsyncCore>d__8.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.RedirectHandler+<SendAsync>d__4.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.DecompressionHandler+<SendAsync>d__16.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpClient+<<SendAsync>g__Core|83_0>d.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.ServiceModel.Channels.HttpChannelFactory`1+HttpClientRequestChannel+HttpClientChannelAsyncRequest+<SendRequestAsync>d__13.MoveNext (System.Private.ServiceModel, Version=4.9.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.Security.Authentication.AuthenticationException handled at System.Net.Http.ConnectHelper+<EstablishSslConnectionAsync>d__2.MoveNext:
at System.Net.Security.SslStream+<ForceAuthenticationAsync>d__173`1.MoveNext (System.Net.Security, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.ConnectHelper+<EstablishSslConnectionAsync>d__2.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.ComponentModel.Win32Exception handled at System.Net.Security.SslStream+<ForceAuthenticationAsync>d__173`1.MoveNext:
After a lot of debugging and research and many talks with the Azure support team, I managed to find the root cause of the exception, which you can see in the picture below from the network logs of my app service.
We can see that the exception that is throwing doesn’t give any information that the real issue it’s a bad certificate, this is already something to be addressed but not my focus on this issue.
Once I discovered that I back the focus on the code downloading the SSL certificate from KeyVault, which it’s basically the following method.
As we can see this method will attempt to download a certificate with a private key but in the case that the private key cannot be downloaded, it will return the certificate with the public key only. So, here it’s where I believe is the issue, if you have a scenerio like mine where you always need to send the certificate with the private key, you don’t have a way to force this method to throw an exception if fail to download the private key.
I believe some option should be included as a parameter or maybe this DownloadCertificate method should be split between one that downloads only with public and the other for the private key.
Just to finalize and provide more information, here it’s a comparison of the certificate being sent to the server from App Service and a Local Environment, as you can see the certificate size it’s different due to the lack of the private key into the App Service one.
Expected behavior
CertificateClient.DownloadCertificate method to expose a parameter to force the download only when containing private key or a new method to download the certificate with private key and in case of fail throw some exception.
Actual behavior
The CertificateClient.DownloadCertificate method always returns the certificate without a private key leading to an Authentication exception when trying to use the returned certificate without a private key into a service that requires the private key.
Reproduction Steps
1 - Add a certificate to the key vault which contains the private key 2 - Create a service that downloads the certificate and attach it to the request to a service that requires the certificate with private key 3 - Deploy to app service that has the Identity ID turned on and authorized to the key vault 4 - You should see any intermittent authorization exception when the app service fails to download the certificate with the private key
Environment
.NET SDK (reflecting any global.json): Version: 6.0.101 Commit: ef49f6213a
Runtime Environment: OS Name: Windows OS Version: 10.0.14393 OS Platform: Windows RID: win10-x86 Base Path: C:\Program Files (x86)\dotnet\sdk\6.0.101\
Host (useful for support): Version: 6.0.1 Commit: 3a25a7f1cc
Issue Analytics
- State:
- Created 2 years ago
- Comments:31 (13 by maintainers)
Top GitHub Comments
I’m having trouble finding the specific thread, but there was one about getting different hosts in App Services that might not allow the private key to be exported. It does so by temporarily saving to disk when constructing an
X509Certificate2
instance. There’s this one that is related: https://stackoverflow.com/questions/63884019/import-private-key-certificate-as-exportable-in-azure-app-serviceThat might explain the seeming randomness. Still, not only would caching the certificate help mitigate this, it should improve performance greatly not to make 2+ calls to Key Vault each time you need the certificate.
Thank you for your feedback. Tagging and routing to the team member best able to assist.