Performance Impact due to HttpClient Socket Closure on `SignAsync`

We notice that our calls to SignAsync KeyVault operation averages around 800ms with some requests taking as long as 1.5s. We profiled your code focusing and found that the HTTP send request is the biggest bottleneck. In other words, this line: _httpResponse = await HttpClient.SendAsync(_httpRequest, cancellationToken).ConfigureAwait(false);

Digging through Wireshark logs, we find that the HTTPClient closes the underlying TCP socket in ~1.5s. So requests before this 1.5s window, get a response in ~90ms and requests after the TCP closure get response between 800ms and 1.5s. I am assuming that the TCP closure causes a new set of SSL handshake + some additional logic on the server that takes longer time(?)

What would be your recommendation to mitigate this bottleneck? Also, why is the server taking a longer time to respond after the TCP Socket close?

Note:

• TCP Socket closure might just be the observed side-effect of the real problem but the repros are consistent that way.
• This issue is not in reference to the first call after KeyVaultClient initialization, which is always longer because of no Bearer token caching(HttpBearerChallengeCache) and hence the longer route through KeyVaultCredential.PostAuthenticate.
• This prolonged server response is not observed with couple of other calls that we use - GetKeyAsync, GetSecretAsync.

[RandalliLama]

@marty2bell Please raise a ticket against the service. That will get it tracked by the correct people for a server side issue.

[marty2bell]

@herveyw

We are seeing the same behaviour talking to KeyVault…we haven’t went to the network level as far as @antonydeepak went, but we are definitely seeing outliers on our soak tests where decrypt calls to keyvault to taking over a minute to come back.

I was going to raise a ticket against the service, but could piggy back onto this issue ??

Cheers

Marty