[Azure Monitor] Expected errors are reported in Application Insights for Blob storage operations

Describe the bug I’m using the Azure.Messaging.EventHubs.Processor (5.0.1) with an EH with 32 partitions. Every partition gets checkpointed every 10 seconds (if there was new data arriving). Now I started to notice in Application Insights, that some of the checkpointing calls to Blob storage fail with a 412 error code.

Azure.RequestFailedException: The condition specified using HTTP conditional header(s) is not met.
RequestId:2130b973-701e-0130-5c14-e1c499000000
Time:2020-02-11T19:47:37.1509796Z
Status: 412 (The condition specified using HTTP conditional header(s) is not met.)

ErrorCode: ConditionNotMet

Headers:
Server: Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id: 2130b973-701e-0130-5c14-e1c499000000
x-ms-client-request-id: f01bd21c-fda0-429a-b88f-2eb41a153efb
x-ms-version: 2019-02-02
x-ms-error-code: ConditionNotMet
Date: Tue, 11 Feb 2020 19:47:36 GMT
Content-Length: 252
Content-Type: application/xml

   at Azure.Storage.Blobs.BlobRestClient.Blob.SetMetadataAsync_CreateResponse(Response response)
   at Azure.Storage.Blobs.BlobRestClient.Blob.SetMetadataAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, Nullable`1 timeout, IDictionary`2 metadata, String leaseId, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, Nullable`1 ifMatch, Nullable`1 ifNoneMatch, String requestId, Boolean async, String operationName, CancellationToken cancellationToken)

I can also see the errors as “Client Error” in my Blob storage metrics. Most of the calls seem to work fine, but some create the error. Looks something inside the SDK to me, not directly related to my code.

Expected behavior Should run without errors.

To Reproduce Hard to tell. The errors also come when I have almost zero load on the EH (only a handful messages). Happy to jump on a screen share if that helps.

Environment:

• Name and version of the Library package used: Azure.Messaging.EventHubs.Processor (5.0.1)
• Hosting platform or OS and .NET runtime version : dotnet 3.1 in Linux container running on AKS.

[jsquire]

There shouldn’t be anything needed from you at this point, @sebader. Actions are needed from the Azure Monitor team. The bot was reacting to tags, but I don’t believe those tags were accurate.

[sebader]

@jsquire thanks for looking into it and the thorough the explanation!

@lmolkova First of all I didn’t have any idea where the error came from. I just saw it popping up in my monitoring. From a user perspective, of course you get concerned if there are unexplained errors. A user does not know that those in this case represent “works-as-expected”. If I as a user see errors, which seem to be related to checkpointing, I get concerned that my checkpoints might not be properly written and thus I will run into issues. So I would say, no, they are not just noise. Without making it clear to the user (and I don’t really know what that could look like here), it raises concern.

I also would expect to see errors of underlying SDKs in my monitoring - if they do represent actual errors that I as the app owner need to take care of. When that’s not the case, I would expect the SDK to hide them, or at least clearly mark them as noise in the monitoring without me as the user looking through github, docs, etc. to find out whats going on and then manually build a filter. And yes, from the issue that @jsquire created, I understand that this might not be so easy to do.

Does this make sense from a user perspective?