[FEATURE REQ] Very difficult to know what went wrong when posts to Azure Monitor fail

Library name

Azure.Monitor.OpenTelemetry.Exporter

Please describe the feature.

Diagnosing why the OpenTelemetry SDK is not working is too hard and un-obvious to non-team members.

Currently errors in reporting are logged as EventSource events, and unbeknownst to me, in the Visual Studio output window. This needs to be made much more obvious to developers. I was running dotnet-monitor and having to do a post to configure the event collection as it’s not a standard source, and having to break in the debugger to enable the collection before some of the events are fired.

curl -X 'POST' \
  'https://localhost:52323/trace?name=loadtest&durationSeconds=30' \
  -H 'accept: application/octet-stream' \
  -H 'Content-Type: application/json' \
  -d '{
  "providers": [
    {"name": "OpenTelemetry-Sdk","eventLevel": "5"},
    {"name": "OpenTelemetry-AzureMonitor-Exporter","eventLevel": "5"},
    {"name": "Azure-Core","5": "Debug"}
  ],
  "requestRundown": true,
  "bufferSizeInMB": 1024
}'

When running ASP.NET Core Apps, typically I look to the console output to tell me what is going wrong. If Open Telemetry is failing, it should probably be using ILogger to write output to the console, and use appropriate levels so only error information is included by default (that can be changed through config).

Similarly, when I have problems with the deployed app, the first place to look is the console. Azure App-Service, Functions and Container Apps all have support for streaming the console output. So reporting to the console is the first place to look for explanations of issues.

I tried using OTEL_DIAGNOSTICS.json - and the file just contained Successfully opened file.

[TimothyMothra]

Multiple points to respond to 😃

1. If you mean the connectionstring was malformatted/unparseable, I’d expect AzMon exporter to throw and let app crash on startup.

   This is correct. If the connectionstring is malformed or if the InstrumentationKey is not present, the exporter will throw/crash with a detailed message.

2. If you mean your connectionstring was correct, but it had wrong ikey or wrong endpoint, then I’d expect AzMon Exporter to log the fact that it received failure (400)

   This is correct, we log these to Event Source which is also visible in Visual Studio’s Debug Output:

   The ingestion sometimes responds with a detailed error message. We should double check that we’re logging those messages!

3. and optionally return “ExportFailed” back to the OTel SDK (which would log it).

   This is correct. Our Transmitter will return ExportResult.Failed. Regarding OTel’s Self Diagnostic behavior, I confirmed this is capturing logs from our EventSource.

4. When running ASP.NET Core Apps, typically I look to the console output to tell me what is going wrong. If Open Telemetry is failing, it should probably be using ILogger to write output to the console, and use appropriate levels so only error information is included by default (that can be changed through config).

   This is a good idea but is a major change from what we and the community are doing today. I’ve seen similar asks in the https://github.com/open-telemetry/opentelemetry-dotnet repo but I haven’t seen any consensus to move away from EventSource. Perhaps we can pilot this as a feature in the AzureMonitorExporter and share learnings with the community.

[cijothomas]

e.g. it’ll be even better if the internal logs are exposed via ILogger so we have the flexibility to write to file/console/etc.

Yes, this is the tracking issue for that : https://github.com/open-telemetry/opentelemetry-dotnet/issues/3881 (surprisingly not many upvotes/comments there!)