question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. ItΒ collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Logging does not show root cause

See original GitHub issue

I am having problems finding the cause of printed errors. They provide information about what the results were, but not what caused it, or what paths were attempted.

In the case of AzureMonitor, when I had invalid role permissions, I was getting

fail: Promitor.Integrations.AzureMonitor.Logging.AzureMonitorIntegrationLogger[0]              
      Exception in 5: Microsoft.Azure.Management.Monitor.Fluent.Models.ErrorResponseException: Operation returned an invalid status code 'Forbidden'

This tells me that some url probably received an HTTP 403, but that’s all I know. I think the URL, status code and the actual error message from the API would of interest here.

When I had configured the proper roles, I was getting the following error

fail: Promitor.Integrations.AzureMonitor.Logging.AzureMonitorIntegrationLogger[0]
      Exception in 5: Microsoft.Azure.Management.Monitor.Fluent.Models.ErrorResponseException: Operation returned an invalid status code 'NotFound'

I don’t know what isn’t found. More verbosity in the logs would be nice. Or am I missing a flag?

Configuration

Config

server:
  httpPort: 8080
prometheus:
  # metricUnavailableValue: NaN
  # enableMetricTimestamps: true
  scrapeEndpoint:
    baseUriPath: /metrics
metricsConfiguration:
  absolutePath: /config/metrics-declaration.yaml
telemetry:
  containerLogs:
    isEnabled: false
  defaultVerbosity: debug

azureMonitor:
  logging:
    informationLevel: BodyAndHeaders
    isEnabled: true

Metrics

version: v1
azureMetadata:
  tenantId: xxx
  subscriptionId: xxx
  resourceGroupName: promitor
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    # Every minute
    schedule: "0 * * ? * *"

metrics:
- name: promitor_demo_azuresqlmanagedinstance_nodes
  description: "The amount of nodes for an Azure SQL Managed Instance."
  resourceType: SqlManagedInstance
  azureMetricConfiguration:
    metricName: virtual_core_count
    aggregation:
      type: Average
  resources:
  - instanceName: my-special-postgres-instance

Specifications

  • Version: 1.4.0
  • Platform: Docker on Linux
  • Subsystem:

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:9 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
stigokcommented, Mar 26, 2020

Thanks for the response. Yeah, it appeared to be an error caused by myself. Sorry about that. Feel free to close this issue now. I don’t expect you to do anything about this feedback, but I’ll just post my thoughts as I’m working with it.

Azure Monitor logging

I’m still not sure what the azureMonitor.logging.informationLevel: BodyAndHeaders does. Is this for the requests towards the Azure Monitor API?

API requests logging

For requests that are being made to the Azure Monitor API, like here, where I’m getting null values, if I could enable verbose logging on the HTTP requests, e.g. see the URL being queried, I could go straight into the API and look around for myself.

β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•— β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•—  β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•— β–ˆβ–ˆβ–ˆβ•—   β–ˆβ–ˆβ–ˆβ•—β–ˆβ–ˆβ•—β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•— β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•— β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•—
β–ˆβ–ˆβ•”β•β•β–ˆβ–ˆβ•—β–ˆβ–ˆβ•”β•β•β–ˆβ–ˆβ•—β–ˆβ–ˆβ•”β•β•β•β–ˆβ–ˆβ•—β–ˆβ–ˆβ–ˆβ–ˆβ•— β–ˆβ–ˆβ–ˆβ–ˆβ•‘β–ˆβ–ˆβ•‘β•šβ•β•β–ˆβ–ˆβ•”β•β•β•β–ˆβ–ˆβ•”β•β•β•β–ˆβ–ˆβ•—β–ˆβ–ˆβ•”β•β•β–ˆβ–ˆβ•—
β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•”β•β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•”β•β–ˆβ–ˆβ•‘   β–ˆβ–ˆβ•‘β–ˆβ–ˆβ•”β–ˆβ–ˆβ–ˆβ–ˆβ•”β–ˆβ–ˆβ•‘β–ˆβ–ˆβ•‘   β–ˆβ–ˆβ•‘   β–ˆβ–ˆβ•‘   β–ˆβ–ˆβ•‘β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•”β•
β–ˆβ–ˆβ•”β•β•β•β• β–ˆβ–ˆβ•”β•β•β–ˆβ–ˆβ•—β–ˆβ–ˆβ•‘   β–ˆβ–ˆβ•‘β–ˆβ–ˆβ•‘β•šβ–ˆβ–ˆβ•”β•β–ˆβ–ˆβ•‘β–ˆβ–ˆβ•‘   β–ˆβ–ˆβ•‘   β–ˆβ–ˆβ•‘   β–ˆβ–ˆβ•‘β–ˆβ–ˆβ•”β•β•β–ˆβ–ˆβ•—
β–ˆβ–ˆβ•‘     β–ˆβ–ˆβ•‘  β–ˆβ–ˆβ•‘β•šβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•”β•β–ˆβ–ˆβ•‘ β•šβ•β• β–ˆβ–ˆβ•‘β–ˆβ–ˆβ•‘   β–ˆβ–ˆβ•‘   β•šβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ•”β•β–ˆβ–ˆβ•‘  β–ˆβ–ˆβ•‘
β•šβ•β•     β•šβ•β•  β•šβ•β• β•šβ•β•β•β•β•β• β•šβ•β•     β•šβ•β•β•šβ•β•   β•šβ•β•    β•šβ•β•β•β•β•β• β•šβ•β•  β•šβ•β•
[09:31:56 DBG] Hosting starting
[09:31:56 INF] Task MetricScrapingJob is running under local time zone
[09:31:56 INF] Task MetricScrapingJob is running under local time zone
[09:31:56 INF] Starting validation of Promitor setup
[09:31:56 INF] Start Validation step 1/3: Metrics Declaration Path
[09:31:56 INF] Scrape configuration found at '/config/metrics-declaration.yaml'
[09:31:56 INF] Validation step 1/3 succeeded
[09:31:56 INF] Start Validation step 2/3: Azure Authentication
[09:31:56 INF] Validation step 2/3 succeeded
[09:31:56 INF] Start Validation step 3/3: Metrics Declaration
[09:31:56 INF] Metrics declaration is using spec version v1
[09:31:56 INF] Following metrics configuration was configured:
azureMetadata:
  resourceGroupName: my-rg
  subscriptionId: xx
  tenantId: xx
  cloud:
    name: AzureGlobalCloud
    authenticationEndpoint: https://login.microsoftonline.com/
    resourceManagerEndpoint: https://management.azure.com/
    graphEndpoint: https://graph.windows.net/
    managementEndpoint: https://management.core.windows.net/
    storageEndpointSuffix: core.windows.net
    keyVaultSuffix: vault.azure.net
metricDefaults:
  aggregation:
    interval: 00:01:00
  scraping: &o0
    schedule: 0 * * ? * *
metrics:
- azureMetricConfiguration:
    metricName: cpu_percent
    aggregation:
      type: Average
      interval: 00:01:00
  prometheusMetricDefinition:
    name: azure_postgre_sql_cpu_percent
    description: Server CPU usage percentage
    labels: {}
  scraping: *o0
  resourceType: PostgreSql
  resources:
  - serverName: myinstance
    resourceType: PostgreSql
- azureMetricConfiguration:
    metricName: memory_percent
    aggregation:
      type: Average
      interval: 00:01:00
  prometheusMetricDefinition:
    name: azure_postgre_sql_memory_percent
    description: Server memory usage percentage
    labels: {}
  scraping: *o0
  resourceType: PostgreSql
  resources:
  - serverName: myinstance
    resourceType: PostgreSql

[09:31:56 INF] Validation step 3/3 succeeded
[09:31:56 INF] Promitor configuration is valid, we are good to go.
[09:31:56 DBG] Failed to locate the development https certificate at 'null'.
[09:31:56 INF] Now listening on: http://[::]:8080
[09:31:56 DBG] Loaded hosting startup assembly Promitor.Scraper.Host
[09:31:56 INF] Application started. Press Ctrl+C to shut down.
[09:31:56 INF] Hosting environment: Production
[09:31:56 INF] Content root path: /app
[09:31:56 DBG] Hosting started
[09:31:59 INF] Found value 14.49 for metric azure_postgre_sql_memory_percent with aggregation interval 00:01:00
[09:31:59 INF] Found value 20.4 for metric azure_postgre_sql_cpu_percent with aggregation interval 00:01:00
[09:32:03 INF] Found value null for metric azure_postgre_sql_memory_percent with aggregation interval 00:01:00
[09:32:03 INF] Found value null for metric azure_postgre_sql_cpu_percent with aggregation interval 00:01:00
[09:32:03 DBG] Connection id "0HLUHD25HHKSP" accepted.
[09:32:03 DBG] Connection id "0HLUHD25HHKSP" started.
[09:32:03 INF] Request starting HTTP/1.1 GET http://localhost:8080/metrics
[09:32:03 DBG] Wildcard detected, all requests with hosts will be allowed.
[09:32:03 DBG] Connection id "0HLUHD25HHKSP" completed keep alive response.
[09:32:03 INF] Request finished in 20.1609ms 200 text/plain; version=0.0.4
[09:32:04 INF] Request starting HTTP/1.1 GET http://localhost:8080/metrics
[09:32:04 DBG] Connection id "0HLUHD25HHKSP" completed keep alive response.
[09:32:04 INF] Request finished in 0.7392ms 200 text/plain; version=0.0.4
[09:32:04 INF] Request starting HTTP/1.1 GET http://localhost:8080/metrics
[09:32:04 DBG] Connection id "0HLUHD25HHKSP" completed keep alive response.
[09:32:04 INF] Request finished in 0.2943ms 200 text/plain; version=0.0.4
[09:32:05 DBG] Connection id "0HLUHD25HHKSP" received FIN.
[09:32:05 DBG] Connection id "0HLUHD25HHKSP" sending FIN because: "The client closed the connection."
[09:32:05 DBG] Connection id "0HLUHD25HHKSP" disconnecting.
[09:32:05 DBG] Connection id "0HLUHD25HHKSP" stopped.
[09:32:05 DBG] Connection id "0HLUHD25HHKSQ" accepted.
[09:32:05 DBG] Connection id "0HLUHD25HHKSQ" started.
[09:32:05 INF] Request starting HTTP/1.1 GET http://localhost:8080/metrics
[09:32:05 DBG] Connection id "0HLUHD25HHKSQ" completed keep alive response.
[09:32:05 INF] Request finished in 0.3555ms 200 text/plain; version=0.0.4
[09:32:05 INF] Request starting HTTP/1.1 GET http://localhost:8080/favicon.ico
[09:32:05 DBG] The request path  does not match the path filter
[09:32:05 DBG] The request path  does not match the path filter
[09:32:05 DBG] No candidates found for the request path '/favicon.ico'
[09:32:05 DBG] Request did not match any endpoints
[09:32:05 INF] HTTP GET /favicon.ico responded 404 in 12.4792 ms
[09:32:05 DBG] Connection id "0HLUHD25HHKSQ" completed keep alive response.
[09:32:05 INF] Request finished in 30.1635ms 404
^C[09:32:40 INF] Application is shutting down...
[09:32:40 DBG] Hosting stopping
[09:32:40 DBG] Connection id "0HLUHD25HHKSQ" disconnecting.
[09:32:40 DBG] Connection id "0HLUHD25HHKSQ" stopped.
[09:32:40 DBG] Connection id "0HLUHD25HHKSQ" sending FIN because: "The Socket transport's send loop completed gracefully."
[09:32:40 DBG] Hosting stopped

In this particular instance, it appeared to be that intervals < 5 minutes only occationally returned data.

[09:31:59 INF] Found value 14.49 for metric azure_postgre_sql_memory_percent with aggregation interval 00:01:00
[09:31:59 INF] Found value 20.4 for metric azure_postgre_sql_cpu_percent with aggregation interval 00:01:00
[09:32:03 INF] Found value null for metric azure_postgre_sql_memory_percent with aggregation interval 00:01:00
[09:32:03 INF] Found value null for metric azure_postgre_sql_cpu_percent with aggregation interval 00:01:00

After setting aggregation interval to 5 minutes, both values returned data for all requests. I suspect this isn’t an isse with Promitor, but rather the upstream API restrictions/limitations (?).

0reactions
tomkerkhovecommented, Mar 27, 2020

Happy to help and thanks for trying/using Promitor.

Feel free to let me know if you want to be listed on https://promitor.io/#Customers

Read more comments on GitHub >

github_iconTop Results From Across the Web

python logging root logger does not show info even if I set ...
Okay, but you found out the problem: when logging level set to DEBUG , the root logger is not working as expected. Debug...
Read more >
Logging HOWTO β€” Python 3.11.4 documentation
Basic Logging Tutorial: Logging is a means of tracking events that happen when ... Don't worry about the 'root' part for now: it...
Read more >
Logging exceptions root cause first - Tomasz Nurkiewicz
Not very intuitive, don't you think? And what is this red ellipse showing? Yes, it is the root cause of the failure (the...
Read more >
Zebrium: Root Cause as a Service | ML for Logs
Root Cause as a Service: automatically see the root cause of any problem with 95.8% accuracy. Works with your monitoring and other tools....
Read more >
Automation for Jira: How to find the Root Cause ...
The root cause for why an Error might appear in the Audit Log can vary greatly, but in this case, we'll use a...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found