Reported values from filtered multi-dimensional metric are all the same

Report

As requested in https://github.com/tomkerkhove/promitor/discussions/1591

I tried to filter a multi-dimensional metric from Azure Application Gateway (see configuration below). Promitor returns the metric names as expected, but the values are the same for all instances (_1xx, _2xx, _3xx, _4xx, _5xx).

Unfortunately there seems to be another issue with the logging. As soon as I add the metrics configuration there are no Azure monitor logs anymore. Therefore I currently can’t really figure out what the exact problem is.

fwiw: For my use-case it’s currently enough to evaluate the “Total Requests” and “Failed Requests” metrics by backendpool. There is no filtering involved and it’s possible to also use service discovery (I really like this feature! 😃).

Expected Behavior

Correct metric values for every instance.

Actual Behavior

Metric values for … azure_application_gateway_backend_response_status_1xx azure_application_gateway_backend_response_status_2xx azure_application_gateway_backend_response_status_3xx azure_application_gateway_backend_response_status_4xx azure_application_gateway_backend_response_status_5xx … are all the same.

Steps to Reproduce the Problem

1. add the scrape configuration
2. restart Promitor
3. let Prometheus scrape (or just call the metrics endpoint)
4. compare metric values …

Component

Scraper

Version

2.1.0 (Helm Chart 2.2.1)

Configuration

Configuration:

metrics:
  - name: azure_application_gateway_backend_response_status_1xx
    description: "Http responses of Azure Application Gateway by backend and status code"
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BackendResponseStatus
      aggregation:
        type: Total
      dimension:
        name: BackendPool
    resources:
      - resourceUri: Microsoft.Network/applicationGateways/my-appgw1234
        resourceGroupName: my-rg
        filter: HttpStatusGroup eq '1xx'
  - name: azure_application_gateway_backend_response_status_2xx
    description: "Http responses of Azure Application Gateway by backend and status code"
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BackendResponseStatus
      aggregation:
        type: Total
      dimension:
        name: BackendPool
    resources:
      - resourceUri: Microsoft.Network/applicationGateways/my-appgw1234
        resourceGroupName: my-rg
        filter: HttpStatusGroup eq '2xx'
  - name: azure_application_gateway_backend_response_status_3xx
    description: "Http responses of Azure Application Gateway by backend and status code"
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BackendResponseStatus
      aggregation:
        type: Total
      dimension:
        name: BackendPool
    resources:
      - resourceUri: Microsoft.Network/applicationGateways/my-appgw1234
        resourceGroupName: my-rg
        filter: HttpStatusGroup eq '3xx'
  - name: azure_application_gateway_backend_response_status_4xx
    description: "Http responses of Azure Application Gateway by backend and status code"
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BackendResponseStatus
      aggregation:
        type: Total
      dimension:
        name: BackendPool
    resources:
      - resourceUri: Microsoft.Network/applicationGateways/my-appgw1234
        resourceGroupName: my-rg
        filter: HttpStatusGroup eq '4xx'
  - name: azure_application_gateway_backend_response_status_5xx
    description: "Http responses of Azure Application Gateway by backend and status code"
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BackendResponseStatus
      aggregation:
        type: Total
      dimension:
        name: BackendPool
    resources:
      - resourceUri: Microsoft.Network/applicationGateways/my-appgw1234
        resourceGroupName: my-rg
        filter: HttpStatusGroup eq '5xx'

Logs

No response

Platform

Microsoft Azure

Contact Details

No response

[tomkerkhove]

May I ask why you’ve added ProtocolType eq 'TCP' to resourceDiscoveryGroups? That is not really supported today.

[DaveOHenry]