Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Scraping resources regularly failing

See original GitHub issue

In combination with the resource discovery agent we see multiple error like the following in the scraper agent logs: [12:12:00 FTL] Failed to scrape resource collection vm-landscape: This instance has already started one or more requests. Properties can only be modified before sending the first request.

Btw. really nice feature this resource discovery agent

Expected Behavior

No such errors as it seems to break the fetching of the metrics.

Actual Behavior

[12:21:01 INF] Found value 100 for metric network_expressroutecircuits_arpavailability with aggregation interval 00:05:00 [12:21:01 INF] Found value 100 for metric network_expressroutecircuits_bgpavailability with aggregation interval 00:05:00 [12:21:01 INF] Metrics declaration is using spec version v1 [12:21:01 INF] Metrics declaration is using spec version v1 [12:21:08 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics [12:21:08 INF] Request finished in 0.2773ms 200 text/plain; version=0.0.4 [12:21:23 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics [12:21:23 INF] Request finished in 0.3357ms 200 text/plain; version=0.0.4 [12:21:38 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics [12:21:38 INF] Request finished in 0.3034ms 200 text/plain; version=0.0.4 [12:21:53 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics [12:21:53 INF] Request finished in 0.3079ms 200 text/plain; version=0.0.4 [12:22:00 INF] Scraping Azure Monitor - 08/27/2020 12:22:00 +00:00 [12:22:00 INF] Scraping network_expressroutecircuits_arpavailability for resource type Generic [12:22:00 INF] Scraping resource collection vm-landscape - 08/27/2020 12:22:00 +00:00 [12:22:00 WRN] Failed to log HTTP dependency. Reason: This operation is not supported for a relative URI. [12:22:00 FTL] Failed to scrape resource collection vm-landscape: This instance has already started one or more requests. Properties can only be modified before sending the first request. System.InvalidOperationException: This instance has already started one or more requests. Properties can only be modified before sending the first request. at System.Net.Http.HttpClient.CheckDisposedOrStarted() at System.Net.Http.HttpClient.set_BaseAddress(Uri value) at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendRequestToApiAsync(HttpRequestMessage request) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 72 at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uri) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 57 at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 40 at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 23 at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80 [12:22:00 INF] Scraping Azure Monitor - 08/27/2020 12:22:00 +00:00 [12:22:00 INF] Scraping network_expressroutecircuits_bgpavailability for resource type Generic [12:22:01 INF] Found value 100 for metric network_expressroutecircuits_arpavailability with aggregation interval 00:05:00 [12:22:01 INF] Metrics declaration is using spec version v1 [12:22:02 INF] Found value 100 for metric network_expressroutecircuits_bgpavailability with aggregation interval 00:05:00 [12:22:02 INF] Metrics declaration is using spec version v1 [12:22:08 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics [12:22:08 INF] Request finished in 0.3236ms 200 text/plain; version=0.0.4 [12:22:23 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics [12:22:23 INF] Request finished in 0.3189ms 200 text/plain; version=0.0.4

Steps to Reproduce the Problem

No specific steps. See configs below. Just starting both agents and we get the errors

Configuration

Used scraping configuration

....
metrics:
  - name: azure_virtual_machine_percentage_cpu
    description: "Average percentage cpu usage on an Azure virtual machine"
    resourceType: VirtualMachine
    scraping:
      # Every 2 minutes
      schedule: "0 */2 * ? * *"
    azureMetricConfiguration:
      metricName: Percentage CPU
      aggregation:
        type: Average
        interval: 00:01:00
    resourceDiscoveryGroups:
    - name: vm-landscape
  - name: network_expressroutecircuits_arpavailability
    description: ""
    resourceType: Generic
    azureMetricConfiguration:
      metricName: ArpAvailability
      aggregation:
        type: Average
    resources:
    - resourceUri: Microsoft.Network/expressRouteCircuits/NAME
  - name: network_expressroutecircuits_bgpavailability
    description: ""
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BgpAvailability
      aggregation:
        type: Average
    resources:
    - resourceUri: Microsoft.Network/expressRouteCircuits/NAME

Used discovery configuration

version: v1
azureLandscape:
  tenantId: ID
  subscriptions:
  - SUBID
  cloud: Global
resourceDiscoveryGroups:
- name: vm-landscape
  type: VirtualMachine