Using excessive file descriptors

Promitor uses too many Linux file descriptors, mainly sockets. We’re running Promitor on Kubernetes and this has resulted in the Node running out of file descriptors a few times, where other Pods will crash with too many open files error.

Expected Behavior

Promitor should re-use sockets or release them after use.

Actual Behavior

Promitor opens sockets and doesn’t seem to release them in a timely fashion.

Steps to Reproduce the Problem

1. Run a Promitor container, with a scrape schedule of once per minute.
2. Attach to the container and count the open file descriptors regularly, this is an example of a container that’s been running for 6 days:

   /app # lsof | wc -l
   376504
   /app # lsof | wc -l
   376845
   

3. The number of open file descriptors should grow over time, nearing the limit of 810243.

Configuration

Provide insights in the configuration that you are using:

• Configured scraping schedule: */1 * * * *

Used scraping configuration

This is our staging configuration, which scrapes less resources but still encounters the same issue.

version: v1
azureMetadata:
  tenantId:
  subscriptionId:
  resourceGroupName:
metricDefaults:
  aggregation:
    interval: "00:05:00"
  scraping:
    schedule: "*/1 * * * *"
metrics:
- name: "azure_sql_cpu_percent_average"
  description: "'cpu_percent' with aggregation 'Average'"
  resourceType: "Generic"
  labels:
    component: "sql-database"
  azureMetricConfiguration:
    metricName: "cpu_percent"
    aggregation:
      type: "Average"
  resources:
  - resourceGroupName: "groupName"
    resourceUri: "Microsoft.Sql/servers/serverName/databases/dbName"
  # 7 more databases
# 11 more sql metrics
# 6 loadbalancer metrics
# 12 redis metrics
# 10 service bus metrics

Specifications

• Version: 1.0.0 (image tag)
• Platform: Docker (Linux)
• Subsystem:

[tomkerkhove]

Might have found something in #844 @ResDiaryLewis.

Looks like Azure SDK create a ton of HttpClients.

[tomkerkhove]

Thanks for the additional information and sorry for that!

Will have a look but since we don’t use HttpClient directly it must be one of our dependencies so all information/telemetry is welcome.