(process.py:454) | No matching process '...' was found (when process is running fine)

Note: If you have a feature request, you should contact support so the request can be properly tracked.

Output of the info page

===============
Agent (v6.23.1)
===============

  Status date: 2020-11-20 07:54:17.694915 UTC
  Agent start: 2020-11-18 02:38:57.638476 UTC
  Pid: 17256
  Go Version: go1.14.7
  Python Version: 2.7.18
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: info

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -2µs
    System UTC time: 2020-11-20 07:54:17.694915 UTC

  Host Info
  =========
    bootTime: 2020-06-10 08:18:57.000000 UTC
    kernelArch: x86_64
    kernelVersion: 5.3.0-1020-gcp
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 18.04
    procs: 150
    uptime: 3858h20m1s
    virtualizationRole: guest

  Hostnames
  =========
    host_aliases: [redacted]
    hostname: redacted
    socket-fqdn: redacted
    socket-hostname: redacted
    host tags:
      env:staging
      env:staging
      http-server
      https-server
      zone:europe-west1-b
      instance-type:n2-standard-4
      internal-hostname:redacted
      instance-id:506**************
      project:redacted
      numeric_project_id:9764**********
    hostname provider: gce
    unused hostname providers:
      configuration/environment: hostname is empty

  Metadata
  ========
    cloud_provider: GCP
    hostname_source: gce

=========
Collector
=========

  Running Checks
  ==============
  
  <snip>

    process (1.16.0)
    ----------------
      Instance ID: process:fail2ban:f69385006d560585 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/process.d/process.yaml
      Total Runs: 12,781
      Metric Samples: Last Run: 17, Total: 217,275
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 12,781
      Average Execution Time : 3ms
      Last Execution Date : 2020-11-20 07:54:12.000000 UTC
      Last Successful Execution Date : 2020-11-20 07:54:12.000000 UTC

      Instance ID: process:sidekiq:2026c1cda5d3f74d [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/process.d/process.yaml
      Total Runs: 12,781
      Metric Samples: Last Run: 17, Total: 213,783
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 12,781
      Average Execution Time : 2ms
      Last Execution Date : 2020-11-20 07:54:05.000000 UTC
      Last Successful Execution Date : 2020-11-20 07:54:05.000000 UTC

  <snip>

Additional environment details (Operating System, Cloud provider, etc):

/etc/datadog-agent/conf.d/process.d/process.yaml

instances:
  - name: sidekiq
    search_string: ['sidekiq']
    exact_match: False
    thresholds:
      critical: [1, 4]
    tags:
      - env:staging

Steps to reproduce the issue:

1. Restarting the processes
2. Processes restart, but they are still running
3. The collector does not seem to find the (running) processes, and we can see a warning; our process monitoring alerts also fire a false negative
4. After a little while, it finds the processes again and the alert goes away

Describe the results you received:

Nov 20 07:44:54 host.name agent[22753]: 2020-11-20 07:44:54 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:120 in LogMessage) | process:sidekiq:6c9c73d8f1648c
27 | (process.py:454) | No matching process 'sidekiq' was found

Describe the results you expected:

the process collector should detect the running processes

Additional information you deem important (e.g. issue happens only occasionally):

It probably started after upgrading the datadog agent. Unfortunately I do not recall what the previous agent was, but we didn’t update it in a little while now… Our configs did not change however, nor anything that I can relate to about these processes.

[gingerlime]

see https://github.com/DataDog/integrations-core/pull/8263

[gingerlime]

see #8414