php_fpm: Sporadic error "Failed to get metrics"

Requesting the PHP-FPM status URL can sporadically fail with an HTTP 503 error “scoreboard already in used”.

See: https://github.com/php/php-src/blob/d84ef967424343abcc48b8f945d5e7807f1c1e09/sapi/fpm/fpm/fpm_status.c#L96

Maybe Datadog could retry once or twice before raising an error?

ERROR (php_fpm.py:91): Failed to get metrics from http://localhost/fpm-status.
  Error 503 Server Error: Service Unavailable for url: http://localhost/fpm-status?json=True
ERROR (__init__.py:829): Check 'php_fpm' instance #0 failed
  Traceback (most recent call last):
    File "/opt/datadog/5.25.0/agent/checks/__init__.py", line 812, in run
      self.check(copy.deepcopy(instance))
    File "/opt/datadog/latest/venv/lib/python2.7/site-packages/datadog_checks/php_fpm/php_fpm.py", line 74, in check
      raise status_exception  # pylint: disable=E0702
  HTTPError: 503 Server Error: Service Unavailable for url: http://localhost/fpm-status?json=True

[masci]

Hi @steffenweber thanks for the detailed report. Not sure how often and for how long we can incur in that lock but a retry policy seems easy to add, going to schedule some work, will keep this issue updated.

[steffenweber]

The fix works! 👍