[disk] Blacklist certain partitions by default

Continuation of https://github.com/DataDog/datadog-agent/issues/1961 Advanced filtering logic introduced in https://github.com/DataDog/integrations-core/pull/2483

Adding new things to blacklist by default will be a breaking change and will likely require a major Agent release. As such, let’s take the time to compile everything that should be excluded by default.

So far we have:

• file_system_blacklist

  • autofs$
  • iso9660$ (cdrom) note we already do this one

• device_blacklist

  • none so far

• mount_point_blacklist

  • /proc/sys/fs/binfmt_misc (added now, see https://github.com/DataDog/integrations-core/pull/7650)

It should also be noted that since blacklists take precedence over whitelists, ~users would need to update both to re-enable something. Therefore, reducing the occurrence of that should be a goal.~ Implemented a better way: https://github.com/DataDog/integrations-core/pull/7648

cc @DataDog/agent-integrations @DataDog/agent-core @DataDog/container-integrations

cc original participants @techdragon @amineo @coreypobrien @sudermanjr @steinnes @j-vizcaino @nerdinand

[sandstrom]

I was bitten by this recently. Mine looked something like this:

udev             1977284       0   1977284   0% /dev
tmpfs             397864     744    397120   1% /run
/dev/nvme0n1p1  24329532 7438244  16874904  31% /
tmpfs            1989304       0   1989304   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs            1989304       0   1989304   0% /sys/fs/cgroup
/dev/loop5         16896   16896         0 100% /snap/amazon-ssm-agent/784
/dev/loop2         18432   18432         0 100% /snap/amazon-ssm-agent/930
…

And the end-result in Datadog was that the value system.disk.in_use would be the average of several devices (some are already filtered by the Datadog agent per default, but not all of them). Since some were at 0%, it would deflate the actual value, so our trigger at 80% usage wouldn’t fire even though the main disk was at ~99% and the system crashed.

For a tool such as DataDog, I don’t think I’m alone in assuming that a monitor with something like warn if disk.in_use > 80 would “do the right thing” out of the box.

Just wanted to share this example.

[zippolyte]

As a user, this behaviour seems a bit awkward: could we make it blacklist first, then whitelist entries after that? That would allow us to blacklist every FS by default, then “punch holes” by whitelisting a small set of FS that provide meaningful stats.

Well if you want just a specific set of FS, you just need to whitelist them, and only the whitelisted ones will be considered, no need to explicitly blacklist the ones you don’t want. The precedence of the blacklist over the whitelist only applies when there is an intersection between whitelisted elements, and blacklisted ones. In this case then the elements that match both the whitelist and the blacklist will be blacklisted. Does that make sense ?

This is the way we have implemented blacklist/whitelist in other integrations as well.