`kubernetes.cpu.usage.total` is reported in nanocores

The Kubernetes integraton (https://github.com/DataDog/integrations-core/tree/master/kubernetes) provides CPU stats in two different units:

• cpus (for kubernetes.cpu.limits and kubernetes.cpu.requests)
• percent_nano (for kubernetes.cpu.usage.total)

These metrics are quite useful to put on the same graph in order to tune the requests/limits settings for Kubernetes containers. To do that, you have to scale kubernetes.cpu.usage.total by 1000000000 (one billion), because the value being reported by the integration is in units of nano CPUs (ie, percent_nano, though there doesn’t appear to be any percent math applying. Somewhat annoyingly, you have to specify 1000000000 as (1000000 * 1000) because the web UI will otherwise convert it to 1e9 in the JSON version of a graph, but 1e9 isn’t actually a valid value to scale a metric by.

The values are scraped from CAdvisor, who defines them here: https://github.com/google/cadvisor/blob/e14ee9be3506d260847d263e26a3e9e27f83ad96/info/v1/container.go#L267-L283, being pulled from the Docker daemon’s statistics (https://github.com/moby/moby/blob/8b1adf55c2af329a4334f21d9444d6a169000c81/daemon/stats/collector_unix.go#L27-L71). It’s essentially a rate (per second) of nanoseconds of CPU time used (thus, nanocpu).

I would like to have the values of kubernetes.cpu.usage.total appear as just “cpus” for use in DataDog. Right off the bat, two solutions jump out at me:

• scale the kubernetes.cpu.usage.total metric by a billion at the agent level, and change its unit from percent_nano to cpu - I’ve made a PR for this (link coming shortly)
• make a new unit, nanocores, that will accept the current values being sent, but display them as cores (similar to how it seems that byte-based metrics work)
• maybe other folks have other ideas?

[balusarakesh]

is there a solution for the above problem from @msvechla? suggestion from @ryanooi doesn’t seem to work, we are looking for a way to get a pod’s CPU utilization

this is the correct function ( ( a / 1000000000 ) / b ) * 100

[msvechla]

@JulienBalestra what would be the best way to plot the percentage of CPU used now? I tried it with kubernetes.cpu.usage.total / kubernetes.cpu.capacity, however I am not getting the correct percentage numbers in the graph: