Question on democratic-csi PVC inode counts / Prometheus

I’m using a Grafana Dashboard / Prometheus (kube-stack-prometheus) to monitor my democratic-csi iSCSI volumes. The “Volume Space Usage” seems to be fine.

However, the “Volume Inodes Usages” shows zeros. I am using XFS instead of EXT4 but XFS still has inodes from what I remember.

I have democratic-csi iSCSI volumes in the namespace “monitoring”, queried via kubelet_volume_stats_inodes_used{namespace="monitoring"} and all of them report a value of zero for inodes used:

kubelet_volume_stats_inodes_used{endpoint="https-metrics", instance="192.168.10.215:10250", job="kubelet", metrics_path="/metrics", namespace="monitoring", node="k3s01", persistentvolumeclaim="kube-prometheus-stack-grafana", service="kube-prometheus-stack-kubelet"} | 0

kubelet_volume_stats_inodes_used{endpoint="https-metrics", instance="192.168.10.216:10250", job="kubelet", metrics_path="/metrics", namespace="monitoring", node="k3s02", persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0", service="kube-prometheus-stack-kubelet"} | 0

kubelet_volume_stats_inodes_used{endpoint="https-metrics", instance="192.168.10.217:10250", job="kubelet", metrics_path="/metrics", namespace="monitoring", node="k3s03", persistentvolumeclaim="alertmanager-kube-prometheus-stack-alertmanager-db-alertmanager-kube-prometheus-stack-alertmanager-0", service="kube-prometheus-stack-kubelet"} | 0

I don’t understand how the data is pulled, but I assume democratic-csi is not providing the data??

If I point it to a Longhorn volume, it does return the number of inodes used:

kubelet_volume_stats_inodes_used{endpoint="https-metrics", instance="192.168.10.217:10250", job="kubelet", metrics_path="/metrics", namespace="unifi", node="k3s03", persistentvolumeclaim="unifi-longhorn-pvc", service="kube-prometheus-stack-kubelet"} | 3293

Prometheus itself is using a 50GB democratic-csi iSCSI PVC, if I go into the container, it reports inodes:

/prometheus $ df -h .
Filesystem                Size      Used Available Use% Mounted on
/dev/sda                 50.0G     19.5G     30.5G  39% /prometheus

/prometheus $ df -i .
Filesystem              Inodes      Used Available Use% Mounted on
/dev/sda              26214400       165  26214235   0% /prometheus

These also return zeros for each PVC: kubelet_volume_stats_inodes{namespace="monitoring"} kubelet_volume_stats_inodes_free{namespace="monitoring"}

[reefland]

It’s ok if you sleep and eat once in a while too. 😃

Fantastic. Spun up a test VM using the next image. Looking good so far.

And double-checked:

$ k exec -it prometheus-kube-prometheus-stack-prometheus-0 -n monitoring -c prometheus -- /bin/sh
/prometheus $ df -i .
Filesystem              Inodes      Used Available Use% Mounted on
/dev/sdf              26214400         9  26214391   0% /prometheus

[travisghansen]

Good question. I’m guessing k8s invokes the GetVolumeStats method of the Node service. I’m currently only returning byte info but inode could also be added to the mix.