NodeGetVolumeStats warning - operation locked due to in progress operation(s)

Hello,

In the last few months of using/testing democratic-csi I stumbled upon a questionable warning regarding the RPC NodeGetVolumeStats. I’m using a zfs-generic-nfs driver and my cluster namespace which depends on the NFS share based PVs, consists of one ReplicaSet based pod, meanwhile other pods are defined through jobs and start at the specific time. Everything related to processing and storing files works without any issue, but I’m getting the following warning with almost every pod which starts via job (usually there are multiple different pods started at the same time and having bound the same PVC):

{"host":"tka0w","level":"error","message":"handler error - driver: ControllerZfsGenericDriver method: NodeGetVolumeStats error: {\"name\":\"GrpcError\",\"code\":10,\"message\":\"operation locked due to in progress operation(s): [\\\"volume_id_pvc-3e586856-c4be-4b1d-b786-0f12c0605fe0\\\"]\"}","service":"democratic-csi","timestamp":"2022-10-21T15:10:02.224Z"}

After looking into the code/logic of driver, the RPC operations working with volumes are “secured” by the lock on a global level (meaning on a specific node where csi node pod is running and getting requests from OC). One of such RPC is also NodeGetVolumeStats, which is returning usages of certain storage unit. The lock names are composed from “volume_id” prefix and volume name and the locks itself are inserted into the set when the RPC starts executing. They are the same for multiple RPCs as visible around the following line in file https://github.com/democratic-csi/democratic-csi/blob/master/src/utils/general.js#L92 .

If I’m correct, CSI spec defines that OC should not fire multiple same NodeStageVolume/NodePublishVolume RPC requests before getting response (except if it loses the current state, but this is then resolved with locks and aborted request in CSI driver implementation). I was searching if the same is also true for NodeGetVolumeStats RPC, but without success. Therefore I looked the source of few other CSI driver implementations:

• csi-driver-host-path: https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/pkg/hostpath/nodeserver.go#L406
• aws-ebs-csi-driver: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/pkg/driver/node.go#L453
• azurefile-csi-driver: https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/pkg/azurefile/nodeserver.go#L417
• azuredisk-csi-driver: https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/pkg/azuredisk/nodeserver.go#L416

I found out that only csi-driver-host-path implementation of NodeGetVolumeStats contains mutex lock on a global level, meanwhile other csi drivers don’t. I also checked where the mentioned RPC is called in Kubernetes source and found out it’s used only for metrics (it’s possible I missed something, as I was searching for the NodeGetVolumeStats via Github search in kubernetes repo).

The biggest question for me after I’ve done this research is, why should we use volume based lock inside NodeGetVolumeStats? To picturize the problem - if I have 3 pods requiring the same PVC, this will result in 3 separate NodePublishVolume calls, each followed by NodeGetVolumeStats. The first NodePublishVolume will be successful and after NodeGetVolumeStats for that volume will be issued. At the same time the second NodePublishVolume call will start, but will fail early because of the lock on current NodeGetVolumeStats operation. At this point the operation locked due to in progress operation(s) warning appears and causes a lot of retries for different RPC calls.

I tried to solve the issue on my own and removed NodeGetVolumeStats from locking (commenting it out in lockKeysFromRequest function) and build my own image. So far everything is working good and since I changed the image there was not a single operation locked due to in progress operation(s) warning. Since I’m probably not fully aware of issues that my arise with such approach, I would appreciate any feedback on this. If I would be 100% sure this will not break something, I’d rather open a PR with the proposed change.

[travisghansen]

Great research! It’s been a long time since I tampered with that code but the only reason it’s locking is because I probably added a default for all calls. I’m happy to remove it for that call yes.

[travisghansen]

Ah! Ok. I’ve been crazy busy lately and haven’t had time to look into that one with the focus it deserves. I’m probably still another week or so out before I can really dig in. We can wait.