Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

strange errors with truenas scale

See original GitHub issue

Hi Travis, After upgrading my storage server to TrueNAS Scale 22.02.1 I am experiencing some strange errors with the latest CSI driver. When a volume is created or deleted the below errors appear in the logs and the csi-controller restarts. At the same time many of the volumes for my other statefulsets become read-only. I checked the logs of the iscsi service on the truenas server but there are no errors there. Same is for the logs of the iscsi client service on the kubernetes nodes.

Do you have any ideas what else can i check to see what the problem might be?

I0525 18:54:11.948231       1 connection.go:187] GRPC error: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); malformed header: missing HTTP content-type
E0525 18:54:11.948265       1 controller.go:1481] delete "pvc-0f6b10ec-2d49-4215-a51a-b94fd2f49efa": volume deletion failed: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); malformed header: missing HTTP content-type
W0525 18:54:11.948298       1 controller.go:989] Retrying syncing volume "pvc-0f6b10ec-2d49-4215-a51a-b94fd2f49efa", failure 4
E0525 18:54:11.948319       1 controller.go:1007] error syncing volume "pvc-0f6b10ec-2d49-4215-a51a-b94fd2f49efa": rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); malformed header: missing HTTP content-type
I0525 18:54:11.948352       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-0f6b10ec-2d49-4215-a51a-b94fd2f49efa", UID:"a9b0cb1d-b997-4c6c-babc-0bf04cef3c55", APIVersion:"v1", ResourceVersion:"506885956", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); malformed header: missing HTTP content-type

I0525 05:01:09.738873       1 feature_gate.go:245] feature gates: &{map[]}
I0525 05:01:09.738953       1 csi-provisioner.go:139] Version: v3.1.0
I0525 05:01:09.738960       1 csi-provisioner.go:162] Building kube configs for running in cluster...
I0525 05:01:09.745901       1 connection.go:154] Connecting to unix:///csi-data/csi.sock
I0525 05:01:11.762817       1 common.go:111] Probing CSI driver for readiness
I0525 05:01:11.763279       1 connection.go:183] GRPC call: /csi.v1.Identity/Probe
I0525 05:01:11.763297       1 connection.go:184] GRPC request: {}
I0525 05:01:31.772525       1 connection.go:186] GRPC response: {}
I0525 05:01:31.772764       1 connection.go:187] GRPC error: rpc error: code = Internal desc = Error: Timed out while waiting for handshake Error: Timed out while waiting for handshake
    at Timeout._onTimeout (/home/csi/app/node_modules/ssh2/lib/client.js:1014:23)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)
E0525 05:01:31.772814       1 csi-provisioner.go:197] CSI driver probe failed: rpc error: code = Internal desc = Error: Timed out while waiting for handshake Error: Timed out while waiting for handshake
    at Timeout._onTimeout (/home/csi/app/node_modules/ssh2/lib/client.js:1014:23)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)

Thanks!