Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

failed to execute filesystem command: realpath

See original GitHub issue

Suddenly my node CSI pod went into a crashloop. It seems to be an issue with one of the PVC’s, it always crashes on the same. My other PVC’s that are already in use are fine. Creating new pods with a PVC fails a couple of times with

MountVolume.MountDevice failed for volume "pvc-95d7ff10-8fcb-4247-9ac2-bddf210f9874" : rpc error: code = Unavailable desc = Bad Gateway: HTTP status code 502; transport: missing content-type field

Since the node CSI pod is down, but eventually succeeds when requesting the PVC is done while the node CSI pod is up for a moment.

Log output from the node CSI pod:

grpc implementation: @grpc/grpc-js
{"host":"k3os-21898","level":"info","message":"initializing csi driver: freenas-iscsi","service":"democratic-csi","timestamp":"2022-05-10T07:44:46.276Z"}
{"host":"k3os-21898","level":"debug","message":"setting default identity service caps","service":"democratic-csi","timestamp":"2022-05-10T07:44:46.382Z"}
{"host":"k3os-21898","level":"debug","message":"setting default identity volume_expansion caps","service":"democratic-csi","timestamp":"2022-05-10T07:44:46.382Z"}
{"host":"k3os-21898","level":"debug","message":"setting default controller caps","service":"democratic-csi","timestamp":"2022-05-10T07:44:46.382Z"}
{"host":"k3os-21898","level":"debug","message":"setting default node caps","service":"democratic-csi","timestamp":"2022-05-10T07:44:46.387Z"}
{"host":"k3os-21898","level":"info","message":"starting csi server - name: org.democratic-csi.iscsi, version: 1.6.3, driver: freenas-iscsi, mode: node, csi version: 1.5.0, address: , socket: unix:///csi-data/csi.sock.internal","service":"democratic-csi","timestamp":"2022-05-10T07:44:46.388Z"}
{"host":"k3os-21898","level":"info","message":"new request - driver: FreeNASSshDriver method: Probe call: {\"_events\":{},\"_eventsCount\":0,\"call\":{\"_events\":{},\"_eventsCount\":5,\"stream\":{\"_readableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"buffer\":{\"head\":null,\"tail\":null,\"length\":0},\"length\":0,\"pipes\":[],\"flowing\":true,\"ended\":true,\"endEmitted\":true,\"reading\":false,\"constructed\":true,\"sync\":false,\"needReadable\":false,\"emittedReadable\":false,\"readableListening\":false,\"resumeScheduled\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"destroyed\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false,\"defaultEncoding\":\"utf8\",\"awaitDrainWriters\":null,\"multiAwaitDrain\":false,\"readingMore\":false,\"dataEmitted\":true,\"decoder\":null,\"encoding\":null},\"_events\":{},\"_eventsCount\":8,\"_writableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"finalCalled\":false,\"needDrain\":false,\"ending\":false,\"ended\":false,\"finished\":false,\"destroyed\":false,\"decodeStrings\":false,\"defaultEncoding\":\"utf8\",\"length\":0,\"writing\":false,\"corked\":0,\"sync\":true,\"bufferProcessing\":false,\"writecb\":null,\"writelen\":0,\"afterWriteTickInfo\":null,\"buffered\":[],\"bufferedIndex\":0,\"allBuffers\":true,\"allNoop\":true,\"pendingcb\":0,\"constructed\":true,\"prefinished\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false},\"allowHalfOpen\":true},\"handler\":{\"type\":\"unary\",\"path\":\"/csi.v1.Identity/Probe\"},\"options\":{},\"cancelled\":false,\"deadlineTimer\":{\"_idleTimeout\":-1,\"_idlePrev\":null,\"_idleNext\":null,\"_idleStart\":11076,\"_onTimeout\":null,\"_repeat\":null,\"_destroyed\":true},\"deadline\":null,\"wantTrailers\":false,\"metadataSent\":false,\"canPush\":false,\"isPushPending\":false,\"bufferedMessages\":[],\"messagesToPush\":[],\"maxSendMessageSize\":-1,\"maxReceiveMessageSize\":4194304},\"metadata\":{\"user-agent\":[\"grpc-node-js/1.6.2\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-05-10T07:44:57.065Z"}
{"host":"k3os-21898","level":"info","message":"new response - driver: FreeNASSshDriver method: Probe response: {\"ready\":{\"value\":true}}","service":"democratic-csi","timestamp":"2022-05-10T07:44:57.065Z"}
{"host":"k3os-21898","level":"info","message":"new request - driver: FreeNASSshDriver method: NodeGetCapabilities call: {\"_events\":{},\"_eventsCount\":0,\"call\":{\"_events\":{},\"_eventsCount\":5,\"stream\":{\"_readableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"buffer\":{\"head\":null,\"tail\":null,\"length\":0},\"length\":0,\"pipes\":[],\"flowing\":true,\"ended\":true,\"endEmitted\":true,\"reading\":false,\"constructed\":true,\"sync\":false,\"needReadable\":false,\"emittedReadable\":false,\"readableListening\":false,\"resumeScheduled\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"destroyed\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false,\"defaultEncoding\":\"utf8\",\"awaitDrainWriters\":null,\"multiAwaitDrain\":false,\"readingMore\":false,\"dataEmitted\":true,\"decoder\":null,\"encoding\":null},\"_events\":{},\"_eventsCount\":8,\"_writableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"finalCalled\":false,\"needDrain\":false,\"ending\":false,\"ended\":false,\"finished\":false,\"destroyed\":false,\"decodeStrings\":false,\"defaultEncoding\":\"utf8\",\"length\":0,\"writing\":false,\"corked\":0,\"sync\":true,\"bufferProcessing\":false,\"writecb\":null,\"writelen\":0,\"afterWriteTickInfo\":null,\"buffered\":[],\"bufferedIndex\":0,\"allBuffers\":true,\"allNoop\":true,\"pendingcb\":0,\"constructed\":true,\"prefinished\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false},\"allowHalfOpen\":true},\"handler\":{\"type\":\"unary\",\"path\":\"/csi.v1.Node/NodeGetCapabilities\"},\"options\":{},\"cancelled\":false,\"deadlineTimer\":{\"_idleTimeout\":120000,\"_idlePrev\":{\"expiry\":137560,\"id\":-9007199254740988,\"msecs\":120000,\"priorityQueuePosition\":1},\"_idleStart\":17560,\"_timerArgs\":[null],\"_repeat\":null,\"_destroyed\":false},\"deadline\":1652168823539,\"wantTrailers\":false,\"metadataSent\":false,\"canPush\":false,\"isPushPending\":false,\"bufferedMessages\":[],\"messagesToPush\":[],\"maxSendMessageSize\":-1,\"maxReceiveMessageSize\":4194304},\"metadata\":{\"user-agent\":[\"grpc-go/1.27.1\"],\"x-forwarded-host\":[\"/var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.540Z"}
{"host":"k3os-21898","level":"info","message":"new response - driver: FreeNASSshDriver method: NodeGetCapabilities response: {\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}},{\"rpc\":{\"type\":\"GET_VOLUME_STATS\"}},{\"rpc\":{\"type\":\"EXPAND_VOLUME\"}},{\"rpc\":{\"type\":\"SINGLE_NODE_MULTI_WRITER\"}}]}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.540Z"}
{"host":"k3os-21898","level":"info","message":"new request - driver: FreeNASSshDriver method: NodeGetVolumeStats call: {\"_events\":{},\"_eventsCount\":0,\"call\":{\"_events\":{},\"_eventsCount\":5,\"stream\":{\"_readableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"buffer\":{\"head\":null,\"tail\":null,\"length\":0},\"length\":0,\"pipes\":[],\"flowing\":true,\"ended\":true,\"endEmitted\":true,\"reading\":false,\"constructed\":true,\"sync\":false,\"needReadable\":false,\"emittedReadable\":false,\"readableListening\":false,\"resumeScheduled\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"destroyed\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false,\"defaultEncoding\":\"utf8\",\"awaitDrainWriters\":null,\"multiAwaitDrain\":false,\"readingMore\":false,\"dataEmitted\":true,\"decoder\":null,\"encoding\":null},\"_events\":{},\"_eventsCount\":8,\"_writableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"finalCalled\":false,\"needDrain\":false,\"ending\":false,\"ended\":false,\"finished\":false,\"destroyed\":false,\"decodeStrings\":false,\"defaultEncoding\":\"utf8\",\"length\":0,\"writing\":false,\"corked\":0,\"sync\":true,\"bufferProcessing\":false,\"writecb\":null,\"writelen\":0,\"afterWriteTickInfo\":null,\"buffered\":[],\"bufferedIndex\":0,\"allBuffers\":true,\"allNoop\":true,\"pendingcb\":0,\"constructed\":true,\"prefinished\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false},\"allowHalfOpen\":true},\"handler\":{\"type\":\"unary\",\"path\":\"/csi.v1.Node/NodeGetVolumeStats\"},\"options\":{},\"cancelled\":false,\"deadlineTimer\":{\"_idleTimeout\":119995,\"_idlePrev\":{\"expiry\":137560,\"id\":-9007199254740985,\"msecs\":119995,\"priorityQueuePosition\":2},\"_idleStart\":17565,\"_timerArgs\":[null],\"_repeat\":null,\"_destroyed\":false},\"deadline\":1652168823539,\"wantTrailers\":false,\"metadataSent\":false,\"canPush\":false,\"isPushPending\":false,\"bufferedMessages\":[],\"messagesToPush\":[],\"maxSendMessageSize\":-1,\"maxReceiveMessageSize\":4194304},\"metadata\":{\"user-agent\":[\"grpc-go/1.27.1\"],\"x-forwarded-host\":[\"/var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock\"]},\"request\":{\"volume_id\":\"pvc-c63ba50f-cbbd-4865-a93a-0eb5cd7cbaaa\",\"volume_path\":\"/var/lib/kubelet/pods/d0b2607c-c163-4bb2-a792-d71dfe8fac62/volumes/kubernetes.io~csi/pvc-c63ba50f-cbbd-4865-a93a-0eb5cd7cbaaa/mount\",\"staging_target_path\":\"\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.545Z"}
{"host":"k3os-21898","level":"debug","message":"operation lock keys: [\"volume_id_pvc-c63ba50f-cbbd-4865-a93a-0eb5cd7cbaaa\"]","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.545Z"}
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/d0b2607c-c163-4bb2-a792-d71dfe8fac62/volumes/kubernetes.io~csi/pvc-c63ba50f-cbbd-4865-a93a-0eb5cd7cbaaa/mount --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/d0b2607c-c163-4bb2-a792-d71dfe8fac62/volumes/kubernetes.io~csi/pvc-c63ba50f-cbbd-4865-a93a-0eb5cd7cbaaa/mount --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /var/lib/kubelet/pods/d0b2607c-c163-4bb2-a792-d71dfe8fac62/volumes/kubernetes.io~csi/pvc-c63ba50f-cbbd-4865-a93a-0eb5cd7cbaaa/mount
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/d0b2607c-c163-4bb2-a792-d71dfe8fac62/volumes/kubernetes.io~csi/pvc-c63ba50f-cbbd-4865-a93a-0eb5cd7cbaaa/mount/block_device --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: lsblk -a -b -l -J -O
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/d0b2607c-c163-4bb2-a792-d71dfe8fac62/volumes/kubernetes.io~csi/pvc-c63ba50f-cbbd-4865-a93a-0eb5cd7cbaaa/mount --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/d0b2607c-c163-4bb2-a792-d71dfe8fac62/volumes/kubernetes.io~csi/pvc-c63ba50f-cbbd-4865-a93a-0eb5cd7cbaaa/mount --output source,target,fstype,label,options,avail,size,used -b -J --nofsroot
executing filesystem command: realpath /dev/loop0
{"host":"k3os-21898","level":"info","message":"new response - driver: FreeNASSshDriver method: NodeGetVolumeStats response: {\"usage\":[{\"available\":49814515712,\"total\":52576092160,\"used\":60444672,\"unit\":\"BYTES\"}]}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.599Z"}
{"host":"k3os-21898","level":"info","message":"new request - driver: FreeNASSshDriver method: NodeGetCapabilities call: {\"_events\":{},\"_eventsCount\":0,\"call\":{\"_events\":{},\"_eventsCount\":5,\"stream\":{\"_readableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"buffer\":{\"head\":null,\"tail\":null,\"length\":0},\"length\":0,\"pipes\":[],\"flowing\":true,\"ended\":true,\"endEmitted\":true,\"reading\":false,\"constructed\":true,\"sync\":false,\"needReadable\":false,\"emittedReadable\":false,\"readableListening\":false,\"resumeScheduled\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"destroyed\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false,\"defaultEncoding\":\"utf8\",\"awaitDrainWriters\":null,\"multiAwaitDrain\":false,\"readingMore\":false,\"dataEmitted\":true,\"decoder\":null,\"encoding\":null},\"_events\":{},\"_eventsCount\":8,\"_writableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"finalCalled\":false,\"needDrain\":false,\"ending\":false,\"ended\":false,\"finished\":false,\"destroyed\":false,\"decodeStrings\":false,\"defaultEncoding\":\"utf8\",\"length\":0,\"writing\":false,\"corked\":0,\"sync\":true,\"bufferProcessing\":false,\"writecb\":null,\"writelen\":0,\"afterWriteTickInfo\":null,\"buffered\":[],\"bufferedIndex\":0,\"allBuffers\":true,\"allNoop\":true,\"pendingcb\":0,\"constructed\":true,\"prefinished\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false},\"allowHalfOpen\":true},\"handler\":{\"type\":\"unary\",\"path\":\"/csi.v1.Node/NodeGetCapabilities\"},\"options\":{},\"cancelled\":false,\"deadlineTimer\":{\"_idleTimeout\":120000,\"_idlePrev\":{\"expiry\":137885,\"id\":-9007199254740978,\"msecs\":120000,\"priorityQueuePosition\":2},\"_idleStart\":17885,\"_timerArgs\":[null],\"_repeat\":null,\"_destroyed\":false},\"deadline\":1652168823864,\"wantTrailers\":false,\"metadataSent\":false,\"canPush\":false,\"isPushPending\":false,\"bufferedMessages\":[],\"messagesToPush\":[],\"maxSendMessageSize\":-1,\"maxReceiveMessageSize\":4194304},\"metadata\":{\"user-agent\":[\"grpc-go/1.27.1\"],\"x-forwarded-host\":[\"/var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.864Z"}
{"host":"k3os-21898","level":"info","message":"new response - driver: FreeNASSshDriver method: NodeGetCapabilities response: {\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}},{\"rpc\":{\"type\":\"GET_VOLUME_STATS\"}},{\"rpc\":{\"type\":\"EXPAND_VOLUME\"}},{\"rpc\":{\"type\":\"SINGLE_NODE_MULTI_WRITER\"}}]}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.865Z"}
{"host":"k3os-21898","level":"info","message":"new request - driver: FreeNASSshDriver method: NodeGetVolumeStats call: {\"_events\":{},\"_eventsCount\":0,\"call\":{\"_events\":{},\"_eventsCount\":5,\"stream\":{\"_readableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"buffer\":{\"head\":null,\"tail\":null,\"length\":0},\"length\":0,\"pipes\":[],\"flowing\":true,\"ended\":true,\"endEmitted\":true,\"reading\":false,\"constructed\":true,\"sync\":false,\"needReadable\":false,\"emittedReadable\":false,\"readableListening\":false,\"resumeScheduled\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"destroyed\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false,\"defaultEncoding\":\"utf8\",\"awaitDrainWriters\":null,\"multiAwaitDrain\":false,\"readingMore\":false,\"dataEmitted\":true,\"decoder\":null,\"encoding\":null},\"_events\":{},\"_eventsCount\":8,\"_writableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"finalCalled\":false,\"needDrain\":false,\"ending\":false,\"ended\":false,\"finished\":false,\"destroyed\":false,\"decodeStrings\":false,\"defaultEncoding\":\"utf8\",\"length\":0,\"writing\":false,\"corked\":0,\"sync\":true,\"bufferProcessing\":false,\"writecb\":null,\"writelen\":0,\"afterWriteTickInfo\":null,\"buffered\":[],\"bufferedIndex\":0,\"allBuffers\":true,\"allNoop\":true,\"pendingcb\":0,\"constructed\":true,\"prefinished\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false},\"allowHalfOpen\":true},\"handler\":{\"type\":\"unary\",\"path\":\"/csi.v1.Node/NodeGetVolumeStats\"},\"options\":{},\"cancelled\":false,\"deadlineTimer\":{\"_idleTimeout\":119998,\"_idlePrev\":{\"expiry\":137885,\"id\":-9007199254740976,\"msecs\":119998,\"priorityQueuePosition\":2},\"_idleStart\":17887,\"_timerArgs\":[null],\"_repeat\":null,\"_destroyed\":false},\"deadline\":1652168823864,\"wantTrailers\":false,\"metadataSent\":false,\"canPush\":false,\"isPushPending\":false,\"bufferedMessages\":[],\"messagesToPush\":[],\"maxSendMessageSize\":-1,\"maxReceiveMessageSize\":4194304},\"metadata\":{\"x-forwarded-host\":[\"/var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock\"],\"user-agent\":[\"grpc-go/1.27.1\"]},\"request\":{\"volume_id\":\"pvc-46605d5e-dd6a-482b-935c-d9fc1fbfe845\",\"volume_path\":\"/var/lib/kubelet/pods/0e82b7bb-725e-4b7d-bc42-b1ffb9e1141c/volumes/kubernetes.io~csi/pvc-46605d5e-dd6a-482b-935c-d9fc1fbfe845/mount\",\"staging_target_path\":\"\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.867Z"}
{"host":"k3os-21898","level":"debug","message":"operation lock keys: [\"volume_id_pvc-46605d5e-dd6a-482b-935c-d9fc1fbfe845\"]","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.867Z"}
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/0e82b7bb-725e-4b7d-bc42-b1ffb9e1141c/volumes/kubernetes.io~csi/pvc-46605d5e-dd6a-482b-935c-d9fc1fbfe845/mount --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/0e82b7bb-725e-4b7d-bc42-b1ffb9e1141c/volumes/kubernetes.io~csi/pvc-46605d5e-dd6a-482b-935c-d9fc1fbfe845/mount --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /var/lib/kubelet/pods/0e82b7bb-725e-4b7d-bc42-b1ffb9e1141c/volumes/kubernetes.io~csi/pvc-46605d5e-dd6a-482b-935c-d9fc1fbfe845/mount
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/0e82b7bb-725e-4b7d-bc42-b1ffb9e1141c/volumes/kubernetes.io~csi/pvc-46605d5e-dd6a-482b-935c-d9fc1fbfe845/mount/block_device --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: lsblk -a -b -l -J -O
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/0e82b7bb-725e-4b7d-bc42-b1ffb9e1141c/volumes/kubernetes.io~csi/pvc-46605d5e-dd6a-482b-935c-d9fc1fbfe845/mount --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/0e82b7bb-725e-4b7d-bc42-b1ffb9e1141c/volumes/kubernetes.io~csi/pvc-46605d5e-dd6a-482b-935c-d9fc1fbfe845/mount --output source,target,fstype,label,options,avail,size,used -b -J --nofsroot
executing filesystem command: realpath /dev/loop0
{"host":"k3os-21898","level":"info","message":"new response - driver: FreeNASSshDriver method: NodeGetVolumeStats response: {\"usage\":[{\"available\":47249960960,\"total\":52576092160,\"used\":2624999424,\"unit\":\"BYTES\"}]}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.908Z"}
{"host":"k3os-21898","level":"info","message":"new request - driver: FreeNASSshDriver method: NodeGetCapabilities call: {\"_events\":{},\"_eventsCount\":0,\"call\":{\"_events\":{},\"_eventsCount\":5,\"stream\":{\"_readableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"buffer\":{\"head\":null,\"tail\":null,\"length\":0},\"length\":0,\"pipes\":[],\"flowing\":true,\"ended\":true,\"endEmitted\":true,\"reading\":false,\"constructed\":true,\"sync\":false,\"needReadable\":false,\"emittedReadable\":false,\"readableListening\":false,\"resumeScheduled\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"destroyed\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false,\"defaultEncoding\":\"utf8\",\"awaitDrainWriters\":null,\"multiAwaitDrain\":false,\"readingMore\":false,\"dataEmitted\":true,\"decoder\":null,\"encoding\":null},\"_events\":{},\"_eventsCount\":8,\"_writableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"finalCalled\":false,\"needDrain\":false,\"ending\":false,\"ended\":false,\"finished\":false,\"destroyed\":false,\"decodeStrings\":false,\"defaultEncoding\":\"utf8\",\"length\":0,\"writing\":false,\"corked\":0,\"sync\":true,\"bufferProcessing\":false,\"writecb\":null,\"writelen\":0,\"afterWriteTickInfo\":null,\"buffered\":[],\"bufferedIndex\":0,\"allBuffers\":true,\"allNoop\":true,\"pendingcb\":0,\"constructed\":true,\"prefinished\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false},\"allowHalfOpen\":true},\"handler\":{\"type\":\"unary\",\"path\":\"/csi.v1.Node/NodeGetCapabilities\"},\"options\":{},\"cancelled\":false,\"deadlineTimer\":{\"_idleTimeout\":120000,\"_idlePrev\":{\"expiry\":138000,\"id\":-9007199254740969,\"msecs\":120000,\"priorityQueuePosition\":2},\"_idleStart\":18000,\"_timerArgs\":[null],\"_repeat\":null,\"_destroyed\":false},\"deadline\":1652168823979,\"wantTrailers\":false,\"metadataSent\":false,\"canPush\":false,\"isPushPending\":false,\"bufferedMessages\":[],\"messagesToPush\":[],\"maxSendMessageSize\":-1,\"maxReceiveMessageSize\":4194304},\"metadata\":{\"x-forwarded-host\":[\"/var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock\"],\"user-agent\":[\"grpc-go/1.27.1\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.980Z"}
{"host":"k3os-21898","level":"info","message":"new response - driver: FreeNASSshDriver method: NodeGetCapabilities response: {\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}},{\"rpc\":{\"type\":\"GET_VOLUME_STATS\"}},{\"rpc\":{\"type\":\"EXPAND_VOLUME\"}},{\"rpc\":{\"type\":\"SINGLE_NODE_MULTI_WRITER\"}}]}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.981Z"}
{"host":"k3os-21898","level":"info","message":"new request - driver: FreeNASSshDriver method: NodeGetVolumeStats call: {\"_events\":{},\"_eventsCount\":0,\"call\":{\"_events\":{},\"_eventsCount\":5,\"stream\":{\"_readableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"buffer\":{\"head\":null,\"tail\":null,\"length\":0},\"length\":0,\"pipes\":[],\"flowing\":true,\"ended\":true,\"endEmitted\":true,\"reading\":false,\"constructed\":true,\"sync\":false,\"needReadable\":false,\"emittedReadable\":false,\"readableListening\":false,\"resumeScheduled\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"destroyed\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false,\"defaultEncoding\":\"utf8\",\"awaitDrainWriters\":null,\"multiAwaitDrain\":false,\"readingMore\":false,\"dataEmitted\":true,\"decoder\":null,\"encoding\":null},\"_events\":{},\"_eventsCount\":8,\"_writableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"finalCalled\":false,\"needDrain\":false,\"ending\":false,\"ended\":false,\"finished\":false,\"destroyed\":false,\"decodeStrings\":false,\"defaultEncoding\":\"utf8\",\"length\":0,\"writing\":false,\"corked\":0,\"sync\":true,\"bufferProcessing\":false,\"writecb\":null,\"writelen\":0,\"afterWriteTickInfo\":null,\"buffered\":[],\"bufferedIndex\":0,\"allBuffers\":true,\"allNoop\":true,\"pendingcb\":0,\"constructed\":true,\"prefinished\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false},\"allowHalfOpen\":true},\"handler\":{\"type\":\"unary\",\"path\":\"/csi.v1.Node/NodeGetVolumeStats\"},\"options\":{},\"cancelled\":false,\"deadlineTimer\":{\"_idleTimeout\":119996,\"_idlePrev\":{\"expiry\":138001,\"id\":-9007199254740967,\"msecs\":119996,\"priorityQueuePosition\":2},\"_idleStart\":18005,\"_timerArgs\":[null],\"_repeat\":null,\"_destroyed\":false},\"deadline\":1652168823980,\"wantTrailers\":false,\"metadataSent\":false,\"canPush\":false,\"isPushPending\":false,\"bufferedMessages\":[],\"messagesToPush\":[],\"maxSendMessageSize\":-1,\"maxReceiveMessageSize\":4194304},\"metadata\":{\"user-agent\":[\"grpc-go/1.27.1\"],\"x-forwarded-host\":[\"/var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock\"]},\"request\":{\"volume_id\":\"pvc-9c0f8520-79d7-4442-8675-44ad595777bb\",\"volume_path\":\"/var/lib/kubelet/pods/ad05d574-c544-4cbc-ab0c-ca8f0a274f63/volumes/kubernetes.io~csi/pvc-9c0f8520-79d7-4442-8675-44ad595777bb/mount\",\"staging_target_path\":\"\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.985Z"}
{"host":"k3os-21898","level":"debug","message":"operation lock keys: [\"volume_id_pvc-9c0f8520-79d7-4442-8675-44ad595777bb\"]","service":"democratic-csi","timestamp":"2022-05-10T07:45:03.985Z"}
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/ad05d574-c544-4cbc-ab0c-ca8f0a274f63/volumes/kubernetes.io~csi/pvc-9c0f8520-79d7-4442-8675-44ad595777bb/mount --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/ad05d574-c544-4cbc-ab0c-ca8f0a274f63/volumes/kubernetes.io~csi/pvc-9c0f8520-79d7-4442-8675-44ad595777bb/mount --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /var/lib/kubelet/pods/ad05d574-c544-4cbc-ab0c-ca8f0a274f63/volumes/kubernetes.io~csi/pvc-9c0f8520-79d7-4442-8675-44ad595777bb/mount
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/ad05d574-c544-4cbc-ab0c-ca8f0a274f63/volumes/kubernetes.io~csi/pvc-9c0f8520-79d7-4442-8675-44ad595777bb/mount/block_device --output source,target,fstype,label,options -b -J --nofsroot
failed to execute filesystem command: realpath /var/lib/kubelet/pods/ad05d574-c544-4cbc-ab0c-ca8f0a274f63/volumes/kubernetes.io~csi/pvc-9c0f8520-79d7-4442-8675-44ad595777bb/mount, response: {"code":1,"stdout":"","stderr":"realpath: /var/lib/kubelet/pods/ad05d574-c544-4cbc-ab0c-ca8f0a274f63/volumes/kubernetes.io~csi/pvc-9c0f8520-79d7-4442-8675-44ad595777bb/mount: Input/output error\n","timeout":false}
{"date":"Tue May 10 2022 07:45:04 GMT+0000 (Coordinated Universal Time)","error":{"code":"ERR_UNHANDLED_REJECTION"},"exception":true,"host":"k3os-21898","level":"error","message":"uncaughtException: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason \"#<Object>\".\nUnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason \"#<Object>\".","os":{"loadavg":[3.42,5.01,4.41],"uptime":569293.76},"process":{"argv":["/usr/local/bin/node","/home/csi/app/bin/democratic-csi","--csi-version=1.5.0","--csi-name=org.democratic-csi.iscsi","--driver-config-file=/config/driver-config-file.yaml","--log-level=silly","--csi-mode=node","--server-socket=/csi-data/csi.sock.internal"],"cwd":"/home/csi/app","execPath":"/usr/local/bin/node","gid":0,"memoryUsage":{"arrayBuffers":93246,"external":17943751,"heapTotal":19595264,"heapUsed":17853520,"rss":65683456},"pid":1,"uid":0,"version":"v16.14.2"},"service":"democratic-csi","stack":"UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason \"#<Object>\".","timestamp":"2022-05-10T07:45:04.020Z","trace":[]}
running server shutdown, exit code: UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "#<Object>".
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/ad05d574-c544-4cbc-ab0c-ca8f0a274f63/volumes/kubernetes.io~csi/pvc-9c0f8520-79d7-4442-8675-44ad595777bb/mount --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/ad05d574-c544-4cbc-ab0c-ca8f0a274f63/volumes/kubernetes.io~csi/pvc-9c0f8520-79d7-4442-8675-44ad595777bb/mount --output source,target,fstype,label,options,avail,size,used -b -J --nofsroot
{"host":"k3os-21898","level":"info","message":"new response - driver: FreeNASSshDriver method: NodeGetVolumeStats response: {\"usage\":[{\"available\":7333806080,\"total\":10726932480,\"used\":3393126400,\"unit\":\"BYTES\"}]}","service":"democratic-csi","timestamp":"2022-05-10T07:45:04.038Z"}
grpc server gracefully closed all connections
server fully shutdown, exiting

iSCSI host: OS: TrueNAS-SCALE-22.02.1

K8s controller/node: OS: k3OS v0.21.5-k3s2r1

Democratic CSI config:

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: democratic-csi-iscsi
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/democratic-csi-0.11.2.tgz
  valuesContent: |-
    csiDriver:
      name: "org.democratic-csi.iscsi"
    storageClasses:
    - name: iscsi-ext4
      defaultClass: false
      reclaimPolicy: Delete
      volumeBindingMode: Immediate
      allowVolumeExpansion: true
      parameters:
        fsType: ext4
    - name: iscsi-xfs
      defaultClass: false
      reclaimPolicy: Delete
      volumeBindingMode: Immediate
      allowVolumeExpansion: true
      parameters:
        fsType: xfs
    controller:
      rbac:
        enabled: true
      enabled: true
      driver:
        logLevel: debug
    node:
      driver:
        logLevel: silly
        localtimeHostPath: null
    driver:
      config:
        driver: freenas-iscsi
        httpConnection:
          protocol: https
          host: freenas.desdichado.se
          port: 443
          username: k8s-csi
          apiKey: ***REDACTED***
          allowInsecure: false
          apiVersion: 2
        sshConnection:
          host: 192.168.2.12
          port: 22
          username: k8s-csi
          privateKey: ***REDACTED***
        zfs:
          cli:
            sudoEnabled: true
            paths:
              zfs: /sbin/zfs
              zpool: /sbin/zpool
              sudo: /bin/sudo
              chroot: /sbin/chroot
          datasetParentName: tank2/k8s/iscsi/v
          detachedSnapshotsDatasetParentName: tank2/k8s/iscsi/s
        iscsi:
          targetPortal: "192.168.2.12:3260"
          targetPortals: []
          # leave empty to omit usage of -I with iscsiadm
          interface:
          namePrefix: csi-
          nameSuffix: "-cluster"
          targetGroups:
            - targetGroupPortalGroup: 1
              targetGroupInitiatorGroup: 1
              targetGroupAuthType: None
              targetGroupAuthGroup:
          extentInsecureTpc: true
          extentXenCompat: false
          extentDisablePhysicalBlocksize: true
          # 512, 1024, 2048, or 4096,
          extentBlocksize: 4096
          extentRpm: "7200"
          extentAvailThreshold: 0