ERR_UNHANDLED_REJECTION exception after TrueNAS reboot.

I had a situation where I had to emergency power down my TrueNAS 12.0-U8.1 server with the Kubernetes cluster still running.

Upon TrueNAS restart it seems 2 of my 3 nodes recovered OK. But the third had the CSI iSCSI driver stuck in a crash loop. I had to drain the node and reboot it. Upon reboot and uncordon everything seemed ok and CSI iSCSI restarted fine. Using Chart 0.13.1.

Not sure if this captured anything useful:

driver logs:

grpc implementation: @grpc/grpc-js
{"host":"k3s01","level":"info","message":"initializing csi driver: freenas-iscsi","service":"democratic-csi","timestamp":"2022-06-23T01:55:49.943Z"}
{"host":"k3s01","level":"info","message":"starting csi server - name: org.democratic-csi.iscsi, version: 1.6.3, driver: freenas-iscsi, mode: node, csi version: 1.5.0, address: , socket: unix:///csi-data/csi.sock.internal","service":"democratic-csi","timestamp":"2022-06-23T01:55:50.094Z"}
{"host":"k3s01","level":"info","message":"new request - driver: FreeNASSshDriver method: Probe call: {\"_events\":{},\"_eventsCount\":0,\"call\":{\"_events\":{},\"_eventsCount\":5,\"stream\":{\"_readableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"buffer\":{\"head\":null,\"tail\":null,\"length\":0},\"length\":0,\"pipes\":[],\"flowing\":true,\"ended\":true,\"endEmitted\":true,\"reading\":false,\"constructed\":true,\"sync\":false,\"needReadable\":false,\"emittedReadable\":false,\"readableListening\":false,\"resumeScheduled\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"destroyed\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false,\"defaultEncoding\":\"utf8\",\"awaitDrainWriters\":null,\"multiAwaitDrain\":false,\"readingMore\":false,\"dataEmitted\":true,\"decoder\":null,\"encoding\":null},\"_events\":{},\"_eventsCount\":8,\"_writableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"finalCalled\":false,\"needDrain\":false,\"ending\":false,\"ended\":false,\"finished\":false,\"destroyed\":false,\"decodeStrings\":false,\"defaultEncoding\":\"utf8\",\"length\":0,\"writing\":false,\"corked\":0,\"sync\":true,\"bufferProcessing\":false,\"writecb\":null,\"writelen\":0,\"afterWriteTickInfo\":null,\"buffered\":[],\"bufferedIndex\":0,\"allBuffers\":true,\"allNoop\":true,\"pendingcb\":0,\"constructed\":true,\"prefinished\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false},\"allowHalfOpen\":true},\"handler\":{\"type\":\"unary\",\"path\":\"/csi.v1.Identity/Probe\"},\"options\":{},\"cancelled\":false,\"deadlineTimer\":{\"_idleTimeout\":-1,\"_idlePrev\":null,\"_idleNext\":null,\"_idleStart\":16827,\"_onTimeout\":null,\"_repeat\":null,\"_destroyed\":true},\"deadline\":null,\"wantTrailers\":false,\"metadataSent\":false,\"canPush\":false,\"isPushPending\":false,\"bufferedMessages\":[],\"messagesToPush\":[],\"maxSendMessageSize\":-1,\"maxReceiveMessageSize\":4194304},\"metadata\":{\"user-agent\":[\"grpc-node-js/1.6.2\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-06-23T01:56:06.262Z"}
{"host":"k3s01","level":"info","message":"new response - driver: FreeNASSshDriver method: Probe response: {\"ready\":{\"value\":true}}","service":"democratic-csi","timestamp":"2022-06-23T01:56:06.263Z"}
{"host":"k3s01","level":"info","message":"new request - driver: FreeNASSshDriver method: NodeGetCapabilities call: {\"_events\":{},\"_eventsCount\":0,\"call\":{\"_events\":{},\"_eventsCount\":5,\"stream\":{\"_readableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"buffer\":{\"head\":null,\"tail\":null,\"length\":0},\"length\":0,\"pipes\":[],\"flowing\":true,\"ended\":true,\"endEmitted\":true,\"reading\":false,\"constructed\":true,\"sync\":false,\"needReadable\":false,\"emittedReadable\":false,\"readableListening\":false,\"resumeScheduled\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"destroyed\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false,\"defaultEncoding\":\"utf8\",\"awaitDrainWriters\":null,\"multiAwaitDrain\":false,\"readingMore\":false,\"dataEmitted\":true,\"decoder\":null,\"encoding\":null},\"_events\":{},\"_eventsCount\":8,\"_writableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"finalCalled\":false,\"needDrain\":false,\"ending\":false,\"ended\":false,\"finished\":false,\"destroyed\":false,\"decodeStrings\":false,\"defaultEncoding\":\"utf8\",\"length\":0,\"writing\":false,\"corked\":0,\"sync\":true,\"bufferProcessing\":false,\"writecb\":null,\"writelen\":0,\"afterWriteTickInfo\":null,\"buffered\":[],\"bufferedIndex\":0,\"allBuffers\":true,\"allNoop\":true,\"pendingcb\":0,\"constructed\":true,\"prefinished\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false},\"allowHalfOpen\":true},\"handler\":{\"type\":\"unary\",\"path\":\"/csi.v1.Node/NodeGetCapabilities\"},\"options\":{},\"cancelled\":false,\"deadlineTimer\":{\"_idleTimeout\":120000,\"_idlePrev\":{\"expiry\":183827,\"id\":-9007199254740988,\"msecs\":120000,\"priorityQueuePosition\":1},\"_idleStart\":63827,\"_timerArgs\":[null],\"_repeat\":null,\"_destroyed\":false},\"deadline\":1655949533253,\"wantTrailers\":false,\"metadataSent\":false,\"canPush\":false,\"isPushPending\":false,\"bufferedMessages\":[],\"messagesToPush\":[],\"maxSendMessageSize\":-1,\"maxReceiveMessageSize\":4194304},\"metadata\":{\"x-forwarded-host\":[\"/var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock\"],\"user-agent\":[\"grpc-go/1.40.0\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-06-23T01:56:53.255Z"}
{"host":"k3s01","level":"info","message":"new response - driver: FreeNASSshDriver method: NodeGetCapabilities response: {\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}},{\"rpc\":{\"type\":\"GET_VOLUME_STATS\"}},{\"rpc\":{\"type\":\"EXPAND_VOLUME\"}},{\"rpc\":{\"type\":\"SINGLE_NODE_MULTI_WRITER\"}}]}","service":"democratic-csi","timestamp":"2022-06-23T01:56:53.256Z"}
{"host":"k3s01","level":"info","message":"new request - driver: FreeNASSshDriver method: NodeGetVolumeStats call: {\"_events\":{},\"_eventsCount\":0,\"call\":{\"_events\":{},\"_eventsCount\":5,\"stream\":{\"_readableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"buffer\":{\"head\":null,\"tail\":null,\"length\":0},\"length\":0,\"pipes\":[],\"flowing\":true,\"ended\":true,\"endEmitted\":true,\"reading\":false,\"constructed\":true,\"sync\":false,\"needReadable\":false,\"emittedReadable\":false,\"readableListening\":false,\"resumeScheduled\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"destroyed\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false,\"defaultEncoding\":\"utf8\",\"awaitDrainWriters\":null,\"multiAwaitDrain\":false,\"readingMore\":false,\"dataEmitted\":true,\"decoder\":null,\"encoding\":null},\"_events\":{},\"_eventsCount\":8,\"_writableState\":{\"objectMode\":false,\"highWaterMark\":16384,\"finalCalled\":false,\"needDrain\":false,\"ending\":false,\"ended\":false,\"finished\":false,\"destroyed\":false,\"decodeStrings\":false,\"defaultEncoding\":\"utf8\",\"length\":0,\"writing\":false,\"corked\":0,\"sync\":true,\"bufferProcessing\":false,\"writecb\":null,\"writelen\":0,\"afterWriteTickInfo\":null,\"buffered\":[],\"bufferedIndex\":0,\"allBuffers\":true,\"allNoop\":true,\"pendingcb\":0,\"constructed\":true,\"prefinished\":false,\"errorEmitted\":false,\"emitClose\":true,\"autoDestroy\":false,\"errored\":null,\"closed\":false,\"closeEmitted\":false},\"allowHalfOpen\":true},\"handler\":{\"type\":\"unary\",\"path\":\"/csi.v1.Node/NodeGetVolumeStats\"},\"options\":{},\"cancelled\":false,\"deadlineTimer\":{\"_idleTimeout\":119990,\"_idlePrev\":{\"expiry\":183826,\"id\":-9007199254740985,\"msecs\":119990,\"priorityQueuePosition\":2},\"_idleStart\":63836,\"_timerArgs\":[null],\"_repeat\":null,\"_destroyed\":false},\"deadline\":1655949533252,\"wantTrailers\":false,\"metadataSent\":false,\"canPush\":false,\"isPushPending\":false,\"bufferedMessages\":[],\"messagesToPush\":[],\"maxSendMessageSize\":-1,\"maxReceiveMessageSize\":4194304},\"metadata\":{\"x-forwarded-host\":[\"/var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock\"],\"user-agent\":[\"grpc-go/1.40.0\"]},\"request\":{\"volume_id\":\"pvc-9604a6ff-4354-4f16-943d-c2b6af6245c5\",\"volume_path\":\"/var/lib/kubelet/pods/d319fc9b-b307-4082-9137-6f20f511a535/volumes/kubernetes.io~csi/pvc-9604a6ff-4354-4f16-943d-c2b6af6245c5/mount\",\"staging_target_path\":\"\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-06-23T01:56:53.265Z"}
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/d319fc9b-b307-4082-9137-6f20f511a535/volumes/kubernetes.io~csi/pvc-9604a6ff-4354-4f16-943d-c2b6af6245c5/mount --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/d319fc9b-b307-4082-9137-6f20f511a535/volumes/kubernetes.io~csi/pvc-9604a6ff-4354-4f16-943d-c2b6af6245c5/mount --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /var/lib/kubelet/pods/d319fc9b-b307-4082-9137-6f20f511a535/volumes/kubernetes.io~csi/pvc-9604a6ff-4354-4f16-943d-c2b6af6245c5/mount
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/d319fc9b-b307-4082-9137-6f20f511a535/volumes/kubernetes.io~csi/pvc-9604a6ff-4354-4f16-943d-c2b6af6245c5/mount/block_device --output source,target,fstype,label,options -b -J --nofsroot
failed to execute filesystem command: realpath /var/lib/kubelet/pods/d319fc9b-b307-4082-9137-6f20f511a535/volumes/kubernetes.io~csi/pvc-9604a6ff-4354-4f16-943d-c2b6af6245c5/mount, response: {"code":1,"stdout":"","stderr":"realpath: /var/lib/kubelet/pods/d319fc9b-b307-4082-9137-6f20f511a535/volumes/kubernetes.io~csi/pvc-9604a6ff-4354-4f16-943d-c2b6af6245c5/mount: Input/output error\n","timeout":false}
{"date":"Thu Jun 23 2022 01:56:53 GMT+0000 (Coordinated Universal Time)","error":{"code":"ERR_UNHANDLED_REJECTION"},"exception":true,"host":"k3s01","level":"error","message":"uncaughtException: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason \"#<Object>\".\nUnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason \"#<Object>\".","os":{"loadavg":[0.15,0.53,0.66],"uptime":537921.56},"process":{"argv":["/usr/local/bin/node","/home/csi/app/bin/democratic-csi","--csi-version=1.5.0","--csi-name=org.democratic-csi.iscsi","--driver-config-file=/config/driver-config-file.yaml","--log-level=info","--csi-mode=node","--server-socket=/csi-data/csi.sock.internal"],"cwd":"/home/csi/app","execPath":"/usr/local/bin/node","gid":0,"memoryUsage":{"arrayBuffers":93649,"external":17906316,"heapTotal":18808832,"heapUsed":17579808,"rss":49135616},"pid":1,"uid":0,"version":"v16.14.2"},"service":"democratic-csi","stack":"UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason \"#<Object>\".","timestamp":"2022-06-23T01:56:53.318Z","trace":[]}
running server shutdown, exit code: UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "#<Object>".
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/d319fc9b-b307-4082-9137-6f20f511a535/volumes/kubernetes.io~csi/pvc-9604a6ff-4354-4f16-943d-c2b6af6245c5/mount --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /var/lib/kubelet/pods/d319fc9b-b307-4082-9137-6f20f511a535/volumes/kubernetes.io~csi/pvc-9604a6ff-4354-4f16-943d-c2b6af6245c5/mount --output source,target,fstype,label,options,avail,size,used -b -J --nofsroot
{"host":"k3s01","level":"info","message":"new response - driver: FreeNASSshDriver method: NodeGetVolumeStats response: {\"usage\":[{\"available\":3154452480,\"total\":3210739712,\"used\":56287232,\"unit\":\"BYTES\"}]}","service":"democratic-csi","timestamp":"2022-06-23T01:56:53.340Z"}
grpc server gracefully closed all connections
server fully shutdown, exiting

Proxy Logs:

2022/06/23 01:52:53 httputil: ReverseProxy read error during body copy: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""
2022/06/23 01:52:56 request (:///var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock) POST /csi.v1.Node/NodeGetCapabilities HTTP/2.0
2022/06/23 01:52:56 request headers map[Content-Type:[application/grpc] Grpc-Timeout:[120000m] Te:[trailers] User-Agent:[grpc-go/1.40.0]]
2022/06/23 01:52:56 Dialing upstream: unix:///csi-data/csi.sock.internal
2022/06/23 01:52:56 http: proxy error: dial unix /csi-data/csi.sock.internal: connect: no such file or directory
2022/06/23 01:54:02 request (:///var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock) POST /csi.v1.Node/NodeGetCapabilities HTTP/2.0
2022/06/23 01:54:02 request headers map[Content-Type:[application/grpc] Grpc-Timeout:[120000m] Te:[trailers] User-Agent:[grpc-go/1.40.0]]
2022/06/23 01:54:02 Dialing upstream: unix:///csi-data/csi.sock.internal
2022/06/23 01:54:02 request (:///var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock) POST /csi.v1.Node/NodeGetVolumeStats HTTP/2.0
2022/06/23 01:54:02 request headers map[Content-Type:[application/grpc] Grpc-Timeout:[119966m] Te:[trailers] User-Agent:[grpc-go/1.40.0]]
2022/06/23 01:54:02 httputil: ReverseProxy read error during body copy: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""
2022/06/23 01:54:02 request (:///var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock) POST /csi.v1.Node/NodeGetCapabilities HTTP/2.0
2022/06/23 01:54:02 request headers map[Content-Type:[application/grpc] Grpc-Timeout:[120000m] Te:[trailers] User-Agent:[grpc-go/1.40.0]]
2022/06/23 01:54:02 Dialing upstream: unix:///csi-data/csi.sock.internal
2022/06/23 01:54:02 http: proxy error: dial unix /csi-data/csi.sock.internal: connect: no such file or directory
2022/06/23 01:54:13 request (:///var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock) POST /csi.v1.Node/NodeGetCapabilities HTTP/2.0
2022/06/23 01:54:13 request headers map[Content-Type:[application/grpc] Grpc-Timeout:[120000m] Te:[trailers] User-Agent:[grpc-go/1.40.0]]
2022/06/23 01:54:13 Dialing upstream: unix:///csi-data/csi.sock.internal
2022/06/23 01:54:13 http: proxy error: dial unix /csi-data/csi.sock.internal: connect: no such file or directory
2022/06/23 01:55:06 request (:///var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock) POST /csi.v1.Node/NodeGetCapabilities HTTP/2.0
2022/06/23 01:55:06 request headers map[Content-Type:[application/grpc] Grpc-Timeout:[120000m] Te:[trailers] User-Agent:[grpc-go/1.40.0]]
2022/06/23 01:55:06 Dialing upstream: unix:///csi-data/csi.sock.internal
2022/06/23 01:55:06 request (:///var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock) POST /csi.v1.Node/NodeGetVolumeStats HTTP/2.0
2022/06/23 01:55:06 request headers map[Content-Type:[application/grpc] Grpc-Timeout:[119967m] Te:[trailers] User-Agent:[grpc-go/1.40.0]]
2022/06/23 01:55:06 httputil: ReverseProxy read error during body copy: http2: server sent GOAWAY and closed the connection; LastStreamID=5, ErrCode=NO_ERROR, debug=""
2022/06/23 01:55:33 request (:///var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock) POST /csi.v1.Node/NodeGetCapabilities HTTP/2.0
2022/06/23 01:55:33 request headers map[Content-Type:[application/grpc] Grpc-Timeout:[120000m] Te:[trailers] User-Agent:[grpc-go/1.40.0]]
2022/06/23 01:55:33 Dialing upstream: unix:///csi-data/csi.sock.internal
2022/06/23 01:55:33 http: proxy error: dial unix /csi-data/csi.sock.internal: connect: no such file or directory
2022/06/23 01:55:46 request (:///var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock) POST /csi.v1.Node/NodeGetCapabilities HTTP/2.0
2022/06/23 01:55:46 request headers map[Content-Type:[application/grpc] Grpc-Timeout:[120000m] Te:[trailers] User-Agent:[grpc-go/1.40.0]]
2022/06/23 01:55:46 Dialing upstream: unix:///csi-data/csi.sock.internal
2022/06/23 01:55:46 http: proxy error: dial unix /csi-data/csi.sock.internal: connect: no such file or directory

[travisghansen]

Was actually off grid for a few days or I would have responded earlier!

[reefland]

I just bumped to v1.7.1 and let ArgoCD do its magic. Everything seems fine. I’ll monitor things for a few days. Appreciate the fast responses!