iscsiadm binary incompatibility with Flatcar Linux

Recently when conducting an upgrade to democratic-csi v1.1.0 persistent volumes began failing to mount on Flatcar Linux.

Worker node log:

Mar 18 12:10:50 <censored> kernel: iscsiadm[5039]: segfault at 55c2dcf89aa9 ip 000055c77fd43530 sp 00007fff3bd75590 error 4 in iscsiadm[55c77fd42000+3d000]
Mar 18 12:10:50 <censored> kernel: Code: 43 08 48 89 18 e8 a0 ef ff ff 48 89 df 48 8b 1b 48 39 ef 75 e5 48 8b 9c 24 10 01 00 00 48 8b 6c 24 18 48 3b 5c 24 18 74 2e 90 <48> 8b 13 48 8b 43 08 48 89 df 48 89 42 08 48 89 10 48 c7 43 08 00

CSI Driver log:

executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-pvc-7fea9dc7-0b9e-4311-b662-9c34278fab80-clustera -p <censored>:3260 -o update --name node.startup --value manual
error: handler error - driver: FreeNASDriver method: NodeStageVolume error: {"code":139,"stdout":"","stderr":"/usr/local/sbin/iscsiadm: line 5:  1230 Segmentation fault      (core dumped) chroot /host /usr/bin/env -i PATH=\"/sbin:/bin:/usr/bin\" iscsiadm \"${@:1}\"\n"}
{
  code: 13,
  message: '{"code":139,"stdout":"","stderr":"/usr/local/sbin/iscsiadm: line 5:  1230 Segmentation fault      (core dumped) chroot /host /usr/bin/env -i PATH=\\"/sbin:/bin:/usr/bin\\" iscsiadm \\"${@:1}\\"\\n"}'
}

There have been changes to iscsiadm recently on Flatcar Linux, as well as various other reports of this segfault. Refer: https://www.suse.com/support/kb/doc/?id=000019883

To test the theory on binary incompatibility, I checked iscsiadm being used by CSI driver and see that the version is packaged from debian. Using kubectl cp I overrode the iscsiadm in all the csi-driver pods with the iscsiadm from Flatcar Linux, as well as the supporting libraries. At that point, mounting of iSCSI targets worked seamlessly.

[mfw78]

FYI this has been fixed with the closure of https://github.com/kinvolk/Flatcar/issues/373.

[mfw78]

Sorry, other work came up. I’ve got this scheduled to attend to in the next day or so.