Failure to mount volume provisioned in Truenas via iscsi driver

Hi @travisghansen,

I’m facing some strange behaviour in my cluster and can’t find whats wrong, but it feels like the code is not reading configuration correctly. If I’m creating PVC via helm chart (democratic-csi-0.8.1 tried with latest and v1.2.0 tags of the image) and mounting to pod, the creation of volumes succeeds but mounting fails. My setup is as follows:

...
iscsi:
  ...
  targetPortal: "192.168.1.200:3260"
  targetPortals:
  - "192.168.1.200"
  ...
...

which creates in k8s secret following section:

iscsi:
  extentAvailThreshold: 0
  extentBlocksize: 512
  extentDisablePhysicalBlocksize: true
  extentInsecureTpc: true
  extentRpm: SSD
  extentXenCompat: false
  namePrefix: k3s-
  nameSuffix: ""
  targetGroups:
  - targetGroupAuthGroup: null
    targetGroupAuthType: None
    targetGroupInitiatorGroup: 1
    targetGroupPortalGroup: 1
  targetPortal: 192.168.1.200:3260
  targetPortals:
  - 192.168.1.200

the message representing the config:

democratic-csi-node-pfcrj csi-driver
{
  "service":"democratic-csi",
    "level":"info",
    "message":"new request - driver: FreeNASDriver method: NodeStageVolume call: {
      "_events":{},
      "_eventsCount":1,
      "call":{},
      "cancelled":false,
      "metadata":{
        "_internal_repr":{
          "user-agent":["grpc-go/1.27.1"]},
        "flags":0},
      "request":{
        "publish_context":{},
        "secrets":"redacted",
        "volume_context":{
          "provisioner_driver":"freenas-iscsi",
          "storage.kubernetes.io/csiProvisionerIdentity":"1623515276448-8081-org.democratic-csi.iscsi",
          "interface":"",
          "iqn":"iqn.2005-10.org.freenas.ctl:k3s-pvc-3b6de84d-0579-47d0-8311-6489f53c9623",
          "lun":"0",
          "node_attach_driver":"iscsi",
          "portal":"undefined",
          "portals":"192.168.1.200"},
        "volume_id":"pvc-3b6de84d-0579-47d0-8311-6489f53c9623",
        "staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3b6de84d-0579-47d0-8311-6489f53c9623/globalmount",
        "volume_capability":{
          "access_mode":{
            "mode":"SINGLE_NODE_WRITER"},
          "mount":{
            "mount_flags":[],
            "fs_type":"ext4"},
          "access_type":"mount"}}}"}

and then when trying to mount volume it shows following messages:

democratic-csi-node-pfcrj csi-driver {"service":"democratic-csi","level":"debug","message":"operation lock keys: [\"volume_id_pvc-3b6de84d-0579-47d0-8311-6489f53c9623\"]"}
democratic-csi-node-pfcrj csi-driver {"service":"democratic-csi","level":"verbose","message":"validating capabilities: [{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"defaults\"],\"fs_type\":\"ext4\"},\"access_type\":\"mount\"}]"}
democratic-csi-node-pfcrj csi-driver executing fileystem command: stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3b6de84d-0579-47d0-8311-6489f53c9623/globalmount
democratic-csi-node-pfcrj csi-driver executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:k3s-pvc-3b6de84d-0579-47d0-8311-6489f53c9623 -p undefined:3260 -o new
democratic-csi-node-pfcrj csi-driver executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:k3s-pvc-3b6de84d-0579-47d0-8311-6489f53c9623 -p undefined:3260 -o update --name node.startup --value manual
democratic-csi-node-pfcrj csi-driver executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:k3s-pvc-3b6de84d-0579-47d0-8311-6489f53c9623 -p undefined:3260 -l
democratic-csi-node-pfcrj csi-driver {"service":"democratic-csi","level":"error","message":"handler error - driver: FreeNASDriver method: NodeStageVolume error: {\"code\":4,\"stdout\":\"Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:k3s-pvc-3b6de84d-0579-47d0-8311-6489f53c9623, portal: undefined,3260] (multiple)\\n\",\"stderr\":\"iscsiadm: Could not login to [iface: default, target: iqn.2005-10.org.freenas.ctl:k3s-pvc-3b6de84d-0579-47d0-8311-6489f53c9623, portal: undefined,3260].\\niscsiadm: initiator reported error (4 - encountered connection failure)\\niscsiadm: Could not log into all portals\\n\"}"}
democratic-csi-node-pfcrj csi-driver {
democratic-csi-node-pfcrj csi-driver   code: 13,
democratic-csi-node-pfcrj csi-driver   message: '{"code":4,"stdout":"Logging in to [iface: default, target: iqn.2005-10.org.freenas.ctl:k3s-pvc-3b6de84d-0579-47d0-8311-6489f53c9623, portal: undefined,3260] (multiple)\\n","stderr":"iscsiadm: Could not login to [iface: default, target: iqn.2005-10.org.freenas.ctl:k3s-pvc-3b6de84d-0579-47d0-8311-6489f53c9623, portal: undefined,3260].\\niscsiadm: initiator reported error (4 - encountered connection failure)\\niscsiadm: Could not log into all portals\\n"}'
democratic-csi-node-pfcrj csi-driver }

where there are 3 sequential commands it tries to run and in all it uses undefined:3260 as IP:port adress of the target server:

iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:k3s-pvc-87dff94e-5801-4100-8f64-e0199af0426a -p undefined:3260 -o new
iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:k3s-pvc-87dff94e-5801-4100-8f64-e0199af0426a -p undefined:3260 -o update --name node.startup --value manual
iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:k3s-pvc-87dff94e-5801-4100-8f64-e0199af0426a -p undefined:3260 -l

and it clearly fails on last command.

But if I’m trying to run these commands from any node on the cluster and pointing to the correct IP address/port, these commands succeed. I also tried various combinations of targetPortal and targetPortals, where one is set and another is not and vice versa - where both are set, tried to open multiple ports and configure these - all without any luck.

It feels like initial assignment happens here:

https://github.com/democratic-csi/democratic-csi/blob/master/src/driver/freenas/index.js#L1251

and then a bit processed here:

https://github.com/democratic-csi/democratic-csi/blob/master/src/driver/index.js#L324

I’ve also tried to enable debugging in both controller and node, but there is very little log messages in the code area of reading and processing configuration, can’t see anything usefull.

May be I’m configuring something wrong, not really sure, any suggestions ?

[travisghansen]

Fix for missing targetPortals has landed in v1.3.0.

[isindir]

I think it is a bug, but to be sure I need to try it again from scratch. From my first iteration over weekends I believe it is the only change in place right now.