Helm install with freenas-iscsi failed as democratic-csi-controller pod CrashLoopBackOff

Issue Description

I setup a TrueNAS server, and try to install democratic-csi with freenas-iscis driver via helm. But the democratic-csi-controller pod always faild to run as “Error: All configured authentication methods failed”.

❯ kubectl get po
NAME                                                       READY   STATUS             RESTARTS       AGE
freenas-iscsi-democratic-csi-controller-75bf57c78b-4lh48   2/5     CrashLoopBackOff   30 (15s ago)   18m
freenas-iscsi-democratic-csi-node-cmb72                    4/4     Running            0              18m
freenas-iscsi-democratic-csi-node-njpzf                    4/4     Running            0              18m
freenas-iscsi-democratic-csi-node-sz25l                    4/4     Running            0              18m

I0601 05:53:13.476174       1 common.go:111] Probing CSI driver for readiness
I0601 05:53:13.476202       1 connection.go:183] GRPC call: /csi.v1.Identity/Probe
I0601 05:53:13.476207       1 connection.go:184] GRPC request: {}
I0601 05:53:13.517234       1 connection.go:186] GRPC response: {}
I0601 05:53:13.517305       1 connection.go:187] GRPC error: rpc error: code = Internal desc = Error: All configured authentication methods failed Error: All configured authentication methods failed
    at doNextAuth (/home/csi/app/node_modules/ssh2/lib/client.js:803:21)
    at tryNextAuth (/home/csi/app/node_modules/ssh2/lib/client.js:993:7)
    at USERAUTH_FAILURE (/home/csi/app/node_modules/ssh2/lib/client.js:373:11)
    at 51 (/home/csi/app/node_modules/ssh2/lib/protocol/handlers.misc.js:337:16)
    at Protocol.onPayload (/home/csi/app/node_modules/ssh2/lib/protocol/Protocol.js:2025:10)

below is my freenas-iscsi.yaml values file:

csiDriver:
  # should be globally unique for a given cluster
  name: "org.democratic-csi.iscsi"

# add note here about volume expansion requirements
storageClasses:
- name: freenas-iscsi-csi
  defaultClass: false
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
  allowVolumeExpansion: true
  parameters:
    # for block-based storage can be ext3, ext4, xfs
    fsType: xfs

  mountOptions: []
  secrets:
    provisioner-secret:
    controller-publish-secret:
    node-stage-secret:
    node-publish-secret:
    controller-expand-secret:

driver:
  config:
    driver: freenas-iscsi
    instance_id:
    httpConnection:
      protocol: http
      host: 192.168.0.130
      port: 80
      # use only 1 of apiKey or username/password
      # if both are present, apiKey is preferred
      # apiKey is only available starting in TrueNAS-12
      apiKey: <apikey>
      username: root
      #password:
      allowInsecure: true
      # use apiVersion 2 for TrueNAS-12 and up (will work on 11.x in some scenarios as well)
      # leave unset for auto-detection
      apiVersion: 2
    sshConnection:
      host: 192.168.0.130
      port: 22
      username: root
      # use either password or key
      #password: ""
      privateKey: |
        -----BEGIN OPENSSH PRIVATE KEY-----
        <myprivatekey>
        -----END OPENSSH PRIVATE KEY-----
    zfs:
      # can be used to override defaults if necessary
      # the example below is useful for TrueNAS 12
      cli:
      #  sudoEnabled: true
      #
      #  leave paths unset for auto-detection
        paths:
          zfs: /usr/local/sbin/zfs
          zpool: /usr/local/sbin/zpool
          sudo: /usr/local/bin/sudo
          chroot: /usr/sbin/chroot
    
      # can be used to set arbitrary values on the dataset/zvol
      # can use handlebars templates with the parameters from the storage class/CO
      #datasetProperties:
      #  "org.freenas:description": "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}/{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
      #  "org.freenas:test": "{{ parameters.foo }}"
      #  "org.freenas:test2": "some value"
      
      # total volume name (zvol/<datasetParentName>/<pvc name>) length cannot exceed 63 chars
      # https://www.ixsystems.com/documentation/freenas/11.2-U5/storage.html#zfs-zvol-config-opts-tab
      # standard volume naming overhead is 46 chars
      # datasetParentName should therefore be 17 chars or less when using TrueNAS 12 or below
      datasetParentName: iSCSIPool/k8s/vols
      # do NOT make datasetParentName and detachedSnapshotsDatasetParentName overlap
      # they may be siblings, but neither should be nested in the other 
      detachedSnapshotsDatasetParentName: iSCSIPool/k8s/snaps
      # "" (inherit), lz4, gzip-9, etc
      zvolCompression:
      # "" (inherit), on, off, verify
      zvolDedup:
      zvolEnableReservation: false
      # 512, 1K, 2K, 4K, 8K, 16K, 64K, 128K default is 16K
      zvolBlocksize:
    iscsi:
      targetPortal: "192.168.0.130:3260"
      # for multipath
      targetPortals: [] # [ "server[:port]", "server[:port]", ... ]
      # leave empty to omit usage of -I with iscsiadm
      interface:
    
      # MUST ensure uniqueness
      # full iqn limit is 223 bytes, plan accordingly
      # default is "{{ name }}"
      #nameTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}-{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
      namePrefix: csi-
      nameSuffix: "-clustera"
    
      # add as many as needed
      targetGroups:
        # get the correct ID from the "portal" section in the UI
        - targetGroupPortalGroup: 1
          # get the correct ID from the "initiators" section in the UI
          targetGroupInitiatorGroup: 1
          # None, CHAP, or CHAP Mutual
          targetGroupAuthType: None
          # get the correct ID from the "Authorized Access" section of the UI
          # only required if using Chap
          #targetGroupAuthGroup:
    
      #extentCommentTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}/{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
      extentInsecureTpc: true
      extentXenCompat: false
      extentDisablePhysicalBlocksize: true
      # 512, 1024, 2048, or 4096,
      extentBlocksize: 512
      # "" (let FreeNAS decide, currently defaults to SSD), Unknown, SSD, 5400, 7200, 10000, 15000
      extentRpm: ""
      # 0-100 (0 == ignore)
      extentAvailThreshold: 0

Versions

k8s:

Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:04:34Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

helm: version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.18.2"}

TrueNAS:
TrueNAS-13.0-RELEASE

[yidaqiang]

See this new: https://www.zdnet.com/article/openssh-to-deprecate-sha-1-logins-due-to-security-risk/ openssh No longer supported ssh-rsa by default. So I used ed25519, everything is going smoothly.

# ssh-keygen -t ed25519 -C [e-mail] -f truenas_rsa

# ls
truenas_rsa truenas_rsa.pub

# cat truenas_rsa
-----BEGIN OPENSSH PRIVATE KEY-----
xxx
...
xx==
-----END OPENSSH PRIVATE KEY-----

update your helm values freenas-iscsi.yaml with ssh private key truenas_rsa

    sshConnection:
      host: 192.168.0.130
      port: 22
      username: root
      # use either password or key
      #password: ""
      privateKey: |
        -----BEGIN OPENSSH PRIVATE KEY-----
        xxx
        ...
        xx==
        -----END OPENSSH PRIVATE KEY-----

[mengzyou]

The PubKeyAuth faild as userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms.
So fixed it by adding PubkeyAcceptedKeyTypes=+ssh-rsa to sshd_config on TrueNAS server.

@travisghansen Thanks for your support, can close this issue now.