[zol] [nfs] Unsupported key format for all keys tried

I’m continuously getting the following error from the controller pod no matter what keys I attempt to use in the helm chart:

❯ kl -n storage-system democratic-csi-nfs-controller-66c778c796-ql6wj csi-driver
grpc implementation: @grpc/grpc-js
{"host":"democratic-csi-nfs-controller-66c778c796-ql6wj","level":"info","message":"initializing csi driver: zfs-generic-nfs","service":"democratic-csi","timestamp":"2022-08-16T12:22:29.058Z"}
{"host":"democratic-csi-nfs-controller-66c778c796-ql6wj","level":"info","message":"starting csi server - node version: v16.16.0, package version: 1.7.6, config file: /config/..2022_08_16_12_22_20.950438490/driver-config-file.yaml, csi-name: org.democratic-csi.nfs, csi-driver: zfs-generic-nfs, csi-mode: controller, csi-version: 1.5.0, address: , socket: unix:///csi-data/csi.sock.internal","service":"democratic-csi","timestamp":"2022-08-16T12:22:29.762Z"}
{"host":"democratic-csi-nfs-controller-66c778c796-ql6wj","level":"info","message":"new request - driver: ControllerZfsGenericDriver method: Probe call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.40.0\"],\"x-forwarded-host\":[\"localhost\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-08-16T12:22:41.734Z"}
{"date":"Tue Aug 16 2022 12:22:41 GMT+0000 (Coordinated Universal Time)","error":{},"exception":true,"host":"democratic-csi-nfs-controller-66c778c796-ql6wj","level":"error","message":"uncaughtException: Cannot parse privateKey: Unsupported key format\nError: Cannot parse privateKey: Unsupported key format\n    at Client.connect (/home/csi/app/node_modules/ssh2/lib/client.js:261:15)\n    at /home/csi/app/src/utils/ssh.js:78:19\n    at /home/csi/app/node_modules/async-mutex/lib/Mutex.js:23:66\n    at Semaphore.<anonymous> (/home/csi/app/node_modules/async-mutex/lib/Semaphore.js:38:46)\n    at step (/home/csi/app/node_modules/async-mutex/node_modules/tslib/tslib.js:144:27)\n    at Object.next (/home/csi/app/node_modules/async-mutex/node_modules/tslib/tslib.js:125:57)\n    at fulfilled (/home/csi/app/node_modules/async-mutex/node_modules/tslib/tslib.js:115:62)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","os":{"loadavg":[0.7,0.37,0.28],"uptime":39520.3},"process":{"argv":["/usr/local/bin/node","/home/csi/app/bin/democratic-csi","--csi-version=1.5.0","--csi-name=org.democratic-csi.nfs","--driver-config-file=/config/driver-config-file.yaml","--log-level=info","--csi-mode=controller","--server-socket=/csi-data/csi.sock.internal"],"cwd":"/home/csi/app","execPath":"/usr/local/bin/node","gid":0,"memoryUsage":{"arrayBuffers":240573,"external":26274350,"heapTotal":66318336,"heapUsed":36533176,"rss":105603072},"pid":1,"uid":0,"version":"v16.16.0"},"service":"democratic-csi","stack":"Error: Cannot parse privateKey: Unsupported key format\n    at Client.connect (/home/csi/app/node_modules/ssh2/lib/client.js:261:15)\n    at /home/csi/app/src/utils/ssh.js:78:19\n    at /home/csi/app/node_modules/async-mutex/lib/Mutex.js:23:66\n    at Semaphore.<anonymous> (/home/csi/app/node_modules/async-mutex/lib/Semaphore.js:38:46)\n    at step (/home/csi/app/node_modules/async-mutex/node_modules/tslib/tslib.js:144:27)\n    at Object.next (/home/csi/app/node_modules/async-mutex/node_modules/tslib/tslib.js:125:57)\n    at fulfilled (/home/csi/app/node_modules/async-mutex/node_modules/tslib/tslib.js:115:62)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","timestamp":"2022-08-16T12:22:41.751Z","trace":[{"column":15,"file":"/home/csi/app/node_modules/ssh2/lib/client.js","function":"Client.connect","line":261,"method":"connect","native":false},{"column":19,"file":"/home/csi/app/src/utils/ssh.js","function":null,"line":78,"method":null,"native":false},{"column":66,"file":"/home/csi/app/node_modules/async-mutex/lib/Mutex.js","function":null,"line":23,"method":null,"native":false},{"column":46,"file":"/home/csi/app/node_modules/async-mutex/lib/Semaphore.js","function":null,"line":38,"method":null,"native":false},{"column":27,"file":"/home/csi/app/node_modules/async-mutex/node_modules/tslib/tslib.js","function":"step","line":144,"method":null,"native":false},{"column":57,"file":"/home/csi/app/node_modules/async-mutex/node_modules/tslib/tslib.js","function":"Object.next","line":125,"method":"next","native":false},{"column":62,"file":"/home/csi/app/node_modules/async-mutex/node_modules/tslib/tslib.js","function":"fulfilled","line":115,"method":null,"native":false},{"column":5,"file":"node:internal/process/task_queues","function":"processTicksAndRejections","line":96,"method":null,"native":false}]}
running server shutdown, exit code: Error: Cannot parse privateKey: Unsupported key format

My helm chart looks like:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: democratic-csi-nfs
  namespace: storage-system
spec:
  interval: 5m
  chart:
    spec:
      chart: democratic-csi
      version: 0.13.4
      sourceRef:
        kind: HelmRepository
        name: democratic-csi-charts
        namespace: flux-system
      interval: 5m
  values:
    csiDriver:
      name: "org.democratic-csi.nfs"

    storageClasses:
    - name: tank-nfs-csi
      defaultClass: true
      reclaimPolicy: Retain
      volumeBindingMode: Immediate
      allowVolumeExpansion: true
      parameters:
        fsType: nfs
      mountOptions:
      - noatime
      - nfsvers=4
      secrets:
        provisioner-secret:
        controller-publish-secret:
        node-stage-secret:
        node-publish-secret:
        controller-expand-secret:

    controller:
      driver:
        imagePullPolicy: IfNotPresent

    driver:
      config:
        driver: zfs-generic-nfs
        sshConnection:
          host: ${NFS_IP}
          port: 22
          username: root
          privateKey: |
            ${NFS_RSA_KEY}
        zfs:
          datasetParentName: tank/kubernetes/nfs
          detachedSnapshotsDatasetParentName: tank/kubernetes/nfs-snapshots
          datasetEnableQuotas: true
          datasetEnableReservation: false
          datasetPermissionsMode: "0777"
          datasetPermissionsUser: 0
          datasetPermissionsGroup: 0
        nfs:
          shareStrategy: "setDatasetProperties"
          shareStrategySetDatasetProperties:
            properties:
              sharenfs: "rw,no_subtree_check,no_root_squash"
              # sharenfs: "on"
              # share: ""
          shareHost: "${NFS_IP}"

I have tried the newer format RSA key that looks like:

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
...
-----END OPENSSH PRIVATE KEY-----

as well as the older PEM style:

-----BEGIN RSA PRIVATE KEY-----
MIIJJwIBAAKCAgEAoONI0CZWyU1ts4k6idwhs6Pm9QVquaZOz4sjui4YrhKHosvP
...
-----END RSA PRIVATE KEY-----

I’ve even tried ED25519 keys. All result in the same error. I know I could use a password, but I’d rather not enable a root password on this host if possible.

I saw there were two other people who had the same issue (#133 and #200 ) but one just magically started working, and the other gave up and used a password.

[travisghansen]

I don’t think that will result in legitimate file syntax that can be used by the ssh client. I’m not sure the best way to get it to result in sane syntax however as I don’t have much experience with the helm operator 😦

[travisghansen]

You can pre create the secret and just point the chart at the existing secret if that’s helpful.