zfs-nfs-democratic-csi-controller CrashLoopBackOff

Environment

• Cluster Kubernetes On Oracle Linux 8.3
• TrueNAS-12.0-U3.1

Issue After installation with the command:

helm upgrade --install \
  --create-namespace \
  --values freenas-nfs.yaml \
  --namespace democratic-csi \
  zfs-nfs democratic-csi/democratic-csi

I get the controller pod in CrashLoopBackOff

democratic-csi   zfs-nfs-democratic-csi-controller-ffbb878db-62jzs   1/4     CrashLoopBackOff   20         10m

Here some logs

{"service":"democratic-csi","level":"info","message":"initializing csi driver: freenas-nfs"}
{"service":"democratic-csi","level":"info","message":"starting csi server - name: org.democratic-csi.nfs, version: 1.2.0, driver: freenas-nfs, mode: controller, csi version: 1.2.0, address: , socket: unix:///csi-data/csi.sock"}
{"service":"democratic-csi","level":"info","message":"new request - driver: FreeNASDriver method: Probe call: {\"_events\":{},\"_eventsCount\":1,\"call\":{},\"cancelled\":false,\"metadata\":{\"_internal_repr\":{\"user-agent\":[\"grpc-node/1.24.0-pre1 grpc-c/8.0.0 (linux; chttp2; game)\"]},\"flags\":0},\"request\":{}}"}
{"service":"democratic-csi","level":"error","message":"handler error - driver: FreeNASDriver method: Probe error: Error: connect EHOSTUNREACH 172.17.3.20:22"}
{ code: 13, message: 'Error: connect EHOSTUNREACH 172.17.3.20:22' }
{"service":"democratic-csi","level":"info","message":"new request - driver: FreeNASDriver method: Probe call: {\"_events\":{},\"_eventsCount\":1,\"call\":{},\"cancelled\":false,\"metadata\":{\"_internal_repr\":{\"user-agent\":[\"grpc-go/1.34.0\"]},\"flags\":0},\"request\":{}}"}
{"service":"democratic-csi","level":"info","message":"new request - driver: FreeNASDriver method: Probe call: {\"_events\":{},\"_eventsCount\":1,\"call\":{},\"cancelled\":false,\"metadata\":{\"_internal_repr\":{\"user-agent\":[\"grpc-go/1.34.0\"]},\"flags\":0},\"request\":{}}"}
{"service":"democratic-csi","level":"info","message":"new request - driver: FreeNASDriver method: GetPluginInfo call: {\"_events\":{},\"_eventsCount\":1,\"call\":{},\"cancelled\":false,\"metadata\":{\"_internal_repr\":{\"user-agent\":[\"grpc-go/1.29.0\"]},\"flags\":0},\"request\":{}}"}
{"service":"democratic-csi","level":"info","message":"new response - driver: FreeNASDriver method: GetPluginInfo response: {\"name\":\"org.democratic-csi.nfs\",\"vendor_version\":\"1.2.0\"}"}
{"service":"democratic-csi","level":"info","message":"new request - driver: FreeNASDriver method: Probe call: {\"_events\":{},\"_eventsCount\":1,\"call\":{},\"cancelled\":false,\"metadata\":{\"_internal_repr\":{\"user-agent\":[\"grpc-go/1.29.0\"]},\"flags\":0},\"request\":{}}"}
{"service":"democratic-csi","level":"error","message":"handler error - driver: FreeNASDriver method: Probe error: Error: connect EHOSTUNREACH 172.17.3.20:22"}
{ code: 13, message: 'Error: connect EHOSTUNREACH 172.17.3.20:22' }
{"service":"democratic-csi","level":"error","message":"handler error - driver: FreeNASDriver method: Probe error: Error: connect EHOSTUNREACH 172.17.3.20:22"}
{ code: 13, message: 'Error: connect EHOSTUNREACH 172.17.3.20:22' }
{"service":"democratic-csi","level":"error","message":"handler error - driver: FreeNASDriver method: Probe error: Error: connect EHOSTUNREACH 172.17.3.20:22"}
{ code: 13, message: 'Error: connect EHOSTUNREACH 172.17.3.20:22' }
{"service":"democratic-csi","level":"info","message":"new request - driver: FreeNASDriver method: Probe call: {\"_events\":{},\"_eventsCount\":1,\"call\":{},\"cancelled\":false,\"metadata\":{\"_internal_repr\":{\"user-agent\":[\"grpc-node/1.24.0-pre1 grpc-c/8.0.0 (linux; chttp2; game)\"]},\"flags\":0},\"request\":{}}"}
{"service":"democratic-csi","level":"error","message":"handler error - driver: FreeNASDriver method: Probe error: Error: connect EHOSTUNREACH 172.17.3.20:22"}
{ code: 13, message: 'Error: connect EHOSTUNREACH 172.17.3.20:22' }

freenas-nfs.yaml

---
csiDriver:
  name: "org.democratic-csi.nfs"

storageClasses:
- name: freenas-nfs-csi
  defaultClass: false
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
  allowVolumeExpansion: true
  parameters:
    fsType: nfs
      
  mountOptions:
  - noatime
  - nfsvers=4
  secrets:
    provisioner-secret:
    controller-publish-secret:
    node-stage-secret:
    node-publish-secret:
    controller-expand-secret:

driver:
  config:
    driver: freenas-nfs
    instance_id:
    httpConnection:
      protocol: http
      host: 172.17.3.20
      port: 80
      # This is the API key that we generated previously
      apiKey: 1-yFiiVLYEdXdjr1MLQwsNO6XkP7Ny3OaliB3WSMBrc1nP9Apf0n96oxJCTEB04SmR
      username: root
      #password:
      allowInsecure: true
      apiVersion: 2
    sshConnection:
      host: 172.17.3.20
      port: 22
      username: root
      #password:
      # This is the SSH key that we generated for passwordless authentication
      privateKey: |
        -----BEGIN OPENSSH PRIVATE KEY-----
        # Here i pasted the cert that is properly working with ssh
        -----END OPENSSH PRIVATE KEY-----
    zfs:
      # Make sure to use the storage pool that was created previously
      datasetParentName: hdd/k8s/nfs/vols
      detachedSnapshotsDatasetParentName: hdd/k8s/nfs/snaps
      datasetEnableQuotas: true
      datasetEnableReservation: false
      datasetPermissionsMode: "0777"
      datasetPermissionsUser: root
      datasetPermissionsGroup: wheel
    nfs:
      shareHost: 172.17.3.20
      shareAlldirs: false
      shareAllowedHosts: []
      shareAllowedNetworks: []
      shareMaprootUser: root
      shareMaprootGroup: wheel
      shareMapallUser: ""
      shareMapallGroup: ""

Any idea what could be the problem ?

Thanks, Yzarca

[yzarca]

I tried to create new pods with ubuntu and centos, and they get CrashLoopBackOff too, so i guess the problem is into my cluster. I try to fix it and let you know.

Thanks, Yzarca

[yzarca]

Hi there,

I remade my k8s cluster (using calico for networking) because i really made too many experiment on it, after that i was able to configure TrueNAS quite easily.

Thanks Everybody, Regards, Yzarca