Unable to mount iscsi devices in nomad deployment

Trying to turn up an *arr application with a /config directory provided by an iscsi mount. I was able to create the volume via csc, and then set it up in nomad via terraform. However the last step, mounting it into the container, does not seem to work.

I have tried both as a file-system and a block-device attachment mode.

There must be something simple and obvious that I’m doing wrong, since I can’t find anything on the internet about this error message, but I’m not sure where to start looking.

Thanks in advance, again!

allocation logs:

2021-08-29T16:14:46-05:00  Setup Failure  failed to setup alloc: pre-run hook "csi_hook" failed: node plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code= Internal desc = {"code":125,"stdout":"","stderr":"chroot: cannot change root directory to '/host': No such file or directory\n"}

when I go look at the stderr for the node job I have this output:

{
  code: 13,
  message: `{"code":125,"stdout":"","stderr":"chroot: cannot change root directory to '/host': No such file or directory\\n"}`
}

stdout contains, along with a lot of status messages,

{"service":"democratic-csi","level":"info","message":"new request - driver: FreeNASDriver method: NodeStageVolume call: {\"_events\":{},\"_eventsCount\":1,\"call\":{},\"cancelled\":false,\"metadata\":{\"_internal_repr\":{\"user-agent\":[\"grpc-go/1.29.1\"]},\"flags\":0},\"request\":{\"publish_context\":{},\"secrets\":\"redacted\",\"volume_context\":{\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-radarr-volume-discovery\",\"lun\":\"0\",\"node_attach_driver\":\"iscsi\",\"portal\":\"10.10.220.110:3260\",\"provisioner_driver\":\"freenas-iscsi\"},\"volume_id\":\"radarr-volume\",\"staging_target_path\":\"/csi-data/staging/radarr-volume/rw-file-system-single-node-writer\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\"],\"fs_type\":\"ext4\"},\"access_type\":\"mount\"}}}"}
{"service":"democratic-csi","level":"debug","message":"operation lock keys: [\"volume_id_radarr-volume\"]"}
{"service":"democratic-csi","level":"verbose","message":"validating capabilities: [{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\",\"defaults\"],\"fs_type\":\"ext4\"},\"access_type\":\"mount\"}]"}
executing fileystem command: stat /csi-data/staging/radarr-volume/rw-file-system-single-node-writer
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-radarr-volume-discovery -p 10.10.220.110:3260 -o new
{"service":"democratic-csi","level":"error","message":"handler error - driver: FreeNASDriver method: NodeStageVolume error: {\"code\":125,\"stdout\":\"\",\"stderr\":\"chroot: cannot change root directory to '/host': No such file or directory\\n\"}"}
{"service":"democratic-csi","level":"info","message":"new request - driver: FreeNASDriver method: NodeUnpublishVolume call: {\"_events\":{},\"_eventsCount\":1,\"call\":{},\"cancelled\":false,\"metadata\":{\"_internal_repr\":{\"user-agent\":[\"grpc-go/1.29.1\"]},\"flags\":0},\"request\":{\"volume_id\":\"radarr-volume\",\"target_path\":\"/csi-data/per-alloc/f08e66af-efc9-b7a6-e3d6-750de3ad87fb/radarr-volume/rw-file-system-single-node-writer\"}}"}
{"service":"democratic-csi","level":"debug","message":"operation lock keys: [\"volume_id_radarr-volume\"]"}
executing mount command: findmnt --mountpoint /csi-data/per-alloc/f08e66af-efc9-b7a6-e3d6-750de3ad87fb/radarr-volume/rw-file-system-single-node-writer --output source,target,fstype,label,options,avail,size,used -b -J
executing fileystem command: stat /csi-data/per-alloc/f08e66af-efc9-b7a6-e3d6-750de3ad87fb/radarr-volume/rw-file-system-single-node-writer
failed to execute filesystem command: stat /csi-data/per-alloc/f08e66af-efc9-b7a6-e3d6-750de3ad87fb/radarr-volume/rw-file-system-single-node-writer, response: {"code":1,"stdout":"","stderr":"stat: cannot stat '/csi-data/per-alloc/f08e66af-efc9-b7a6-e3d6-750de3ad87fb/radarr-volume/rw-file-system-single-node-writer': No such file or directory\n"}
{"service":"democratic-csi","level":"info","message":"new response - driver: FreeNASDriver method: NodeUnpublishVolume response: {}"}
{"service":"democratic-csi","level":"info","message":"new request - driver: FreeNASDriver method: NodeUnstageVolume call: {\"_events\":{},\"_eventsCount\":1,\"call\":{},\"cancelled\":false,\"metadata\":{\"_internal_repr\":{\"user-agent\":[\"grpc-go/1.29.1\"]},\"flags\":0},\"request\":{\"volume_id\":\"radarr-volume\",\"staging_target_path\":\"/csi-data/staging/radarr-volume/rw-file-system-single-node-writer\"}}"}
{"service":"democratic-csi","level":"debug","message":"operation lock keys: [\"volume_id_radarr-volume\"]"}
executing mount command: findmnt --mountpoint /csi-data/staging/radarr-volume/rw-file-system-single-node-writer/block_device --output source,target,fstype,label,options,avail,size,used -b -J
executing mount command: findmnt --mountpoint /csi-data/staging/radarr-volume/rw-file-system-single-node-writer --output source,target,fstype,label,options,avail,size,used -b -J
executing mount command: findmnt --mountpoint /csi-data/staging/radarr-volume/rw-file-system-single-node-writer --output source,target,fstype,label,options,avail,size,used -b -J
executing fileystem command: stat /csi-data/staging/radarr-volume/rw-file-system-single-node-writer
executing fileystem command: rmdir /csi-data/staging/radarr-volume/rw-file-system-single-node-writer
{"service":"democratic-csi","level":"info","message":"new response - driver: FreeNASDriver method: NodeUnstageVolume response: {}"}

[travisghansen]

Basic docs and examples started here: https://github.com/democratic-csi/democratic-csi/tree/master/docs/Nomad

[dkowis]

Are you opening a port on the node deployment?

No, zero ports are opened there.

Node Job Config

job "democratic-storage-node-iscsi" {
  datacenters = ["discovery"]
  type        = "system"

  group "node" {
    task "node" {
      driver = "docker"

      config {
        image = "registry.light.kow.is/docker-hub/democraticcsi/democratic-csi:latest"
        auth {
          username = "robot$readbot"
          password = "qIw1FceV5Z8YuGMYaj5O3iltjrAbrJRp"
        }

        args = [
          "--csi-version=1.2.0",
          "--csi-name=org.democratic-csi.iscsi",
          "--driver-config-file=${NOMAD_TASK_DIR}/driver-config-file.yaml",
          "--log-level=debug",
          "--csi-mode=node",
          "--server-socket=/csi-data/csi.sock",
        ]

        network_mode = "host"
        ipc_mode = "host"
        privileged = true

        mount {
          type = "bind"
          target = "/host"
          source = "/"
          readonly=false
        }
        mount {
          type = "bind"
          target = "/etc/iscsi"
          source = "/etc/iscsi"
          readonly = false
        }
        mount {
          type = "bind"
          target = "/var/lib/iscsi"
          source = "/var/lib/iscsi"
          readonly = false
        }
      }

      csi_plugin {
        id        = "truenas"
        type      = "node"
        mount_dir = "/csi-data"
      }

      template {
        destination = "${NOMAD_TASK_DIR}/driver-config-file.yaml"

        data = <<EOH
driver: freenas-iscsi
instance_id:
httpConnection:
  protocol: https
  host: 10.10.220.110
  port: 443
  # use only 1 of apiKey or username/password
  # if both are present, apiKey is preferred
  # apiKey is only available starting in TrueNAS-12
  apiKey: LOLAPIKEY
  username: root
  #password:
  allowInsecure: true
  # use apiVersion 2 for TrueNAS-12 and up (will work on 11.x in some scenarios as well)
  # leave unset for auto-detection
  apiVersion: 2
sshConnection:
  host: 10.10.220.110
  port: 22
  username: root
  # use either password or key
  privateKey: |
    LOL PRIVATE KEY LOLOLOLOL
zfs:
  # can be used to override defaults if necessary
  # the example below is useful for TrueNAS 12
  #cli:
  #  sudoEnabled: true
  #
  #  leave paths unset for auto-detection
  #  paths:
  #    zfs: /usr/local/sbin/zfs
  #    zpool: /usr/local/sbin/zpool
  #    sudo: /usr/local/bin/sudo
  #    chroot: /usr/sbin/chroot
  
  # can be used to set arbitrary values on the dataset/zvol
  # can use handlebars templates with the parameters from the storage class/CO
  # TODO: set up handlebars templates to make this far more awesome
  datasetProperties:
     "org.freenas:description": "created via democratic-csi"

  datasetParentName: vmPool/nomad/vols
  # do NOT make datasetParentName and detachedSnapshotsDatasetParentName overlap
  # they may be siblings, but neither should be nested in the other
  detachedSnapshotsDatasetParentName: vmPool/nomad/snaps
  zvolCompression: ""
  zvolDedupe: ""
  zvolEnableReservation: False
  zvolBlocksize: ""
iscsi:
  targetPortal: "10.10.220.110:3260"
  targetPortals: []
  interface:

  # MUST iensure uniqueness
  # full iqn limit is 223 bytes, plan accordingly
  # default is //template name isn't defined!
  #nameTemplate: "{ { parameters.[csi.storage.k8s.io/pvc/namespace] }}-{ { parameters.[csi.storage.k8s.io/pvc/name] }}"
  namePrefix: csi-
  nameSuffix: "-discovery"
  # add as many as needed
  targetGroups:
    # get the correct ID from the "portal" section in the UI
    - targetGroupPortalGroup: 2
      # get the correct ID from the "initiators" section in the UI
      targetGroupInitiatorGroup: 1
      # None, CHAP, or CHAP Mutual
      targetGroupAuthType: None
      # get the correct ID from the "Authorized Access" section of the UI
      # only required if using Chap
      targetGroupAuthGroup:

  extentInsecureTpc: true
  extentXenCompat: false
  extentDisablePhysicalBlocksize: true
  # 512, 1024, 2048, or 4096,
  extentBlocksize: 512
  # "" (let FreeNAS decide, currently defaults to SSD), Unknown, SSD, 5400, 7200, 10000, 15000
  extentRpm: "SSD"
  # 0-100 (0 == ignore)
  extentAvailThreshold: 0
EOH
      }

      resources {
        cpu    = 30
        memory = 150
      }
    }
  }
}