Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Freenas-api-iscsi driver formatting disk every time instead of just initially

See original GitHub issue

Hi! I am newish to Nomad and the Democratic-csi plugin; I had gotten local host folders to work and thought I should switch to network storage since I have TrueNAS Scale.

Democratic-csi seems to be formatting iscsi disks every time they are mounted in a container, which erases all the data.

I am using Nomad 1.3.1 with TrueNAS Scale 22.02.2 and the freenas-api-iscsi driver.

I initially created the container and added a text file to the mounted iscsi folder. I then stopped the container, waited for the node processes to finish, and then started it again. When I started it for the second time I noticed that the democratic node had this line in the log file:

executing filesystem command: mkfs.ext4 /dev/sde

The mounted iscsi folder did not contain the text file from the first container. Is there a check for existing file systems that is failing? Otherwise I am not sure why the democratic node would format the iscsi disk every time.

See the complete logs and configurations below:

storage_wikijs-db.hcl 
id           = "wikijs-db-data"
name         = "wikijs-db-data"
type         = "csi"
plugin_id    = "org.democratic-csi.iscsi"
capacity_max = "5Gib"
capacity_min = "2Gib"

//Suggested to have only one capability
// capability {
//   access_mode     = "single-node-reader-only"
//   attachment_mode = "block-device"
// }

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

mount_options {
  fs_type     = "ext4"
  mount_flags = ["noatime"]
}
wikijs.job 
job "wikijs" {
  datacenters = ["dc1"]
  type        = "service"

  meta {
    version = "23"
  }

  group "Wikijs" {
    update {
      max_parallel = 1
    }

    # Prestart task
    task "set-permissions" {
      driver = "docker"
      volume_mount {
          volume = "wikijs-data"
          destination = "/wiki/data"
          read_only = false
        }

      config {
        image        = "busybox:1.28"
        command      = "sh"
        args         = ["-c", "echo -n 'Setting permissions\n'; chown 1000:1000 -R /wiki/data"]
        network_mode = "bridge"
      }

      resources {
        cpu    = 200
        memory = 128
      }

      lifecycle {
        hook    = "prestart"
        sidecar = false
      }
    }

    # End of Prestart Task


    volume "wikijs-data" {
      type = "csi"
      read_only = false
      source = "wikijs-data"
      attachment_mode = "file-system"
      access_mode     = "single-node-writer"
      mount_options {
        fs_type     = "ext4"
        mount_flags = ["noatime"]
      }
    }

    volume "wikijs-db-data" {
      type = "csi"
      read_only = false
      source = "wikijs-db-data"
      attachment_mode = "file-system"
      access_mode     = "single-node-writer"
      mount_options {
        fs_type     = "ext4"
        mount_flags = ["noatime"]
      }
    }
    
    task "wiki" {
        driver = "docker"
        volume_mount {
          volume = "wikijs-data"
          destination = "/wiki/data"
          read_only = false
        } 
        config {
            image = "ghcr.io/requarks/wiki:2"
            port_map {
                http = 3000
            }

        }

        env {
            DB_TYPE = "postgres"
            DB_HOST = "${NOMAD_IP_http}"
            #DB_HOST = "wiki-db.services.dc1.consul"
            #DB_HOST = "{{ env "NOMAD_IP_http" }}wiki-db.homelab"
            DB_PORT = "25432"
            DB_USER = "postgres"
            DB_PASS = "<password here>"
            DB_NAME = "wikijs"
        }

    service {
        name = "wiki"
        port = "http"
        tags = [
          "traefik.enable=true",
          "traefik.http.middlewares.httpsRedirect.redirectscheme.scheme=https",
          "traefik.http.routers.${NOMAD_TASK_NAME}_insecure.middlewares=httpsRedirect",
          "traefik.http.routers.${NOMAD_TASK_NAME}_insecure.rule=Host(`${NOMAD_TASK_NAME}`)",
          "traefik.http.routers.${NOMAD_TASK_NAME}.tls.domains[0].sans=${NOMAD_TASK_NAME}",
          "traefik.http.routers.${NOMAD_TASK_NAME}.tls.certresolver=myresolver",
          "traefik.http.routers.${NOMAD_TASK_NAME}.middlewares=httpsRedirect"
        ]
        check {
          type     = "http"
          path     = "/"
          #port     = "http"
          interval = "30s"
          timeout  = "2s"
        }
      }

      resources {
        cpu    = 500
        memory = 512
        network {
          port  "http"  {}
        }
      }
    }

    
    task "wiki-db" {
      driver = "docker"

      volume_mount {
          volume = "wikijs-db-data"
          destination = "/appdata"
          read_only = false
        }

      #user = "postgres"
      config {
        image = "postgres:14-bullseye"
        

        port_map {
          db = 5432
        }
      }

      env {
        POSTGRES_DB       = "wikijs"
        POSTGRES_USER     = "postgres"
        POSTGRES_PASSWORD = "<password here>"
        PGPASSWORD        = "<password here>"
        PGDATA            = "/appdata/db/postgres/"
      }

      service {
        name = "${NOMAD_TASK_NAME}"
        tags = ["postgres"]

        port = "db"

        check {
          type     = "tcp"
          port     = "db"
          interval = "30s"
          timeout  = "2s"
        }
      }

      resources {
        network {
          port  "db"  {
            static = "25432"
          }
        }
      }
    }
    


    
  }
}
democratic-csi iscsi node logs

These are the logs from the node that was hosting this container. This issue affects all of the nodes.

Initial creation of container

{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeStageVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"publish_context\":{},\"secrets\":\"redacted\",\"volume_context\":{\"node_attach_driver\":\"iscsi\",\"portal\":\"192.168.1.130:3260\",\"portals\":\"\",\"interface\":\"\",\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera\",\"lun\":\"0\",\"provisioner_driver\":\"freenas-api-iscsi\"},\"volume_id\":\"wikijs-db-data\",\"staging_target_path\":\"/local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\"],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"}},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:40:06.066Z"}
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera -p 192.168.1.130:3260 -o new
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera -p 192.168.1.130:3260 -o update --name node.startup --value manual
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera -p 192.168.1.130:3260 -l
executing iscsi command: iscsiadm -m session
executing iscsi command: iscsiadm -m session -r 336 --rescan
retry - failed condition, not trying again
executing filesystem command: realpath /dev/disk/by-path/ip-192.168.1.130:3260-iscsi-iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera-lun-0
{"host":"nomad4","level":"info","message":"successfully logged into portal 192.168.1.130:3260 and created device /dev/disk/by-path/ip-192.168.1.130:3260-iscsi-iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera-lun-0 with realpath /dev/sdc","service":"democratic-csi","timestamp":"2022-07-01T18:40:08.189Z"}
executing filesystem command: sh -c for file in $(ls -la /dev/mapper/* | grep "\->" | grep -oP "\-> .+" | grep -oP " .+"); do echo $(F=$(echo $file | grep -oP "[a-z0-9-]+");echo $F":"$(ls "/sys/block/${F}/slaves/");); done;
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -J -O /dev/sdc
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -J -O /dev/sdc
executing filesystem command: mkfs.ext4 /dev/sdc
executing filesystem command: blkid -p -o export /dev/sdc
executing filesystem command: realpath /dev/sdc
executing mount command: findmnt --source /dev/sdc --mountpoint /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /dev/sdc
executing mount command: findmnt --source /dev/sdc --mountpoint /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing mount command: mount -t ext4 -o noatime,defaults /dev/sdc /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing filesystem command: resize2fs /dev/sdc
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeStageVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:40:10.457Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodePublishVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"publish_context\":{},\"secrets\":\"redacted\",\"volume_context\":{\"portal\":\"192.168.1.130:3260\",\"portals\":\"\",\"interface\":\"\",\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera\",\"lun\":\"0\",\"provisioner_driver\":\"freenas-api-iscsi\",\"node_attach_driver\":\"iscsi\"},\"volume_id\":\"wikijs-db-data\",\"staging_target_path\":\"/local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer\",\"target_path\":\"/local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\"],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"},\"readonly\":false},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:40:10.458Z"}
retry - failed condition, not trying again
executing filesystem command: mkdir -p -m 0750 /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: mount --bind -o noatime,defaults /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodePublishVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:40:10.515Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeStageVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"publish_context\":{},\"secrets\":\"redacted\",\"volume_context\":{\"provisioner_driver\":\"freenas-api-iscsi\",\"node_attach_driver\":\"iscsi\",\"portal\":\"192.168.1.130:3260\",\"portals\":\"\",\"interface\":\"\",\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera\",\"lun\":\"0\"},\"volume_id\":\"wikijs-data\",\"staging_target_path\":\"/local/csi/staging/wikijs-data/rw-file-system-single-node-writer\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\"],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"}},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:40:10.518Z"}
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera -p 192.168.1.130:3260 -o new
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera -p 192.168.1.130:3260 -o update --name node.startup --value manual
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera -p 192.168.1.130:3260 -l
executing iscsi command: iscsiadm -m session
executing iscsi command: iscsiadm -m session -r 337 --rescan
retry - failed condition, not trying again
executing filesystem command: realpath /dev/disk/by-path/ip-192.168.1.130:3260-iscsi-iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera-lun-0
{"host":"nomad4","level":"info","message":"successfully logged into portal 192.168.1.130:3260 and created device /dev/disk/by-path/ip-192.168.1.130:3260-iscsi-iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera-lun-0 with realpath /dev/sde","service":"democratic-csi","timestamp":"2022-07-01T18:40:12.625Z"}
executing filesystem command: sh -c for file in $(ls -la /dev/mapper/* | grep "\->" | grep -oP "\-> .+" | grep -oP " .+"); do echo $(F=$(echo $file | grep -oP "[a-z0-9-]+");echo $F":"$(ls "/sys/block/${F}/slaves/");); done;
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -J -O /dev/sde
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -J -O /dev/sde
executing filesystem command: mkfs.ext4 /dev/sde
executing filesystem command: blkid -p -o export /dev/sde
executing filesystem command: realpath /dev/sde
executing mount command: findmnt --source /dev/sde --mountpoint /local/csi/staging/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /dev/sde
executing mount command: findmnt --source /dev/sde --mountpoint /local/csi/staging/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing mount command: mount -t ext4 -o noatime,defaults /dev/sde /local/csi/staging/wikijs-data/rw-file-system-single-node-writer
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing filesystem command: resize2fs /dev/sde
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeStageVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:40:14.873Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodePublishVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"publish_context\":{},\"secrets\":\"redacted\",\"volume_context\":{\"node_attach_driver\":\"iscsi\",\"portal\":\"192.168.1.130:3260\",\"portals\":\"\",\"interface\":\"\",\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera\",\"lun\":\"0\",\"provisioner_driver\":\"freenas-api-iscsi\"},\"volume_id\":\"wikijs-data\",\"staging_target_path\":\"/local/csi/staging/wikijs-data/rw-file-system-single-node-writer\",\"target_path\":\"/local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-data/rw-file-system-single-node-writer\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\"],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"},\"readonly\":false},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:40:14.874Z"}
retry - failed condition, not trying again
executing filesystem command: mkdir -p -m 0750 /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-data/rw-file-system-single-node-writer
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: mount --bind -o noatime,defaults /local/csi/staging/wikijs-data/rw-file-system-single-node-writer /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-data/rw-file-system-single-node-writer
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodePublishVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:40:14.929Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: Probe call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:40:18.951Z"}
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: Probe response: {\"ready\":{\"value\":true}}","service":"democratic-csi","timestamp":"2022-07-01T18:40:18.951Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: Probe call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:40:33.082Z"}
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: Probe response: {\"ready\":{\"value\":true}}","service":"democratic-csi","timestamp":"2022-07-01T18:40:33.082Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeGetCapabilities call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:40:33.085Z"}
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeGetCapabilities response: {\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}},{\"rpc\":{\"type\":\"GET_VOLUME_STATS\"}},{\"rpc\":{\"type\":\"EXPAND_VOLUME\"}},{\"rpc\":{\"type\":\"SINGLE_NODE_MULTI_WRITER\"}}]}","service":"democratic-csi","timestamp":"2022-07-01T18:40:33.085Z"}

Stopping container

{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeUnpublishVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"volume_id\":\"wikijs-data\",\"target_path\":\"/local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-data/rw-file-system-single-node-writer\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.244Z"}
executing mount command: findmnt --mountpoint /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeUnpublishVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"volume_id\":\"wikijs-db-data\",\"target_path\":\"/local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.252Z"}
executing mount command: findmnt --mountpoint /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: umount /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-data/rw-file-system-single-node-writer
executing mount command: umount /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer
executing filesystem command: rmdir /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-data/rw-file-system-single-node-writer
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeUnpublishVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.285Z"}
executing filesystem command: rmdir /local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeUnstageVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"volume_id\":\"wikijs-data\",\"staging_target_path\":\"/local/csi/staging/wikijs-data/rw-file-system-single-node-writer\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.289Z"}
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-data/rw-file-system-single-node-writer/block_device --output source,target,fstype,label,options -b -J --nofsroot
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeUnpublishVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.292Z"}
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeUnstageVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"volume_id\":\"wikijs-db-data\",\"staging_target_path\":\"/local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.301Z"}
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer/block_device --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /dev/sde
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -J -O /dev/sde
executing filesystem command: realpath /dev/sda
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -J -O /dev/sdc
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: umount /local/csi/staging/wikijs-data/rw-file-system-single-node-writer
executing mount command: umount /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing filesystem command: realpath /dev/sde
executing filesystem command: realpath /dev/sde
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -J -O /dev/sde
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -J -O /dev/sde
executing filesystem command: realpath /dev/sda
executing filesystem command: realpath /dev/sdc
executing iscsi command: iscsiadm -m session -P 3
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -J -O /dev/sdc
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -J -O /dev/sdc
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera -p 192.168.1.130:3260,1 -u
executing iscsi command: iscsiadm -m session -P 3
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera -p 192.168.1.130:3260,1 -u
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera -p 192.168.1.130:3260,1 -o delete
executing filesystem command: rmdir /local/csi/staging/wikijs-data/rw-file-system-single-node-writer
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeUnstageVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.568Z"}
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera -p 192.168.1.130:3260,1 -o delete
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeUnpublishVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"volume_id\":\"wikijs-db-data\",\"target_path\":\"/local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.578Z"}
{"host":"nomad4","level":"error","message":"handler error - driver: FreeNASApiDriver method: NodeUnpublishVolume error: {\"name\":\"GrpcError\",\"code\":10,\"message\":\"operation locked due to in progress operation(s): [\\\"volume_id_wikijs-db-data\\\"]\"}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.578Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeUnpublishVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"volume_id\":\"wikijs-db-data\",\"target_path\":\"/local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.580Z"}
{"host":"nomad4","level":"error","message":"handler error - driver: FreeNASApiDriver method: NodeUnpublishVolume error: {\"name\":\"GrpcError\",\"code\":10,\"message\":\"operation locked due to in progress operation(s): [\\\"volume_id_wikijs-db-data\\\"]\"}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.580Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeUnpublishVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"volume_id\":\"wikijs-db-data\",\"target_path\":\"/local/csi/per-alloc/979f2355-c806-90e6-0145-e7f9af2111f7/wikijs-db-data/rw-file-system-single-node-writer\"},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.585Z"}
{"host":"nomad4","level":"error","message":"handler error - driver: FreeNASApiDriver method: NodeUnpublishVolume error: {\"name\":\"GrpcError\",\"code\":10,\"message\":\"operation locked due to in progress operation(s): [\\\"volume_id_wikijs-db-data\\\"]\"}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.585Z"}
executing filesystem command: rmdir /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeUnstageVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:42:21.590Z"}

Starting container

{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: Probe response: {\"ready\":{\"value\":true}}","service":"democratic-csi","timestamp":"2022-07-01T18:45:18.972Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeStageVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"publish_context\":{},\"secrets\":\"redacted\",\"volume_context\":{\"provisioner_driver\":\"freenas-api-iscsi\",\"node_attach_driver\":\"iscsi\",\"portal\":\"192.168.1.130:3260\",\"portals\":\"\",\"interface\":\"\",\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera\",\"lun\":\"0\"},\"volume_id\":\"wikijs-data\",\"staging_target_path\":\"/local/csi/staging/wikijs-data/rw-file-system-single-node-writer\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\"],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"}},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:45:23.092Z"}
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera -p 192.168.1.130:3260 -o new
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera -p 192.168.1.130:3260 -o update --name node.startup --value manual
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera -p 192.168.1.130:3260 -l
executing iscsi command: iscsiadm -m session
executing iscsi command: iscsiadm -m session -r 338 --rescan
retry - failed condition, not trying again
executing filesystem command: realpath /dev/disk/by-path/ip-192.168.1.130:3260-iscsi-iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera-lun-0
{"host":"nomad4","level":"info","message":"successfully logged into portal 192.168.1.130:3260 and created device /dev/disk/by-path/ip-192.168.1.130:3260-iscsi-iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera-lun-0 with realpath /dev/sdc","service":"democratic-csi","timestamp":"2022-07-01T18:45:26.272Z"}
executing filesystem command: sh -c for file in $(ls -la /dev/mapper/* | grep "\->" | grep -oP "\-> .+" | grep -oP " .+"); do echo $(F=$(echo $file | grep -oP "[a-z0-9-]+");echo $F":"$(ls "/sys/block/${F}/slaves/");); done;
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -J -O /dev/sdc
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -J -O /dev/sdc
executing filesystem command: mkfs.ext4 /dev/sdc
executing filesystem command: blkid -p -o export /dev/sdc
executing filesystem command: realpath /dev/sdc
executing mount command: findmnt --source /dev/sdc --mountpoint /local/csi/staging/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /dev/sdc
executing mount command: findmnt --source /dev/sdc --mountpoint /local/csi/staging/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing mount command: mount -t ext4 -o noatime,defaults /dev/sdc /local/csi/staging/wikijs-data/rw-file-system-single-node-writer
executing filesystem command: realpath /dev/sdc
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing filesystem command: resize2fs /dev/sdc
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeStageVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:45:28.553Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodePublishVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"publish_context\":{},\"secrets\":\"redacted\",\"volume_context\":{\"lun\":\"0\",\"provisioner_driver\":\"freenas-api-iscsi\",\"node_attach_driver\":\"iscsi\",\"portal\":\"192.168.1.130:3260\",\"portals\":\"\",\"interface\":\"\",\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-wikijs-data-clustera\"},\"volume_id\":\"wikijs-data\",\"staging_target_path\":\"/local/csi/staging/wikijs-data/rw-file-system-single-node-writer\",\"target_path\":\"/local/csi/per-alloc/7df45fbd-0ecf-dc5c-72ae-e843c25feff8/wikijs-data/rw-file-system-single-node-writer\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\"],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"},\"readonly\":false},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:45:28.555Z"}
retry - failed condition, not trying again
executing filesystem command: mkdir -p -m 0750 /local/csi/per-alloc/7df45fbd-0ecf-dc5c-72ae-e843c25feff8/wikijs-data/rw-file-system-single-node-writer
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /local/csi/per-alloc/7df45fbd-0ecf-dc5c-72ae-e843c25feff8/wikijs-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: mount --bind -o noatime,defaults /local/csi/staging/wikijs-data/rw-file-system-single-node-writer /local/csi/per-alloc/7df45fbd-0ecf-dc5c-72ae-e843c25feff8/wikijs-data/rw-file-system-single-node-writer
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodePublishVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:45:28.610Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeStageVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"publish_context\":{},\"secrets\":\"redacted\",\"volume_context\":{\"interface\":\"\",\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera\",\"lun\":\"0\",\"provisioner_driver\":\"freenas-api-iscsi\",\"node_attach_driver\":\"iscsi\",\"portal\":\"192.168.1.130:3260\",\"portals\":\"\"},\"volume_id\":\"wikijs-db-data\",\"staging_target_path\":\"/local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\"],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"}},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:45:28.612Z"}
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera -p 192.168.1.130:3260 -o new
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera -p 192.168.1.130:3260 -o update --name node.startup --value manual
executing iscsi command: iscsiadm -m node -T iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera -p 192.168.1.130:3260 -l
executing iscsi command: iscsiadm -m session
executing iscsi command: iscsiadm -m session -r 339 --rescan
retry - failed condition, not trying again
executing filesystem command: realpath /dev/disk/by-path/ip-192.168.1.130:3260-iscsi-iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera-lun-0
{"host":"nomad4","level":"info","message":"successfully logged into portal 192.168.1.130:3260 and created device /dev/disk/by-path/ip-192.168.1.130:3260-iscsi-iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera-lun-0 with realpath /dev/sde","service":"democratic-csi","timestamp":"2022-07-01T18:45:30.709Z"}
executing filesystem command: sh -c for file in $(ls -la /dev/mapper/* | grep "\->" | grep -oP "\-> .+" | grep -oP " .+"); do echo $(F=$(echo $file | grep -oP "[a-z0-9-]+");echo $F":"$(ls "/sys/block/${F}/slaves/");); done;
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -J -O /dev/sde
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -J -O /dev/sde
executing filesystem command: mkfs.ext4 /dev/sde
executing filesystem command: blkid -p -o export /dev/sde
executing filesystem command: realpath /dev/sde
executing mount command: findmnt --source /dev/sde --mountpoint /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /dev/sde
executing mount command: findmnt --source /dev/sde --mountpoint /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing mount command: mount -t ext4 -o noatime,defaults /dev/sde /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer
executing filesystem command: realpath /dev/sde
executing filesystem command: lsblk -a -b -l -J -O
executing filesystem command: realpath /dev/sda
executing filesystem command: resize2fs /dev/sde
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodeStageVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:45:33.004Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodePublishVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{\"publish_context\":{},\"secrets\":\"redacted\",\"volume_context\":{\"lun\":\"0\",\"provisioner_driver\":\"freenas-api-iscsi\",\"node_attach_driver\":\"iscsi\",\"portal\":\"192.168.1.130:3260\",\"portals\":\"\",\"interface\":\"\",\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-wikijs-db-data-clustera\"},\"volume_id\":\"wikijs-db-data\",\"staging_target_path\":\"/local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer\",\"target_path\":\"/local/csi/per-alloc/7df45fbd-0ecf-dc5c-72ae-e843c25feff8/wikijs-db-data/rw-file-system-single-node-writer\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{\"mount_flags\":[\"noatime\"],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"},\"readonly\":false},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:45:33.007Z"}
retry - failed condition, not trying again
executing filesystem command: mkdir -p -m 0750 /local/csi/per-alloc/7df45fbd-0ecf-dc5c-72ae-e843c25feff8/wikijs-db-data/rw-file-system-single-node-writer
executing mount command: findmnt --mountpoint /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: findmnt --mountpoint /local/csi/per-alloc/7df45fbd-0ecf-dc5c-72ae-e843c25feff8/wikijs-db-data/rw-file-system-single-node-writer --output source,target,fstype,label,options -b -J --nofsroot
executing mount command: mount --bind -o noatime,defaults /local/csi/staging/wikijs-db-data/rw-file-system-single-node-writer /local/csi/per-alloc/7df45fbd-0ecf-dc5c-72ae-e843c25feff8/wikijs-db-data/rw-file-system-single-node-writer
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: NodePublishVolume response: {}","service":"democratic-csi","timestamp":"2022-07-01T18:45:33.071Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: Probe call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:45:33.110Z"}
{"host":"nomad4","level":"info","message":"new response - driver: FreeNASApiDriver method: Probe response: {\"ready\":{\"value\":true}}","service":"democratic-csi","timestamp":"2022-07-01T18:45:33.110Z"}
{"host":"nomad4","level":"info","message":"new request - driver: FreeNASApiDriver method: NodeGetCapabilities call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.45.0\"]},\"request\":{},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-07-01T18:45:33.111Z"}

freenas-api-iscsi.yaml.txt

Issue Analytics

  • State:open
  • Created a year ago
  • Comments:50 (28 by maintainers)

github_iconTop GitHub Comments

1reaction
travisghansencommented, Aug 2, 2022

I have added an ability to tweak this behavior, but am unlikely to alter the default for the time being: https://github.com/democratic-csi/democratic-csi/commit/516701bd29a41f4b6ffe0b90d26e286c5196706c

If you try the new next images and set the FILESYSTEM_TYPE_DETECTION_STRATEGY=blkid env var in the node deployment it should use blkid for the check instead of lsblk. I don’t really love the blkid feature because if the block device has no partition table and isn’t formatted the command exits with a 2 exit code. I have no idea if other scenarios could lead to that exit code which is why I’m a bit hesitant to go that route by default.

At this point I think we’re best to simply properly document the need for the udev mount and make sure it is added to all the examples etc.

1reaction
TripleE-0commented, Jul 31, 2022

I just tried the --sysroot /host option and it doesn’t appear to supply the fstype information either.

Can you try —sysroot /?

lsblk --sysroot / -a -J -O /dev/sdc gives the same fstype: null value.

Read more comments on GitHub >

github_iconTop Results From Across the Web

How To Fix "Format The Disk in Drive Before You Can Use it"
Sometimes our Windows 10 Drive showing You need to format the disk in ... This video has followed all the YouTube community guideline...
Read more >
Fix 'You Need to Format the Disk in Drive Before You Can Use ...
Get the error message you need to format the disk in drive before you can use it on SD memory card, USB drive...
Read more >
Windows 11 Error: You Need to Format the Disk in Drive ...
In this case, don't format the drive as it will just erase all the information. Instead, do the following to have your files...
Read more >
How to Format Your External Hard Drive - The New York Times
First, partition the drive using Disk Utility in macOS and (using our walk-through below) format one part as HFS+ and the other as...
Read more >
How to Format Hard Drives - Boston University
Format Drive Instructions. MAC Instructions. To reformat a hard drive or USB stick first open Disk Utility. To find it quickly, do a...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

iSCSI Kernel Messages

Next page

Question on democratic-csi PVC inode counts / Prometheus