Poor performance over freenas-nfs csi driver

I’m seeing some poor performance of file operations within a k8s container using a freenas-nfs volume, e.g. 5 seconds to ‘touch’ a new file:

root@gitlab-79958f5dd7-xzzvd:/var/log/gitlab/test# time touch a.txt

real    0m5.560s
user    0m0.001s
sys     0m0.000s

I tried to get some performance data via ‘dd’, and can see 40-77 MB/sec when writing 500MB, which seems ok:

root@gitlab-79958f5dd7-xzzvd:/var/log/gitlab/test# dd if=/dev/zero of=test5.img bs=500M count=1 oflag=dsync
1+0 records in
1+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 8.82374 s, 59.4 MB/s

however if I change the dd parameters to be write 512-byte blocks, then performance falls off a cliff to 0.2kB/s:

root@gitlab-79958f5dd7-xzzvd:/var/log/gitlab/test# dd if=/dev/zero of=test1.img bs=512 count=10 oflag=dsync
10+0 records in
10+0 records out
5120 bytes (5.1 kB, 5.0 KiB) copied, 25.1562 s, 0.2 kB/s

Are there any options / parameters I can modify to improve performance ?

As a bit of background, I’m trying to get gitlab running in k8s and most operations appear to be timing out due to poor NFS performance.

NB: I saw the ‘MountOptions’ on this issue: https://github.com/democratic-csi/democratic-csi/issues/6#issuecomment-654303441 so tried setting these in my helm yaml:

  mountOptions:
  - noatime
  - nfsvers=4
  - rsize=16384
  - wsize=16384

but I don’t think they’re taking effect; from the k8s node the mounts are still using a large rsize/wsize:

$ cat /proc/mounts | grep gitlab
192.168.0.129:/mnt/raidvolume/k8s/bnekub02/nfs/volumes/pvc-15fc69e7-e5f0-4651-9532-14f0ef5dd2a9/logs /var/lib/kubelet/pods/6faa681a-bbfa-41b2-8ea1-a5c7f289b5fa/volume-subpaths/pvc-15fc69e7-e5f0-4651-9532-14f0ef5dd2a9/gitlab/1 nfs4 rw,noatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.131,local_lock=none,addr=192.168.0.129 0 0

but I dont think that’s the problem either, tried connecting from another machine with a smaller rsize/wsize ( sudo mount -t nfs -o rsize=16485 -o wsize=14384 … ) but still getting around 0.2 kB/s using dd

[knoxg]

magic, thanks for that,

good to see some numbers to compare against, and thanks for creating a great project 😃

[travisghansen]

So I think you are about on par.

dd if=/dev/zero of=test4.img bs=512 count=10000 oflag=dsync
10000+0 records in
10000+0 records out
5120000 bytes (5.1 MB, 4.9 MiB) copied, 30.2612 s, 169 kB/s

Again, my server is quite busy already so those numbers are probably about right.

My setup is nothing special, gigabit networking gear, no jumbo frames, etc.