Bad performance after load, root case is NFS ? SAS Expander ? something else?

TrueNAS General
1

Hello guys,
I wite to you because I’m out of idea

I have a 10G network
2 ESXi with EPYC 7551P 32-Core and 256 GB of ram
One truenas with
EPYC 7261 8-Core
512 GB of ram
A sas initiator with 2 sas expanders
20 HDD as stripe (one disk one pool). I know which risks it implies
2x20 TB HDD with idle vm in one pool raidz1
2x6 TB HDD with idle vm in one pool raidz1
2x2 TB SSD with idle vm in one pool raidz1

I mount the 20 pool each with nfs on a dataset with Sync disabled (to maxout write performance), compression lz4 which is default

My problem is I use these pool to host vm with vmdk that perform some random read/write but nothing crazy. When I poweron these vm, it seems that everything become slow, even the 2x2 TB SSD raidz1 which host boot vmdk. It becomes so slow that I have difficulty to boot these vm on ubuntu

I’m open to any suggestions

2

Have you tried playing around with the record size of your dataset? Perhaps give 16K a shot or even smaller.

3

Hello Johnny,

Thanks for your answer, I saw that this parameter is applied on new written data so I gonna try it and try to move some stuff then keep you informed on results

4

Hello again Johnny,

I found out my problem, one of my ESXi has an hardware problem (they are really the worst to find, this one made me crazy) that kill the performance on my 10G link. I get back on my 1G that I kept for backup only and I got far better performance

I found my problem but wanted to give a shot to your advice
I made some benchmarks about 16k vs 128k (then 1M because why not)
With this command from an ubuntu vm

fio --name=randread --size=10G --filename=testfile10G --bs=4k --rw=randread --direct=1 --numjobs=1 --iodepth=32 --time_based --runtime=60
With 16k I got 132 IOPS and 531 KiB/s
With 128K I got 127 IOPS and 510 KiB/s
With 1M I got 121 IOPS and 487 KiB/s

fio --name=seqread --filename=testfile10G --size=10G --bs=1M --rw=read --direct=1 --numjobs=1 --iodepth=1
With 16k I got 38 IOPS and 38 MiB/s
With 128K I got 41 IOPS and 41 MiB/s
With 1M I got 41 IOPS and 41.6 KiB/s

ChatGPT told me that using smaller record size was good for ARC, because for 4k access (from inside my VM) truenas would load into ram 16k instead of 128k
I’m most surprised that 1M don’t get any improvement on squential read