3 TrueNAS servers and performance

Apps and Virtualization
1

Hi,
I have 3 TrueNAS servers, I seem to be struggling with performance (I think) on all of them, the first server is mixed use of media storage and also has iSCSI extents. I will break down this post into 3 sections for each server, their config and some test results. I would like to get community input as to see if I actually am having performance issues.

Server 1:
Spec
Dell R710
3 Mirrored vDevs using 6tb wd reds
CPU: Dual E5520’s
RAM: 96 GB
Intel X710
Record Size 128K
iSCSI logical block size: 512
iPerf3 results just under 10gb
This server also is used as an NFS share

FIO test: fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=12 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=128k

Gets stuck at

fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=12 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=128k

4ktest: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=128
…
fio-3.28
Starting 12 processes
4ktest: Laying out IO file (1 file / 4096MiB)

Server 2:
**Spec **
Desktop PC:
CPu: Intel(R) Core™ i7-6700 CPU @ 3.40GHz
RAM: 32GB
Mirrored vDev: 2x2 sata drives
SSD slog: 1tb
Intel x710
Record Size 128K
iSCSI logical block size: 512
iPerf3 results just under 10gb

fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=12 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=128k

4ktest: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=128
…
fio-3.28
Starting 12 processes
4ktest: Laying out IO file (1 file / 4096MiB)
Jobs: 12 (f=12): [r(12)][80.0%][r=11.8GiB/s][r=96.4k IOPS][eta 00m:01s]
4ktest: (groupid=0, jobs=12): err= 0: pid=25690: Tue Jul 1 16:28:45 2025
read: IOPS=100k, BW=12.3GiB/s (13.2GB/s)(48.0GiB/3918msec)
clat (usec): min=9, max=168498, avg=113.97, stdev=1300.02
lat (usec): min=9, max=168498, avg=114.09, stdev=1300.24
clat percentiles (usec):
| 1.00th=[ 28], 5.00th=[ 41], 10.00th=[ 47], 20.00th=[ 54],
| 30.00th=[ 65], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 83],
| 70.00th=[ 86], 80.00th=[ 88], 90.00th=[ 91], 95.00th=[ 93],
| 99.00th=[ 117], 99.50th=[ 172], 99.90th=[10945], 99.95th=[27132],
| 99.99th=[62653]
bw ( MiB/s): min= 8774, max=17373, per=100.00%, avg=12709.92, stdev=242.22, samples=84
iops : min=70192, max=138986, avg=101676.00, stdev=1937.76, samples=84
lat (usec) : 10=0.01%, 20=0.30%, 50=15.06%, 100=83.20%, 250=1.14%
lat (usec) : 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.06%, 10=0.05%, 20=0.05%, 50=0.05%
lat (msec) : 100=0.02%, 250=0.01%
cpu : usr=1.51%, sys=63.12%, ctx=7331, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=393216,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
READ: bw=12.3GiB/s (13.2GB/s), 12.3GiB/s-12.3GiB/s (13.2GB/s-13.2GB/s), io=48.0GiB (51.5GB), run=3918-3918msec

**Server3: **
Dell R510:
CPu: Dual Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
RAM: 128GB
Mirrored vDEV: 6 4tb sata drives
SLOG: 250gb SSD
Intel X710
Iperf just under 10gb/s

fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=12 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=128k
4ktest: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=128
…
fio-3.28
Starting 12 processes
4ktest: Laying out IO file (1 file / 4096MiB)
Jobs: 12 (f=12): [r(12)][80.0%][r=13.6GiB/s][r=112k IOPS][eta 00m:01s]
4ktest: (groupid=0, jobs=12): err= 0: pid=36176: Tue Jul 1 16:28:16 2025
read: IOPS=107k, BW=13.1GiB/s (14.1GB/s)(48.0GiB/3658msec)
clat (usec): min=27, max=4256, avg=103.39, stdev=54.33
lat (usec): min=27, max=4256, avg=103.74, stdev=54.43
clat percentiles (usec):
| 1.00th=[ 43], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 62],
| 30.00th=[ 68], 40.00th=[ 73], 50.00th=[ 84], 60.00th=[ 110],
| 70.00th=[ 125], 80.00th=[ 139], 90.00th=[ 163], 95.00th=[ 212],
| 99.00th=[ 273], 99.50th=[ 297], 99.90th=[ 424], 99.95th=[ 510],
| 99.99th=[ 775]
bw ( MiB/s): min=11844, max=15748, per=100.00%, avg=13845.66, stdev=110.39, samples=77
iops : min=94751, max=125985, avg=110760.38, stdev=883.11, samples=77
lat (usec) : 50=3.65%, 100=51.81%, 250=42.52%, 500=1.96%, 750=0.04%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=5.31%, sys=93.76%, ctx=5008, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=393216,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
READ: bw=13.1GiB/s (14.1GB/s), 13.1GiB/s-13.1GiB/s (14.1GB/s-14.1GB/s), io=48.0GiB (51.5GB), run=3658-3658msec

I’d be interested to get some thoughts and also work out why server 1 never kicked on with the test?

2

Server 1 finally completed its fio test

fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=12 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=128k
4ktest: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=128
…
fio-3.28
Starting 12 processes
4ktest: Laying out IO file (1 file / 4096MiB)

Jobs: 9 (f=9): [r(3),_(3),r(6)][100.0%][r=9.77GiB/s][r=80.0k IOPS][eta 00m:00s]
4ktest: (groupid=0, jobs=12): err= 0: pid=49048: Tue Jul 1 17:02:44 2025
read: IOPS=77.5k, BW=9691MiB/s (10.2GB/s)(48.0GiB/5072msec)
clat (usec): min=28, max=7078, avg=143.01, stdev=99.23
lat (usec): min=28, max=7079, avg=143.39, stdev=99.27
clat percentiles (usec):
| 1.00th=[ 49], 5.00th=[ 60], 10.00th=[ 65], 20.00th=[ 77],
| 30.00th=[ 103], 40.00th=[ 119], 50.00th=[ 130], 60.00th=[ 141],
| 70.00th=[ 155], 80.00th=[ 202], 90.00th=[ 243], 95.00th=[ 260],
| 99.00th=[ 289], 99.50th=[ 359], 99.90th=[ 1336], 99.95th=[ 1827],
| 99.99th=[ 3064]
bw ( MiB/s): min= 7781, max=13522, per=100.00%, avg=9971.95, stdev=152.51, samples=108
iops : min=62252, max=108178, avg=79771.37, stdev=1220.02, samples=108
lat (usec) : 50=1.23%, 100=27.53%, 250=63.38%, 500=7.47%, 750=0.15%
lat (usec) : 1000=0.08%
lat (msec) : 2=0.12%, 4=0.03%, 10=0.01%
cpu : usr=5.47%, sys=94.33%, ctx=2683, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=393216,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
READ: bw=9691MiB/s (10.2GB/s), 9691MiB/s-9691MiB/s (10.2GB/s-10.2GB/s), io=48.0GiB (51.5GB), run=5072-5072msec

3

Crystal Disk mark on all of these servers from a Windows VM hits between 113MB/s to 121MB/s

4

Forgot to add the disk details
Server 1: Mirror
WDC WD60EFRX-68L x4
WDC WD60EFZX-68B x2

Server 2: Mirror + slog
WD2003FYYS-18W0B0 x2
ST320004CLAR2000
ST32000444SS
INTEL SSDSC2BW240A4

Server 3: Mirror + slog
HGST H7240AS60SUN4.0T x4
SEAGATE ST4000NM0023 x2
Crucial_CT256M55

5

Hello and welcome to the forums.

Well Crystal Disk Mark tests lots of things, SEQ write, 4K read, …
Maybe you want to share a screenshot. :wink:

Beside that, did you come up with fio parameters yoruself, or was it something you found in internet?
You name it " --name=4ktest" but you are actually testing " --bs=128k".
Another thing is " --iodepth=128 --numjobs=12" does not nearly resemble a single Windows VM.
Also funny is initiating read-and-write test " --rw=randrw", but then setting it to " --rwmixread=100" 100% read.

Btw: you might want to mention which TrueNAS you are using.

Last but not least: using a single “SLOG” SSD for pools risks your pool.

1 Like
6

Apologies, here is a screen shot testing server 3 from a windows VM.

I got the command off the internet, happy to do more testing where required.

All the ESXi hosts are running 10GB nics (same ones) as well.

7

hmm wouldn’t let me attach the image for some reason, but results are as follows
Settings from the top 2, 4GiB, MB/s
SEQ1m Q8T1 Read 119.95 Write 102.14
SEQ1M Q1T1 Read 102.96 Write 92.49
RND4K Q32T16 Read116.75 Write 27.01
RND4K Q1T1 Read 8.38 Write 8.71

Hope this helps

8

Without knowing all details about architecture,configuration,etc…
I think its pretty normal for a Windows VM accessing a NFS Share.

9

Its iSCSI and if you could clarify what details you want, I’m happy to share.

10

If I am not mistaken Crystal Disk Mark on Windows VM tested against iSCSI Share causes WRITES to be SYNC on your TrueNAS. This means WRITES go through your ZFS pools SLOG SSD. Crucial_CT256M55 and INTEL SSDSC2BW240A4 are relatively slow with SYNC WRITES. (Since these are no Enterprise SSDs offering Power Loss Protection (PLP)).

For a test you could set the corresponding ZFS pools ZVOL to handle WRITES as ASYNC:
iSCSI zvol sync = disabled
If this is the culprit, then Crystal Disk Mark results should improve alot.

But be warned “sync = disabled” leaves your data at risk!

1 Like
11

Nothing to add on performance but noticed SLOG devices may not be the correct ones and are over sized.

SLOG devices do not need a large capacity since they only need to service five seconds of data writes delivered by the network or a local application. A high-endurance, low-latency device between 8 GB and 32 GB is adequate for most modern networks, and you can strip or mirror several devices for either performance or redundancy. Pay attention to the published endurance claims for the device since a SLOG acts as the funnel point for most of the writes made to the system.

SLOG devices also need power protection. The purpose of the ZFS intent log (ZIL), and thus the SLOG, is to keep sync writes safe during a crash or power failure. If the SLOG is not power-protected and loses data after a power failure, it defeats the purpose of using a SLOG in the first place. Check the manufacturer specifications for the device to ensure the SLOG device is power-safe or has power loss/failure protection.

See Hybrid Storage & Flash Cache (SLOG/ZIL/L2ARC) part.

12

PLP is a mechanism to avoid data loss in the event of power failure, not directly related to performance.

In either case, I already attempted to disable sync writes on server 3 and set it to disabled. The performance actually went down.

13

Thanks, I am aware they are larger than need be, the document also says only ‘need to be 8/32gb’. I don’t think having them larger would make it slower (I know you weren’t mentioning this for performance reasons).

14

Actually it is directly related to SYNC WRITE performance. Why?
PLP enables SSD to cache SYNC WRITES to their internal DRAM, making it not necessary for them to really flush incoming data immediately to NAND as it is required for SYNC WRITES. For that reason they are tremendously faster doing SYNC WRITES than regular consumer SSDs.

Here is a link to a thread in old forums, where someone did also fio tests and could not believe it at first. Have a look. :wink:

Too bad, then the issue remains to be found.

That is correct.

15

Anyone else have any ideas that I can try?

16

I realised MPIO wasn’t enabled, after it was set benchmark results doubled in some areas. I believe 200~MB/s is still on the lower side for this setup though?