levtol
April 15, 2024, 5:53pm
1
Hello, community. I have a problem with low write speed to TrueNAS ISCSI share and I can not understand where the bottleneck is.
Not so long ago I’ve installed TrueNAS Scale, created 2 pools for tests (SSD and NVME), created 2 zvols. Then I’ve configured two ISCSI shares (SSD and NVME ones) and connected to them from Windows built-in ISCSI Initiator.
I’m trying to write a large file (~11GB) to each pool and I get ~150-180MB of constant write speed. Read speed is ok: if I copy file from TrueNAS transfer speed is about 280MB/s.
Windows host is connected to TrueNAS server via 2.5Gb/s NIC → 2.5Gb/s switch (with 10Gb SFP port) → 10Gb Intel NIC at TrueNAS server. There is no network problem: iperf3 shows ~270-280MB
What can be a reason of low write speed?
Write to TrueNAS speed:
White speed 2 files simultaneously:
Read from TrueNAS speed:
I’ve read this and this topics, but I have not found any solution for me: my SSD and NVME write speeds are the same an hit some invisible wall :-/
Hardware:
Intel SSD 520 Series - SSDSC2BW480A301 - 480GB X 2 - one for OS, other one is in “SSD” pool
NVME Pool: 2 X ADATA LEGEND 960 MAX [ALEG-960M-4TCS]
MBILC
April 15, 2024, 6:52pm
2
Consider overhead, once the cache fills on the drives, performance will drop also.
Have you run any local disk tests to test the pools directly?
levtol
April 15, 2024, 7:40pm
3
once the cache fills on the drives
You mean SLC cache? I think it not the issue due to disks are empty.
Have you run any local disk tests to test the pools directly?
What tests do I use to test a pool?
levtol
April 15, 2024, 7:42pm
4
Also, here is iowait output while writing to SSD
Summary
root@truenas[/home/admin]# iostat -dx 1 20 /dev/sdb /dev/zd64
Linux 6.1.74-production+truenas (truenas) 04/15/24 _x86_64_ (44 CPU)
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 29.36 1431.60 0.06 0.19 1.70 48.76 119.41 13400.63 0.77 0.64 1.66 112.22 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.03 0.25 6.49
zd64 0.12 9.04 0.00 0.00 0.20 72.55 19.77 5049.24 0.00 0.00 0.25 255.36 0.05 14079.98 0.00 0.00 1.69 257491.56 0.00 0.00 0.00 2.83
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 606.00 153984.00 0.00 0.00 0.29 254.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 89.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 688.00 87840.00 2.00 0.29 1.77 127.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.22 63.60
zd64 0.00 0.00 0.00 0.00 0.00 0.00 697.00 178176.00 0.00 0.00 0.22 255.63 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.16 100.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1053.00 132888.00 1.00 0.09 1.67 126.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.76 92.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 668.00 171008.00 0.00 0.00 0.25 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 99.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1302.00 136568.00 8.00 0.61 0.92 104.89 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.20 64.40
zd64 0.00 0.00 0.00 0.00 0.00 0.00 674.00 171792.00 0.00 0.00 0.27 254.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 99.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1762.00 187940.00 3.00 0.17 1.06 106.66 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.87 100.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 696.00 178176.00 0.00 0.00 0.26 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 99.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1767.00 186940.00 1.00 0.06 1.06 105.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.87 100.40
zd64 0.00 0.00 0.00 0.00 0.00 0.00 691.00 176384.00 0.00 0.00 0.32 255.26 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.22 100.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1496.00 155000.00 5.00 0.33 1.23 103.61 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 1.84 98.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 668.00 170752.00 0.00 0.00 0.23 255.62 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.16 97.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1098.00 140500.00 1.00 0.09 1.76 127.96 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.93 100.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 686.00 175360.00 0.00 0.00 0.17 255.63 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.12 98.80
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 1.00 16.00 0.00 0.00 1.00 16.00 1304.00 166860.00 3.00 0.23 1.46 127.96 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.90 100.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 657.00 167436.00 0.00 0.00 0.24 254.85 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.16 99.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1093.00 139688.00 1.00 0.09 1.75 127.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.92 100.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 675.00 172800.00 0.00 0.00 0.25 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 99.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1236.00 157728.00 0.00 0.00 1.55 127.61 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.91 100.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 701.00 179200.00 0.00 0.00 0.21 255.63 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.15 99.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1238.00 158840.00 6.00 0.48 1.55 128.30 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.91 100.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 673.00 172288.00 0.00 0.00 0.20 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.13 99.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1197.00 153200.00 2.00 0.17 1.60 127.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.92 100.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 732.00 187392.00 0.00 0.00 0.21 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.16 100.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 359.00 44192.00 2.00 0.55 1.37 123.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.49 25.60
zd64 0.00 0.00 0.00 0.00 0.00 0.00 707.00 180740.00 0.00 0.00 0.18 255.64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.13 98.80
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 1380.00 150972.00 21.00 1.50 2.24 109.40 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.10 42.80
zd64 0.00 0.00 0.00 0.00 0.00 0.00 705.00 180480.00 0.00 0.00 0.20 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.14 99.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 2854.00 312372.00 24.00 0.83 2.76 109.45 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.86 100.00
zd64 0.00 0.00 0.00 0.00 0.00 0.00 709.00 181504.00 0.00 0.00 0.25 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 99.60
and to NVME
Summary
root@truenas[/home/admin]# iostat -dx 1 20 /dev/nvme0n1 zd0
Linux 6.1.74-production+truenas (truenas) 04/15/24 _x86_64_ (44 CPU)
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 5.63 54.52 0.00 0.00 0.42 9.69 72.11 5234.33 0.00 0.00 0.08 72.59 0.00 0.00 0.00 0.00 0.00 0.00 0.04 1.00 0.01 0.57
zd0 0.16 13.04 0.00 0.00 0.58 80.76 24.88 6355.66 0.00 0.00 0.23 255.48 0.11 29095.75 0.00 0.00 0.57 258820.05 0.00 0.00 0.01 3.70
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 11.00 1412.00 0.00 0.00 4.36 128.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 5.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 627.00 160256.00 0.00 0.00 0.23 255.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.14 95.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 1644.00 141008.00 0.00 0.00 0.10 85.77 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.16 13.60
zd0 0.00 0.00 0.00 0.00 0.00 0.00 647.00 164884.00 0.00 0.00 0.29 254.84 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.19 94.80
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 1478.00 99052.00 0.00 0.00 0.13 67.02 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.50 0.20 14.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 642.00 164352.00 0.00 0.00 0.21 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.14 99.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 697.00 178432.00 0.00 0.00 0.26 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 99.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 680.00 174080.00 0.00 0.00 0.25 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 100.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 663.00 169728.00 0.00 0.00 0.27 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 100.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 5408.00 411368.00 0.00 0.00 0.06 76.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.32 34.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 600.00 153096.00 0.00 0.00 0.29 255.16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 94.80
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 2.00 8.00 0.00 0.00 22.00 4.00 3716.00 223656.00 0.00 0.00 0.05 60.19 0.00 0.00 0.00 0.00 0.00 0.00 2.00 1.00 0.23 20.80
zd0 0.00 0.00 0.00 0.00 0.00 0.00 596.00 152576.00 0.00 0.00 0.19 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 98.80
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 659.00 168704.00 0.00 0.00 0.23 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.15 98.80
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 683.00 174848.00 0.00 0.00 0.26 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 99.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 690.00 176640.00 0.00 0.00 0.17 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.12 100.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 4895.00 422132.00 0.00 0.00 0.08 86.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.37 33.60
zd0 0.00 0.00 0.00 0.00 0.00 0.00 552.00 140808.00 0.00 0.00 0.30 255.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.16 86.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 1.00 4.00 0.00 0.00 44.00 4.00 3784.00 230200.00 0.00 0.00 0.05 60.84 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.50 0.24 21.60
zd0 0.00 0.00 0.00 0.00 0.00 0.00 667.00 170752.00 0.00 0.00 0.19 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.12 99.60
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 706.00 180736.00 0.00 0.00 0.26 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 99.20
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 255.00 65280.00 0.00 0.00 0.16 256.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04 34.00
MBILC
April 15, 2024, 8:02pm
5
Some NVMe drives have DRAM cache on them, this is often what gives you the top speeds. Once that cache fills and cycles through, you wont see those 5,6,7GB/sec speeds, but should still see 3-4GB/s (2TB Performance Results - Adata Legend 960 Max SSD Review: Now With Extra Toppings - Page 2 | Tom's Hardware ) as it uses the SLC cache.
Your 2.5Gb NIC is what would cap your max transfer speeds out from TrueNAS, theoretical would be a steady 300MB/s or so, now factor in iSCSI overhead…
May be some info here
It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times.
ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data...
fio would of been it, to test internally to see max speeds, but there are also different tests that can be run for different workloads, queue depth matters and size of tests.
Like many workloads - a single threaded load may not max things out, but if you run or get several loads hitting it all at once, that is where it starts to shine.
several different test with fio here:
Hello All,
As I lurk these forums often and read far too much, I finally got some time to do some testing on my set up (in sig)
Along with that, I added 2 more NVMe's to my VM pool.
Using FIO (all tests run local on TrueNAS via SSH), I began doing...
levtol
April 15, 2024, 9:49pm
6
Like many workloads - a single threaded load may not max things out, but if you run or get several loads hitting it all at once, that is where it starts to shine.
Take a look at “White speed 2 files simultaneously:” in original message, there is 150MB/s anyway.
Also, see second spoiler at Low zvol write speed - #4 by levtol . NVME is busy max at 30%, while “zd0” is 100% busy.
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 5408.00 411368.00 0.00 0.00 0.06 76.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.32 34.00
zd0 0.00 0.00 0.00 0.00 0.00 0.00 600.00 153096.00 0.00 0.00 0.29 255.16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 94.80
