When last I worked in Ethernet at Intel, (just under a year ago) there were no 100Gb Ethernet driver for windows. Especially not for Desktop versions.
!00Gb is not a desktop setup. It’s a serious cluster of servers - which tend to run on Linux.
1 Like
Going to use Starwinds NVMe-of
1 Like
Chelsio made drivers also for Windows Desktop:
https://service.chelsio.com/downloads/Microsoft/
Also 100 gbps is not very recent, in server you can see 200 and 400 gbps (not sure about 800 gbps).
Best Regards,
Antonio
On target side (Truenas Scale?), is it possible to use NVMe-of ?
Best Regards,
Antonio
Here is my test on TrueNAS Core Target with Chelsio 100 Gbps network.
But first some test on Fedora 41 local nvme disk (1TB 980 pro Samsung).
I’m using fio commands from https://docs.oracle.com/en-us/iaas/Content/Block/References/samplefiocommandslinux.htm with some modifications.
Local Disk
Random Read IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
iops-test-job: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [r(4)][3.3%][r=1221MiB/s][r=313k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [r(4)][100.0%][r=1215MiB/s][r=311k IOPS][eta 00m:00s]
Jobs: 1 (f=0): [_(1),E(2),f(1)][100.0%][r=1193MiB/s][r=305k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=5248: Sat Mar 1 09:52:36 2025
read: IOPS=312k, BW=1220MiB/s (1279MB/s)(143GiB/120002msec)
slat (usec): min=4, max=4018, avg=11.57, stdev= 6.47
clat (usec): min=47, max=10524, avg=3267.55, stdev=189.29
lat (usec): min=56, max=10538, avg=3279.13, stdev=189.51
clat percentiles (usec):
| 1.00th=[ 2900], 5.00th=[ 3064], 10.00th=[ 3097], 20.00th=[ 3163],
| 30.00th=[ 3195], 40.00th=[ 3228], 50.00th=[ 3261], 60.00th=[ 3261],
| 70.00th=[ 3294], 80.00th=[ 3359], 90.00th=[ 3458], 95.00th=[ 3589],
| 99.00th=[ 3884], 99.50th=[ 4080], 99.90th=[ 4621], 99.95th=[ 5014],
| 99.99th=[ 5997]
bw ( MiB/s): min= 1130, max= 1242, per=100.00%, avg=1219.66, stdev= 3.45, samples=959
iops : min=289318, max=318204, avg=312233.26, stdev=882.44, samples=959
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=99.35%, 10=0.63%, 20=0.01%
cpu : usr=6.87%, sys=77.27%, ctx=187151, majf=0, minf=1064
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=37466441,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=1220MiB/s (1279MB/s), 1220MiB/s-1220MiB/s (1279MB/s-1279MB/s), io=143GiB (153GB), run=120002-120002msec
Random Write IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [w(4)][2.5%][w=187MiB/s][w=47.9k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [w(4)][100.0%][w=185MiB/s][w=47.4k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=6073: Sat Mar 1 10:03:17 2025
write: IOPS=45.0k, BW=176MiB/s (184MB/s)(20.6GiB/120001msec); 0 zone resets
slat (usec): min=9, max=768157, avg=87.02, stdev=994.98
clat (usec): min=18, max=837109, avg=22666.47, stdev=17316.73
lat (usec): min=44, max=837176, avg=22753.49, stdev=17351.74
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 16],
| 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 24],
| 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 32], 95.00th=[ 35],
| 99.00th=[ 43], 99.50th=[ 46], 99.90th=[ 61], 99.95th=[ 338],
| 99.99th=[ 802]
bw ( KiB/s): min= 5288, max=234048, per=100.00%, avg=180748.51, stdev=6044.45, samples=956
iops : min= 1322, max=58512, avg=45187.12, stdev=1511.11, samples=956
lat (usec) : 20=0.01%, 50=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=4.44%, 20=34.49%, 50=60.78%
lat (msec) : 100=0.23%, 500=0.02%, 1000=0.04%
cpu : usr=1.98%, sys=25.21%, ctx=9072534, majf=0, minf=47
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,5399909,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=176MiB/s (184MB/s), 176MiB/s-176MiB/s (184MB/s-184MB/s), io=20.6GiB (22.1GB), run=120001-120001msec
Random Read/Write IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [m(4)][2.5%][r=139MiB/s,w=139MiB/s][r=35.7k,w=35.7k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [m(4)][100.0%][r=115MiB/s,w=115MiB/s][r=29.5k,w=29.6k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=6008: Sat Mar 1 09:59:29 2025
read: IOPS=32.6k, BW=127MiB/s (134MB/s)(14.9GiB/120001msec)
slat (usec): min=5, max=66299, avg=48.89, stdev=89.03
clat (usec): min=52, max=91551, avg=15638.88, stdev=2298.01
lat (usec): min=119, max=91655, avg=15687.77, stdev=2302.54
clat percentiles (usec):
| 1.00th=[11338], 5.00th=[12518], 10.00th=[13042], 20.00th=[13829],
| 30.00th=[14484], 40.00th=[15008], 50.00th=[15533], 60.00th=[16057],
| 70.00th=[16581], 80.00th=[17171], 90.00th=[18220], 95.00th=[19268],
| 99.00th=[21103], 99.50th=[21890], 99.90th=[25297], 99.95th=[39060],
| 99.99th=[82314]
bw ( KiB/s): min=101856, max=148248, per=100.00%, avg=130411.33, stdev=1916.47, samples=960
iops : min=25464, max=37062, avg=32602.84, stdev=479.12, samples=960
write: IOPS=32.6k, BW=127MiB/s (133MB/s)(14.9GiB/120001msec); 0 zone resets
slat (usec): min=8, max=66289, avg=69.53, stdev=93.13
clat (usec): min=19, max=91621, avg=15658.89, stdev=2287.69
lat (usec): min=42, max=91642, avg=15728.42, stdev=2294.05
clat percentiles (usec):
| 1.00th=[11338], 5.00th=[12518], 10.00th=[13173], 20.00th=[13960],
| 30.00th=[14484], 40.00th=[15008], 50.00th=[15533], 60.00th=[16057],
| 70.00th=[16581], 80.00th=[17171], 90.00th=[18220], 95.00th=[19268],
| 99.00th=[21103], 99.50th=[21890], 99.90th=[25560], 99.95th=[39060],
| 99.99th=[82314]
bw ( KiB/s): min=102192, max=147144, per=100.00%, avg=130315.57, stdev=1874.24, samples=960
iops : min=25548, max=36786, avg=32578.89, stdev=468.56, samples=960
lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.09%, 20=97.33%, 50=2.56%
lat (msec) : 100=0.01%
cpu : usr=3.08%, sys=32.74%, ctx=8973608, majf=0, minf=58
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=3912341,3909467,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=127MiB/s (134MB/s), 127MiB/s-127MiB/s (134MB/s-134MB/s), io=14.9GiB (16.0GB), run=120001-120001msec
WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=14.9GiB (16.0GB), run=120001-120001msec
Sequential Read IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=read --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=1262MiB/s][r=323k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [R(4)][100.0%][r=1262MiB/s][r=323k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=6111: Sat Mar 1 10:06:19 2025
read: IOPS=321k, BW=1255MiB/s (1315MB/s)(147GiB/120001msec)
slat (usec): min=4, max=8699, avg=11.32, stdev= 9.57
clat (usec): min=48, max=14200, avg=3176.40, stdev=232.71
lat (usec): min=57, max=14264, avg=3187.72, stdev=233.30
clat percentiles (usec):
| 1.00th=[ 2737], 5.00th=[ 2868], 10.00th=[ 2933], 20.00th=[ 3032],
| 30.00th=[ 3064], 40.00th=[ 3130], 50.00th=[ 3163], 60.00th=[ 3195],
| 70.00th=[ 3261], 80.00th=[ 3294], 90.00th=[ 3392], 95.00th=[ 3523],
| 99.00th=[ 4015], 99.50th=[ 4293], 99.90th=[ 5211], 99.95th=[ 5538],
| 99.99th=[ 6259]
bw ( MiB/s): min= 1191, max= 1315, per=100.00%, avg=1254.83, stdev= 4.35, samples=956
iops : min=304914, max=336646, avg=321235.68, stdev=1114.75, samples=956
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=98.98%, 10=1.01%, 20=0.01%
cpu : usr=6.14%, sys=76.18%, ctx=580150, majf=0, minf=1067
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=38539444,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=1255MiB/s (1315MB/s), 1255MiB/s-1255MiB/s (1315MB/s-1315MB/s), io=147GiB (158GB), run=120001-120001msec
Sequential Write IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=write --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [W(4)][2.5%][w=221MiB/s][w=56.7k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [W(4)][100.0%][w=207MiB/s][w=53.1k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=6201: Sat Mar 1 10:09:32 2025
write: IOPS=49.8k, BW=195MiB/s (204MB/s)(22.8GiB/120001msec); 0 zone resets
slat (usec): min=7, max=859952, avg=78.72, stdev=782.88
clat (usec): min=22, max=915225, avg=20464.42, stdev=14985.50
lat (usec): min=40, max=915237, avg=20543.14, stdev=15017.45
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 13],
| 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 20], 60.00th=[ 22],
| 70.00th=[ 25], 80.00th=[ 28], 90.00th=[ 33], 95.00th=[ 37],
| 99.00th=[ 47], 99.50th=[ 52], 99.90th=[ 67], 99.95th=[ 86],
| 99.99th=[ 894]
bw ( KiB/s): min=18512, max=307128, per=100.00%, avg=199362.53, stdev=8429.34, samples=960
iops : min= 4628, max=76782, avg=49840.63, stdev=2107.33, samples=960
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=11.99%, 20=41.20%, 50=46.17%
lat (msec) : 100=0.58%, 250=0.01%, 500=0.02%, 1000=0.02%
cpu : usr=1.75%, sys=24.36%, ctx=10538353, majf=0, minf=40
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,5980876,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=195MiB/s (204MB/s), 195MiB/s-195MiB/s (204MB/s-204MB/s), io=22.8GiB (24.5GB), run=120001-120001msec
Random Read Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randread --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
throughput-test-job: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [r(4)][3.3%][r=2409MiB/s][r=9636 IOPS][eta 01m:56s]
...
Jobs: 4 (f=0): [f(4)][100.0%][r=5154MiB/s][r=20.6k IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=6837: Sat Mar 1 10:32:06 2025
read: IOPS=20.4k, BW=5105MiB/s (5353MB/s)(598GiB/120013msec)
slat (usec): min=20, max=36676, avg=40.45, stdev=188.60
clat (usec): min=4776, max=85105, avg=12492.93, stdev=5392.59
lat (usec): min=4810, max=85133, avg=12533.38, stdev=5393.77
clat percentiles (usec):
| 1.00th=[ 6194], 5.00th=[ 6587], 10.00th=[ 6980], 20.00th=[ 7832],
| 30.00th=[ 8848], 40.00th=[10028], 50.00th=[11469], 60.00th=[12780],
| 70.00th=[14353], 80.00th=[16319], 90.00th=[19006], 95.00th=[21890],
| 99.00th=[30540], 99.50th=[35914], 99.90th=[47449], 99.95th=[52167],
| 99.99th=[64226]
bw ( MiB/s): min= 2196, max= 5827, per=100.00%, avg=5106.03, stdev=177.21, samples=960
iops : min= 8786, max=23308, avg=20424.13, stdev=708.86, samples=960
lat (msec) : 10=39.36%, 20=52.72%, 50=7.86%, 100=0.07%
cpu : usr=1.07%, sys=19.45%, ctx=1952921, majf=0, minf=16421
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=2450899,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=5105MiB/s (5353MB/s), 5105MiB/s-5105MiB/s (5353MB/s-5353MB/s), io=598GiB (642GB), run=120013-120013msec
Random Write Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randwrite --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=randwrite, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [w(4)][2.5%][w=2630MiB/s][w=10.5k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [w(4)][100.0%][w=1769MiB/s][w=7075 IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=6915: Sat Mar 1 10:36:42 2025
write: IOPS=8526, BW=2132MiB/s (2235MB/s)(250GiB/120034msec); 0 zone resets
slat (usec): min=51, max=75924, avg=334.37, stdev=597.67
clat (msec): min=6, max=212, avg=29.68, stdev=14.27
lat (msec): min=6, max=212, avg=30.02, stdev=14.24
clat percentiles (msec):
| 1.00th=[ 18], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 23],
| 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 25], 60.00th=[ 26],
| 70.00th=[ 27], 80.00th=[ 32], 90.00th=[ 48], 95.00th=[ 64],
| 99.00th=[ 87], 99.50th=[ 97], 99.90th=[ 123], 99.95th=[ 133],
| 99.99th=[ 161]
bw ( MiB/s): min= 1141, max= 2750, per=100.00%, avg=2131.90, stdev=115.39, samples=960
iops : min= 4566, max=11002, avg=8527.62, stdev=461.56, samples=960
lat (msec) : 10=0.01%, 20=6.54%, 50=84.20%, 100=8.84%, 250=0.42%
cpu : usr=1.75%, sys=30.21%, ctx=1327165, majf=0, minf=41
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,1023503,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=2132MiB/s (2235MB/s), 2132MiB/s-2132MiB/s (2235MB/s-2235MB/s), io=250GiB (268GB), run=120034-120034msec
Random Read/Write Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randrw --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [m(4)][2.5%][r=1778MiB/s,w=1792MiB/s][r=7113,w=7168 IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [m(4)][100.0%][r=1262MiB/s,w=1249MiB/s][r=5046,w=4997 IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=7067: Sat Mar 1 10:41:26 2025
read: IOPS=6344, BW=1586MiB/s (1663MB/s)(186GiB/120020msec)
slat (usec): min=22, max=90351, avg=167.86, stdev=396.22
clat (usec): min=570, max=141947, avg=23740.82, stdev=12724.66
lat (usec): min=704, max=141981, avg=23908.68, stdev=12700.97
clat percentiles (msec):
| 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 16],
| 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 20],
| 70.00th=[ 23], 80.00th=[ 32], 90.00th=[ 43], 95.00th=[ 52],
| 99.00th=[ 70], 99.50th=[ 77], 99.90th=[ 97], 99.95th=[ 107],
| 99.99th=[ 126]
bw ( MiB/s): min= 973, max= 2059, per=100.00%, avg=1586.43, stdev=83.42, samples=960
iops : min= 3894, max= 8238, avg=6345.72, stdev=333.70, samples=960
write: IOPS=6357, BW=1589MiB/s (1667MB/s)(186GiB/120020msec); 0 zone resets
slat (usec): min=41, max=89977, avg=311.37, stdev=407.54
clat (usec): min=122, max=126398, avg=16089.63, stdev=7087.97
lat (usec): min=191, max=126612, avg=16401.00, stdev=7109.83
clat percentiles (usec):
| 1.00th=[ 799], 5.00th=[ 2868], 10.00th=[ 6521], 20.00th=[13435],
| 30.00th=[14877], 40.00th=[15795], 50.00th=[16450], 60.00th=[17171],
| 70.00th=[17957], 80.00th=[18744], 90.00th=[20841], 95.00th=[25297],
| 99.00th=[42730], 99.50th=[48497], 99.90th=[63701], 99.95th=[72877],
| 99.99th=[95945]
bw ( MiB/s): min= 982, max= 1994, per=100.00%, avg=1589.61, stdev=82.44, samples=960
iops : min= 3930, max= 7976, avg=6358.44, stdev=329.76, samples=960
lat (usec) : 250=0.03%, 500=0.19%, 750=0.24%, 1000=0.26%
lat (msec) : 2=1.05%, 4=1.63%, 10=3.73%, 20=68.77%, 50=21.11%
lat (msec) : 100=2.96%, 250=0.04%
cpu : usr=1.75%, sys=28.47%, ctx=1714484, majf=0, minf=44
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=761486,763013,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1586MiB/s (1663MB/s), 1586MiB/s-1586MiB/s (1663MB/s-1663MB/s), io=186GiB (200GB), run=120020-120020msec
WRITE: bw=1589MiB/s (1667MB/s), 1589MiB/s-1589MiB/s (1667MB/s-1667MB/s), io=186GiB (200GB), run=120020-120020msec
Sequential Read Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=read --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=5346MiB/s][r=21.4k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [R(4)][100.0%][r=5335MiB/s][r=21.3k IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=7166: Sat Mar 1 10:44:18 2025
read: IOPS=21.3k, BW=5326MiB/s (5585MB/s)(624GiB/120012msec)
slat (usec): min=19, max=13599, avg=37.56, stdev=71.48
clat (usec): min=2031, max=41378, avg=11976.43, stdev=4611.71
lat (usec): min=2061, max=41413, avg=12013.99, stdev=4611.94
clat percentiles (usec):
| 1.00th=[ 6194], 5.00th=[ 6521], 10.00th=[ 6849], 20.00th=[ 7701],
| 30.00th=[ 8717], 40.00th=[ 9765], 50.00th=[10945], 60.00th=[12387],
| 70.00th=[13960], 80.00th=[15795], 90.00th=[18482], 95.00th=[20841],
| 99.00th=[25297], 99.50th=[26870], 99.90th=[30278], 99.95th=[31851],
| 99.99th=[35390]
bw ( MiB/s): min= 4845, max= 5887, per=100.00%, avg=5326.79, stdev=46.26, samples=960
iops : min=19380, max=23548, avg=21307.17, stdev=185.05, samples=960
lat (msec) : 4=0.01%, 10=42.00%, 20=51.72%, 50=6.28%
cpu : usr=0.98%, sys=19.20%, ctx=2111084, majf=0, minf=16419
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=2556860,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=5326MiB/s (5585MB/s), 5326MiB/s-5326MiB/s (5585MB/s-5585MB/s), io=624GiB (670GB), run=120012-120012msec
Sequential Write Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=write --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=write, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
throughput-test-job: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [W(4)][2.5%][w=2672MiB/s][w=10.7k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [W(4)][100.0%][w=1957MiB/s][w=7826 IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=10598: Sat Mar 1 12:36:17 2025
write: IOPS=9031, BW=2258MiB/s (2367MB/s)(265GiB/120027msec); 0 zone resets
slat (usec): min=46, max=49191, avg=426.15, stdev=650.17
clat (msec): min=3, max=131, avg=27.91, stdev= 9.37
lat (msec): min=3, max=132, avg=28.34, stdev= 9.43
clat percentiles (msec):
| 1.00th=[ 19], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23],
| 30.00th=[ 23], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 26],
| 70.00th=[ 29], 80.00th=[ 34], 90.00th=[ 41], 95.00th=[ 47],
| 99.00th=[ 65], 99.50th=[ 72], 99.90th=[ 90], 99.95th=[ 97],
| 99.99th=[ 113]
bw ( MiB/s): min= 1350, max= 2873, per=100.00%, avg=2258.31, stdev=115.26, samples=960
iops : min= 5402, max=11494, avg=9033.22, stdev=461.05, samples=960
lat (msec) : 4=0.01%, 10=0.01%, 20=2.72%, 50=93.74%, 100=3.51%
lat (msec) : 250=0.04%
cpu : usr=1.78%, sys=33.18%, ctx=1344398, majf=0, minf=35
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,1083987,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=2258MiB/s (2367MB/s), 2258MiB/s-2258MiB/s (2367MB/s-2367MB/s), io=265GiB (284GB), run=120027-120027msec
Random Read Latency
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=latency-test-job --eta-newline=1 --size=100G
latency-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.37
Starting 1 process
Jobs: 1 (f=1): [r(1)][2.5%][r=49.5MiB/s][r=12.7k IOPS][eta 01m:57s]
...
Jobs: 1 (f=1): [r(1)][100.0%][r=49.7MiB/s][r=12.7k IOPS][eta 00m:00s]
latency-test-job: (groupid=0, jobs=1): err= 0: pid=7785: Sat Mar 1 10:53:57 2025
read: IOPS=12.6k, BW=49.3MiB/s (51.7MB/s)(5919MiB/120001msec)
slat (usec): min=5, max=122, avg= 7.87, stdev= 1.75
clat (nsec): min=1684, max=8548.7k, avg=70815.44, stdev=13044.77
lat (usec): min=32, max=8612, avg=78.69, stdev=13.26
clat percentiles (usec):
| 1.00th=[ 59], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 67],
| 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 68], 60.00th=[ 69],
| 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 80], 95.00th=[ 81],
| 99.00th=[ 83], 99.50th=[ 85], 99.90th=[ 95], 99.95th=[ 110],
| 99.99th=[ 594]
bw ( KiB/s): min=48416, max=51352, per=100.00%, avg=50521.67, stdev=519.86, samples=239
iops : min=12104, max=12838, avg=12630.42, stdev=129.97, samples=239
lat (usec) : 2=0.01%, 10=0.01%, 50=0.26%, 100=99.68%, 250=0.05%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=1.41%, sys=13.36%, ctx=1516671, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1515324,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=49.3MiB/s (51.7MB/s), 49.3MiB/s-49.3MiB/s (51.7MB/s-51.7MB/s), io=5919MiB (6207MB), run=120001-120001msec
Random Write Latency
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=latency-test-job --eta-newline=1 --size=100G
latency-test-job: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.37
Starting 1 process
Jobs: 1 (f=1): [w(1)][2.5%][w=107MiB/s][w=27.3k IOPS][eta 01m:57s]
...
Jobs: 1 (f=1): [w(1)][100.0%][w=100MiB/s][w=25.7k IOPS][eta 00m:00s]
latency-test-job: (groupid=0, jobs=1): err= 0: pid=7908: Sat Mar 1 10:58:00 2025
write: IOPS=25.7k, BW=100MiB/s (105MB/s)(11.8GiB/120001msec); 0 zone resets
slat (usec): min=8, max=2249, avg=20.42, stdev= 7.26
clat (nsec): min=621, max=7425.8k, avg=17899.41, stdev=25831.63
lat (usec): min=23, max=7468, avg=38.32, stdev=27.24
clat percentiles (usec):
| 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16],
| 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 16], 60.00th=[ 17],
| 70.00th=[ 18], 80.00th=[ 18], 90.00th=[ 20], 95.00th=[ 23],
| 99.00th=[ 51], 99.50th=[ 60], 99.90th=[ 77], 99.95th=[ 94],
| 99.99th=[ 343]
bw ( KiB/s): min=77592, max=111872, per=100.00%, avg=102782.80, stdev=7154.26, samples=240
iops : min=19398, max=27968, avg=25695.70, stdev=1788.57, samples=240
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=91.49%, 50=7.37%
lat (usec) : 100=1.04%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=3.27%, sys=56.42%, ctx=3319031, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3083484,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=11.8GiB (12.6GB), run=120001-120001msec
Random Read/Write Latency
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=latency-test-job --eta-newline=1 --size=100G
latency-test-job: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.37
Starting 1 process
Jobs: 1 (f=1): [m(1)][2.5%][r=44.1MiB/s,w=44.4MiB/s][r=11.3k,w=11.4k IOPS][eta 01m:57s]
...
Jobs: 1 (f=1): [m(1)][100.0%][r=34.0MiB/s,w=34.3MiB/s][r=8694,w=8769 IOPS][eta 00m:00s]
latency-test-job: (groupid=0, jobs=1): err= 0: pid=8011: Sat Mar 1 11:01:25 2025
read: IOPS=11.1k, BW=43.5MiB/s (45.6MB/s)(5216MiB/120001msec)
slat (usec): min=5, max=2029, avg= 8.58, stdev= 4.00
clat (nsec): min=792, max=6472.5k, avg=43578.34, stdev=22468.62
lat (usec): min=21, max=6481, avg=52.16, stdev=22.98
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27],
| 30.00th=[ 28], 40.00th=[ 45], 50.00th=[ 46], 60.00th=[ 48],
| 70.00th=[ 55], 80.00th=[ 56], 90.00th=[ 57], 95.00th=[ 60],
| 99.00th=[ 84], 99.50th=[ 95], 99.90th=[ 161], 99.95th=[ 172],
| 99.99th=[ 562]
bw ( KiB/s): min=32504, max=54960, per=100.00%, avg=44511.23, stdev=6588.94, samples=240
iops : min= 8126, max=13740, avg=11127.81, stdev=1647.24, samples=240
write: IOPS=11.1k, BW=43.4MiB/s (45.5MB/s)(5210MiB/120001msec); 0 zone resets
slat (usec): min=8, max=2065, avg=17.57, stdev=10.00
clat (nsec): min=632, max=6194.6k, avg=18935.95, stdev=29326.24
lat (usec): min=23, max=6217, avg=36.50, stdev=31.24
clat percentiles (usec):
| 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17],
| 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 18], 60.00th=[ 18],
| 70.00th=[ 19], 80.00th=[ 19], 90.00th=[ 21], 95.00th=[ 24],
| 99.00th=[ 52], 99.50th=[ 62], 99.90th=[ 80], 99.95th=[ 100],
| 99.99th=[ 701]
bw ( KiB/s): min=31680, max=54384, per=100.00%, avg=44459.30, stdev=6655.62, samples=240
iops : min= 7920, max=13596, avg=11114.83, stdev=1663.90, samples=240
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=44.42%, 50=37.28%
lat (usec) : 100=18.02%, 250=0.24%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=2.87%, sys=34.44%, ctx=2794668, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1335337,1333779,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=43.5MiB/s (45.6MB/s), 43.5MiB/s-43.5MiB/s (45.6MB/s-45.6MB/s), io=5216MiB (5470MB), run=120001-120001msec
WRITE: bw=43.4MiB/s (45.5MB/s), 43.4MiB/s-43.4MiB/s (45.5MB/s-45.5MB/s), io=5210MiB (5463MB), run=120001-120001msec
ISCSI Disk (no optimization)
Pool and disks (Target Side on TrueNAS Core)
root@tn-epyc[~]# zpool list -v test-sabrent
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
test-sabrent 3.62T 111G 3.52T - - 5% 2% 1.00x ONLINE /mnt
gptid/69e1c7cb-84e5-11ef-9adb-a8a159f24f0f 1.82T 56.2G 1.76T - - 5% 3.02% - ONLINE
gptid/69e3d451-84e5-11ef-9adb-a8a159f24f0f 1.82T 55.1G 1.76T - - 5% 2.96% - ONLINE
root@tn-epyc[~]# geom disk list nvd11
Geom name: nvd11
Providers:
1. Name: nvd11
Mediasize: 2000398934016 (1.8T)
Sectorsize: 4096
Mode: r2w2e5
descr: Sabrent SB-RKT4P-2TB
lunid: 6479a7835ac029b4
ident: 48821069800643
rotationrate: 0
fwsectors: 0
fwheads: 0
root@tn-epyc[~]# geom disk list nvd12
Geom name: nvd12
Providers:
1. Name: nvd12
Mediasize: 2000398934016 (1.8T)
Sectorsize: 4096
Mode: r2w2e5
descr: Sabrent SB-RKT4P-2TB
lunid: 6479a7835ac029b5
ident: 48821069800642
rotationrate: 0
fwsectors: 0
fwheads: 0
Random Read IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
iops-test-job: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [r(4)][2.5%][r=345MiB/s][r=88.3k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [r(4)][100.0%][r=354MiB/s][r=90.6k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=9130: Sat Mar 1 11:45:15 2025
read: IOPS=86.7k, BW=339MiB/s (355MB/s)(39.7GiB/120003msec)
slat (usec): min=2, max=12254, avg=45.00, stdev=272.93
clat (usec): min=816, max=38563, avg=11767.90, stdev=4161.60
lat (usec): min=822, max=38729, avg=11812.89, stdev=4175.74
clat percentiles (usec):
| 1.00th=[ 4047], 5.00th=[ 5735], 10.00th=[ 6783], 20.00th=[ 8160],
| 30.00th=[ 9372], 40.00th=[10290], 50.00th=[11338], 60.00th=[12387],
| 70.00th=[13566], 80.00th=[15139], 90.00th=[17171], 95.00th=[19268],
| 99.00th=[23200], 99.50th=[24773], 99.90th=[28181], 99.95th=[29492],
| 99.99th=[32113]
bw ( KiB/s): min=296424, max=408560, per=100.00%, avg=346758.09, stdev=4936.16, samples=956
iops : min=74106, max=102140, avg=86689.52, stdev=1234.05, samples=956
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.93%, 10=35.73%, 20=59.54%, 50=3.79%
cpu : usr=2.14%, sys=23.09%, ctx=10010936, majf=0, minf=1062
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=10401512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=339MiB/s (355MB/s), 339MiB/s-339MiB/s (355MB/s-355MB/s), io=39.7GiB (42.6GB), run=120003-120003msec
Disk stats (read/write):
sdd: ios=10379235/6, sectors=83113648/72, merge=9982/0, ticks=24485086/253, in_queue=24485342, util=70.21%
Random Write IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [w(4)][2.5%][w=182MiB/s][w=46.5k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [w(4)][100.0%][w=187MiB/s][w=47.9k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=9385: Sat Mar 1 11:48:38 2025
write: IOPS=48.7k, BW=190MiB/s (200MB/s)(22.3GiB/120003msec); 0 zone resets
slat (usec): min=2, max=15531, avg=80.96, stdev=451.30
clat (usec): min=1341, max=63440, avg=20928.33, stdev=6662.47
lat (usec): min=2424, max=63787, avg=21009.30, stdev=6686.44
clat percentiles (usec):
| 1.00th=[ 8455], 5.00th=[11207], 10.00th=[12911], 20.00th=[15139],
| 30.00th=[16909], 40.00th=[18744], 50.00th=[20317], 60.00th=[21890],
| 70.00th=[23987], 80.00th=[26346], 90.00th=[29754], 95.00th=[32900],
| 99.00th=[39060], 99.50th=[41681], 99.90th=[46924], 99.95th=[49021],
| 99.99th=[53740]
bw ( KiB/s): min=157360, max=240152, per=100.00%, avg=195004.08, stdev=3162.93, samples=956
iops : min=39340, max=60038, avg=48751.01, stdev=790.73, samples=956
lat (msec) : 2=0.01%, 4=0.01%, 10=2.78%, 20=45.43%, 50=51.75%
lat (msec) : 100=0.04%
cpu : usr=1.35%, sys=11.31%, ctx=6013200, majf=0, minf=47
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,5848292,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=190MiB/s (200MB/s), 190MiB/s-190MiB/s (200MB/s-200MB/s), io=22.3GiB (24.0GB), run=120003-120003msec
Disk stats (read/write):
sdd: ios=0/5842671, sectors=0/46746752, merge=0/673, ticks=0/18270822, in_queue=18271927, util=80.38%
Random Read/Write IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [m(4)][2.5%][r=119MiB/s,w=120MiB/s][r=30.5k,w=30.6k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [m(4)][100.0%][r=122MiB/s,w=123MiB/s][r=31.3k,w=31.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=9446: Sat Mar 1 11:52:03 2025
read: IOPS=31.5k, BW=123MiB/s (129MB/s)(14.4GiB/120002msec)
slat (usec): min=2, max=1276, avg= 5.26, stdev= 3.15
clat (usec): min=1487, max=49842, avg=16120.18, stdev=5028.03
lat (usec): min=1501, max=49846, avg=16125.45, stdev=5028.03
clat percentiles (usec):
| 1.00th=[ 6652], 5.00th=[ 8717], 10.00th=[10028], 20.00th=[11731],
| 30.00th=[13173], 40.00th=[14353], 50.00th=[15664], 60.00th=[16909],
| 70.00th=[18482], 80.00th=[20055], 90.00th=[22938], 95.00th=[25035],
| 99.00th=[29754], 99.50th=[31851], 99.90th=[35914], 99.95th=[37487],
| 99.99th=[40633]
bw ( KiB/s): min=104640, max=147464, per=100.00%, avg=126148.15, stdev=1829.32, samples=956
iops : min=26160, max=36866, avg=31537.04, stdev=457.33, samples=956
write: IOPS=31.5k, BW=123MiB/s (129MB/s)(14.4GiB/120002msec); 0 zone resets
slat (usec): min=2, max=12912, avg=119.30, stdev=466.60
clat (usec): min=980, max=50445, avg=16243.29, stdev=5029.64
lat (usec): min=2612, max=52993, avg=16362.59, stdev=5064.09
clat percentiles (usec):
| 1.00th=[ 6783], 5.00th=[ 8848], 10.00th=[10159], 20.00th=[11863],
| 30.00th=[13304], 40.00th=[14484], 50.00th=[15795], 60.00th=[17171],
| 70.00th=[18482], 80.00th=[20317], 90.00th=[22938], 95.00th=[25297],
| 99.00th=[30016], 99.50th=[31851], 99.90th=[35914], 99.95th=[37487],
| 99.99th=[40633]
bw ( KiB/s): min=105104, max=146632, per=100.00%, avg=126067.05, stdev=1793.06, samples=956
iops : min=26276, max=36658, avg=31516.76, stdev=448.26, samples=956
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=9.37%, 20=69.47%, 50=21.14%
lat (msec) : 100=0.01%
cpu : usr=1.75%, sys=16.06%, ctx=7663026, majf=0, minf=53
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=3783144,3780896,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=14.4GiB (15.5GB), run=120002-120002msec
WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=14.4GiB (15.5GB), run=120002-120002msec
Disk stats (read/write):
sdd: ios=3779570/3777296, sectors=30238248/30220224, merge=211/232, ticks=9183070/9208916, in_queue=18392752, util=77.24%
Sequential Read IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=read --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=513MiB/s][r=131k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [R(4)][100.0%][r=835MiB/s][r=214k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=9513: Sat Mar 1 11:55:56 2025
read: IOPS=165k, BW=646MiB/s (677MB/s)(75.7GiB/120002msec)
slat (usec): min=2, max=1548, avg= 3.52, stdev= 3.85
clat (usec): min=79, max=256475, avg=6189.15, stdev=18027.43
lat (usec): min=86, max=256478, avg=6192.67, stdev=18027.41
clat percentiles (usec):
| 1.00th=[ 1516], 5.00th=[ 1713], 10.00th=[ 1811], 20.00th=[ 1926],
| 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2147], 60.00th=[ 2245],
| 70.00th=[ 2343], 80.00th=[ 2474], 90.00th=[ 2802], 95.00th=[ 3982],
| 99.00th=[ 86508], 99.50th=[ 86508], 99.90th=[ 87557], 99.95th=[170918],
| 99.99th=[170918]
bw ( KiB/s): min=197584, max=1856648, per=99.95%, avg=661081.24, stdev=62427.94, samples=956
iops : min=49396, max=464162, avg=165270.35, stdev=15606.98, samples=956
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=28.30%, 4=66.69%, 10=0.09%, 20=0.01%, 50=0.42%
lat (msec) : 100=4.39%, 250=0.08%, 500=0.01%
cpu : usr=2.78%, sys=14.86%, ctx=485135, majf=0, minf=1078
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=19841859,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=646MiB/s (677MB/s), 646MiB/s-646MiB/s (677MB/s-677MB/s), io=75.7GiB (81.3GB), run=120002-120002msec
Disk stats (read/write):
sdd: ios=762501/0, sectors=158644368/0, merge=19068802/0, ticks=4152660/0, in_queue=4152661, util=77.32%
Sequential Write IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=write --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [W(4)][2.5%][w=723MiB/s][w=185k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [W(4)][100.0%][w=812MiB/s][w=208k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=9769: Sat Mar 1 12:01:15 2025
write: IOPS=208k, BW=812MiB/s (851MB/s)(95.1GiB/120005msec); 0 zone resets
slat (usec): min=2, max=1546, avg= 3.21, stdev= 2.33
clat (usec): min=1010, max=62690, avg=4925.34, stdev=684.32
lat (usec): min=1014, max=62693, avg=4928.55, stdev=684.33
clat percentiles (usec):
| 1.00th=[ 3785], 5.00th=[ 4146], 10.00th=[ 4293], 20.00th=[ 4490],
| 30.00th=[ 4621], 40.00th=[ 4752], 50.00th=[ 4817], 60.00th=[ 4948],
| 70.00th=[ 5080], 80.00th=[ 5276], 90.00th=[ 5604], 95.00th=[ 5932],
| 99.00th=[ 6980], 99.50th=[ 7439], 99.90th=[ 8455], 99.95th=[ 8979],
| 99.99th=[11731]
bw ( KiB/s): min=726584, max=862712, per=100.00%, avg=831383.50, stdev=4430.42, samples=956
iops : min=181646, max=215678, avg=207845.88, stdev=1107.61, samples=956
lat (msec) : 2=0.01%, 4=2.57%, 10=97.41%, 20=0.01%, 100=0.01%
cpu : usr=3.79%, sys=17.23%, ctx=691693, majf=0, minf=42
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,24931262,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=812MiB/s (851MB/s), 812MiB/s-812MiB/s (851MB/s-851MB/s), io=95.1GiB (102GB), run=120005-120005msec
Disk stats (read/write):
sdd: ios=0/875260, sectors=0/199270600, merge=0/24034507, ticks=0/4042132, in_queue=4043142, util=89.00%
Random Read Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randread --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
throughput-test-job: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [r(4)][3.3%][r=1983MiB/s][r=7931 IOPS][eta 01m:56s]
...
Jobs: 4 (f=4): [r(4)][56.5%][r=1820MiB/s][r=7280 IOPS][eta 01m:33s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=9931: Sat Mar 1 12:06:12 2025
read: IOPS=7859, BW=1965MiB/s (2060MB/s)(230GiB/120028msec)
slat (usec): min=12, max=108689, avg=187.62, stdev=1778.31
clat (msec): min=3, max=284, avg=32.38, stdev=24.58
lat (msec): min=3, max=284, avg=32.56, stdev=24.72
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 15],
| 30.00th=[ 20], 40.00th=[ 24], 50.00th=[ 29], 60.00th=[ 34],
| 70.00th=[ 39], 80.00th=[ 43], 90.00th=[ 53], 95.00th=[ 72],
| 99.00th=[ 130], 99.50th=[ 140], 99.90th=[ 161], 99.95th=[ 205],
| 99.99th=[ 228]
bw ( MiB/s): min= 810, max= 2679, per=100.00%, avg=1965.59, stdev=100.99, samples=960
iops : min= 3242, max=10718, avg=7862.07, stdev=403.95, samples=960
lat (msec) : 4=0.76%, 10=11.36%, 20=19.79%, 50=56.74%, 100=7.25%
lat (msec) : 250=4.10%, 500=0.01%
cpu : usr=0.42%, sys=4.84%, ctx=1682947, majf=0, minf=16429
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=943395,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1965MiB/s (2060MB/s), 1965MiB/s-1965MiB/s (2060MB/s-2060MB/s), io=230GiB (247GB), run=120028-120028msec
Disk stats (read/write):
sdd: ios=942259/11, sectors=482945024/136, merge=1013/0, ticks=26345738/564, in_queue=26346313, util=85.54%
Random Write Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randwrite --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=randwrite, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [w(4)][2.5%][w=934MiB/s][w=3736 IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [w(4)][100.0%][w=953MiB/s][w=3812 IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=9992: Sat Mar 1 12:09:04 2025
write: IOPS=3752, BW=938MiB/s (984MB/s)(110GiB/120044msec); 0 zone resets
slat (usec): min=12, max=95223, avg=646.80, stdev=3730.50
clat (msec): min=7, max=337, avg=67.56, stdev=46.52
lat (msec): min=7, max=369, avg=68.21, stdev=47.30
clat percentiles (msec):
| 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 20], 20.00th=[ 30],
| 30.00th=[ 40], 40.00th=[ 49], 50.00th=[ 58], 60.00th=[ 67],
| 70.00th=[ 75], 80.00th=[ 100], 90.00th=[ 138], 95.00th=[ 165],
| 99.00th=[ 215], 99.50th=[ 232], 99.90th=[ 271], 99.95th=[ 288],
| 99.99th=[ 317]
bw ( KiB/s): min=323584, max=1570304, per=100.00%, avg=961032.53, stdev=96652.79, samples=960
iops : min= 1264, max= 6134, avg=3754.03, stdev=377.55, samples=960
lat (msec) : 10=1.30%, 20=9.49%, 50=31.05%, 100=38.48%, 250=19.43%
lat (msec) : 500=0.25%
cpu : usr=0.78%, sys=2.13%, ctx=754875, majf=0, minf=38
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,450484,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=938MiB/s (984MB/s), 938MiB/s-938MiB/s (984MB/s-984MB/s), io=110GiB (118GB), run=120044-120044msec
Disk stats (read/write):
sdd: ios=0/449599, sectors=0/230340288, merge=0/320, ticks=0/19479932, in_queue=19480981, util=88.07%
Random Read/Write Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randrw --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [m(4)][2.5%][r=682MiB/s,w=721MiB/s][r=2726,w=2882 IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [m(4)][100.0%][r=717MiB/s,w=715MiB/s][r=2866,w=2861 IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=10054: Sat Mar 1 12:12:12 2025
read: IOPS=2827, BW=707MiB/s (741MB/s)(82.9GiB/120030msec)
slat (usec): min=11, max=521, avg=20.00, stdev= 5.56
clat (msec): min=3, max=196, avg=44.54, stdev=25.35
lat (msec): min=3, max=196, avg=44.56, stdev=25.35
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 11], 10.00th=[ 15], 20.00th=[ 23],
| 30.00th=[ 29], 40.00th=[ 36], 50.00th=[ 42], 60.00th=[ 47],
| 70.00th=[ 54], 80.00th=[ 64], 90.00th=[ 80], 95.00th=[ 93],
| 99.00th=[ 121], 99.50th=[ 132], 99.90th=[ 155], 99.95th=[ 165],
| 99.99th=[ 178]
bw ( KiB/s): min=385024, max=1032192, per=100.00%, avg=724213.64, stdev=28294.00, samples=957
iops : min= 1504, max= 4032, avg=2828.96, stdev=110.52, samples=957
write: IOPS=2828, BW=707MiB/s (742MB/s)(82.9GiB/120030msec); 0 zone resets
slat (usec): min=11, max=79530, avg=1087.45, stdev=4302.40
clat (msec): min=3, max=200, avg=44.85, stdev=25.39
lat (msec): min=3, max=224, avg=45.94, stdev=25.99
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 11], 10.00th=[ 16], 20.00th=[ 23],
| 30.00th=[ 30], 40.00th=[ 36], 50.00th=[ 42], 60.00th=[ 47],
| 70.00th=[ 55], 80.00th=[ 65], 90.00th=[ 80], 95.00th=[ 93],
| 99.00th=[ 122], 99.50th=[ 133], 99.90th=[ 155], 99.95th=[ 163],
| 99.99th=[ 180]
bw ( KiB/s): min=408576, max=1041920, per=100.00%, avg=724467.93, stdev=27746.27, samples=957
iops : min= 1596, max= 4070, avg=2829.95, stdev=108.38, samples=957
lat (msec) : 4=0.01%, 10=4.28%, 20=12.24%, 50=48.33%, 100=31.75%
lat (msec) : 250=3.40%
cpu : usr=0.66%, sys=3.49%, ctx=917525, majf=0, minf=49
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=339410,339556,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=707MiB/s (741MB/s), 707MiB/s-707MiB/s (741MB/s-741MB/s), io=82.9GiB (89.0GB), run=120030-120030msec
WRITE: bw=707MiB/s (742MB/s), 707MiB/s-707MiB/s (742MB/s-742MB/s), io=82.9GiB (89.0GB), run=120030-120030msec
Disk stats (read/write):
sdd: ios=338915/339083, sectors=173573632/173650112, merge=103/111, ticks=9384131/9412950, in_queue=18798250, util=87.38%
Sequential Read Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=read --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=96.2MiB/s][r=384 IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [R(4)][100.0%][r=98.3MiB/s][r=393 IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=10117: Sat Mar 1 12:15:17 2025
read: IOPS=317, BW=79.3MiB/s (83.1MB/s)(9546MiB/120379msec)
slat (usec): min=11, max=4487, avg=16.75, stdev=34.05
clat (msec): min=4, max=1826, avg=806.94, stdev=232.63
lat (msec): min=4, max=1826, avg=806.96, stdev=232.63
clat percentiles (msec):
| 1.00th=[ 279], 5.00th=[ 447], 10.00th=[ 527], 20.00th=[ 617],
| 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 869],
| 70.00th=[ 911], 80.00th=[ 995], 90.00th=[ 1083], 95.00th=[ 1167],
| 99.00th=[ 1368], 99.50th=[ 1469], 99.90th=[ 1687], 99.95th=[ 1821],
| 99.99th=[ 1821]
bw ( KiB/s): min= 2048, max=228864, per=100.00%, avg=86673.56, stdev=9905.08, samples=899
iops : min= 8, max= 894, avg=338.56, stdev=38.69, samples=899
lat (msec) : 10=0.26%, 20=0.01%, 100=0.06%, 250=0.38%, 500=9.05%
lat (msec) : 750=33.49%, 1000=39.39%, 2000=17.36%
cpu : usr=0.01%, sys=0.12%, ctx=5693, majf=0, minf=16430
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=38183,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=79.3MiB/s (83.1MB/s), 79.3MiB/s-79.3MiB/s (83.1MB/s-83.1MB/s), io=9546MiB (10.0GB), run=120379-120379msec
Disk stats (read/write):
sdd: ios=5124/2, sectors=19429888/40, merge=33025/0, ticks=3986837/1965, in_queue=3989620, util=90.20%
Sequential Write Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=write --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=write, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [W(4)][2.5%][w=952MiB/s][w=3807 IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [W(4)][100.0%][w=1000MiB/s][w=3998 IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=10211: Sat Mar 1 12:20:55 2025
write: IOPS=3927, BW=982MiB/s (1030MB/s)(115GiB/120058msec); 0 zone resets
slat (usec): min=10, max=383, avg=18.60, stdev= 5.06
clat (usec): min=1196, max=162247, avg=65154.29, stdev=5006.14
lat (usec): min=1284, max=162262, avg=65172.88, stdev=5005.79
clat percentiles (msec):
| 1.00th=[ 55], 5.00th=[ 58], 10.00th=[ 60], 20.00th=[ 62],
| 30.00th=[ 63], 40.00th=[ 64], 50.00th=[ 66], 60.00th=[ 67],
| 70.00th=[ 68], 80.00th=[ 69], 90.00th=[ 71], 95.00th=[ 73],
| 99.00th=[ 79], 99.50th=[ 82], 99.90th=[ 95], 99.95th=[ 109],
| 99.99th=[ 148]
bw ( KiB/s): min=883712, max=1114112, per=100.00%, avg=1005845.33, stdev=11009.80, samples=960
iops : min= 3452, max= 4352, avg=3929.08, stdev=43.01, samples=960
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.16%
lat (msec) : 100=99.75%, 250=0.07%
cpu : usr=0.70%, sys=1.25%, ctx=66008, majf=0, minf=50
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,471553,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=982MiB/s (1030MB/s), 982MiB/s-982MiB/s (1030MB/s-1030MB/s), io=115GiB (124GB), run=120058-120058msec
Disk stats (read/write):
sdd: ios=0/62819, sectors=0/241066744, merge=0/408280, ticks=0/3986910, in_queue=3988584, util=97.47%
Random Read Latency
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=latency-test-job --eta-newline=1 --size=100G
latency-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.37
Starting 1 process
Jobs: 1 (f=1): [r(1)][2.5%][r=66.8MiB/s][r=17.1k IOPS][eta 01m:57s]
...
Jobs: 1 (f=1): [r(1)][100.0%][r=66.4MiB/s][r=17.0k IOPS][eta 00m:00s]
latency-test-job: (groupid=0, jobs=1): err= 0: pid=10328: Sat Mar 1 12:24:03 2025
read: IOPS=17.8k, BW=69.5MiB/s (72.9MB/s)(8341MiB/120001msec)
slat (usec): min=5, max=206, avg= 6.66, stdev= 1.30
clat (nsec): min=1042, max=16736k, avg=49028.40, stdev=17194.54
lat (usec): min=40, max=16741, avg=55.69, stdev=17.31
clat percentiles (usec):
| 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47],
| 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 49],
| 70.00th=[ 49], 80.00th=[ 50], 90.00th=[ 54], 95.00th=[ 60],
| 99.00th=[ 73], 99.50th=[ 77], 99.90th=[ 90], 99.95th=[ 106],
| 99.99th=[ 482]
bw ( KiB/s): min=61744, max=78344, per=100.00%, avg=71207.73, stdev=2159.91, samples=239
iops : min=15436, max=19586, avg=17801.93, stdev=539.98, samples=239
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=79.80%, 100=20.14%
lat (usec) : 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 10=0.01%, 20=0.01%
cpu : usr=1.64%, sys=13.53%, ctx=2136141, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2135223,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=69.5MiB/s (72.9MB/s), 69.5MiB/s-69.5MiB/s (72.9MB/s-72.9MB/s), io=8341MiB (8746MB), run=120001-120001msec
Disk stats (read/write):
sdd: ios=2133358/2, sectors=17066864/40, merge=0/0, ticks=108385/16, in_queue=108401, util=87.83%
Random Write Latency
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=latency-test-job --eta-newline=1 --size=100G
latency-test-job: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.37
Starting 1 process
Jobs: 1 (f=1): [w(1)][2.5%][w=53.6MiB/s][w=13.7k IOPS][eta 01m:57s]
...
Jobs: 1 (f=1): [w(1)][100.0%][w=51.1MiB/s][w=13.1k IOPS][eta 00m:00s]
latency-test-job: (groupid=0, jobs=1): err= 0: pid=10408: Sat Mar 1 12:27:32 2025
write: IOPS=13.5k, BW=52.8MiB/s (55.4MB/s)(6341MiB/120001msec); 0 zone resets
slat (usec): min=5, max=454, avg= 7.55, stdev= 1.96
clat (usec): min=2, max=9987, avg=65.75, stdev=34.26
lat (usec): min=53, max=9994, avg=73.30, stdev=34.43
clat percentiles (usec):
| 1.00th=[ 53], 5.00th=[ 57], 10.00th=[ 59], 20.00th=[ 60],
| 30.00th=[ 62], 40.00th=[ 63], 50.00th=[ 64], 60.00th=[ 64],
| 70.00th=[ 66], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 86],
| 99.00th=[ 104], 99.50th=[ 121], 99.90th=[ 198], 99.95th=[ 277],
| 99.99th=[ 1926]
bw ( KiB/s): min=48328, max=65352, per=100.00%, avg=54107.90, stdev=2745.37, samples=240
iops : min=12082, max=16338, avg=13526.98, stdev=686.34, samples=240
lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.14%, 100=98.51%
lat (usec) : 250=1.29%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=1.71%, sys=13.02%, ctx=1626737, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1623237,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=52.8MiB/s (55.4MB/s), 52.8MiB/s-52.8MiB/s (55.4MB/s-55.4MB/s), io=6341MiB (6649MB), run=120001-120001msec
Disk stats (read/write):
sdd: ios=0/1621806, sectors=0/12974448, merge=0/0, ticks=0/103638, in_queue=103809, util=85.10%
Random Read/Write Latency
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=latency-test-job --eta-newline=1 --size=100G
latency-test-job: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.37
Starting 1 process
Jobs: 1 (f=1): [m(1)][2.5%][r=30.0MiB/s,w=29.8MiB/s][r=7688,w=7631 IOPS][eta 01m:57s]
...
Jobs: 1 (f=1): [m(1)][100.0%][r=29.7MiB/s,w=29.7MiB/s][r=7603,w=7604 IOPS][eta 00m:00s]
latency-test-job: (groupid=0, jobs=1): err= 0: pid=10485: Sat Mar 1 12:30:28 2025
read: IOPS=7665, BW=29.9MiB/s (31.4MB/s)(3593MiB/120001msec)
slat (usec): min=5, max=943, avg= 7.34, stdev= 2.01
clat (usec): min=2, max=1279, avg=50.09, stdev= 7.92
lat (usec): min=41, max=1319, avg=57.43, stdev= 8.47
clat percentiles (usec):
| 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 47],
| 30.00th=[ 48], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 50],
| 70.00th=[ 51], 80.00th=[ 52], 90.00th=[ 55], 95.00th=[ 61],
| 99.00th=[ 77], 99.50th=[ 81], 99.90th=[ 97], 99.95th=[ 113],
| 99.99th=[ 241]
bw ( KiB/s): min=27496, max=35720, per=100.00%, avg=30671.97, stdev=1502.42, samples=239
iops : min= 6874, max= 8930, avg=7667.99, stdev=375.61, samples=239
write: IOPS=7652, BW=29.9MiB/s (31.3MB/s)(3587MiB/120001msec); 0 zone resets
slat (usec): min=5, max=184, avg= 7.69, stdev= 1.80
clat (nsec): min=1313, max=9946.0k, avg=64243.86, stdev=28755.42
lat (usec): min=52, max=9952, avg=71.94, stdev=28.94
clat percentiles (usec):
| 1.00th=[ 53], 5.00th=[ 57], 10.00th=[ 58], 20.00th=[ 60],
| 30.00th=[ 61], 40.00th=[ 62], 50.00th=[ 63], 60.00th=[ 64],
| 70.00th=[ 65], 80.00th=[ 67], 90.00th=[ 69], 95.00th=[ 78],
| 99.00th=[ 99], 99.50th=[ 109], 99.90th=[ 178], 99.95th=[ 221],
| 99.99th=[ 1139]
bw ( KiB/s): min=27424, max=36384, per=100.00%, avg=30623.97, stdev=1483.10, samples=239
iops : min= 6856, max= 9096, avg=7655.99, stdev=370.77, samples=239
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=31.79%
lat (usec) : 100=67.70%, 250=0.48%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=1.82%, sys=13.85%, ctx=1840737, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=919833,918351,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=29.9MiB/s (31.4MB/s), 29.9MiB/s-29.9MiB/s (31.4MB/s-31.4MB/s), io=3593MiB (3768MB), run=120001-120001msec
WRITE: bw=29.9MiB/s (31.3MB/s), 29.9MiB/s-29.9MiB/s (31.3MB/s-31.3MB/s), io=3587MiB (3762MB), run=120001-120001msec
Disk stats (read/write):
sdd: ios=919031/917550, sectors=7352248/7340400, merge=0/0, ticks=47818/57391, in_queue=105327, util=86.29%
There will be another post with ISCSI disk with some Chelsio optimization.
Best Regards,
Antonio
I have TrueNAS running on my old MacBook Pro. 2 of the 3 attached drives are connected via the TB ports on the Mac.
I also have an iSCSI share mounted from TN to my desktop Mac.
So these 2 points are not correct.
Interesting.I doubt you’d be able to saturate 100gbe but at least it won’t be a bottleneck.
I’d use Scale bare metal for a new system. I used to run FreeNAS virtualized under ESXi. It worked just fine, but I’d rather keep the two roles separate.
I used to load my steam library from the NAS. I did this because my internet connection wasn’t that great, and it allowed me to have my entire library downloaded and up-to-date at all times. It seemed to play just fine, but if there was any game that was ‘slow’ from the NAS I would just move the install to my local storage through Steam. I can’t remember how I had it mounted on Windows, but I’m pretty sure it wasn’t iSCSI.
Good for you running an unsupported setup (ZFS direct access to disks).
Unfortunately that is not what OP is talking about. It was about how to access data from TrueNAS over Thunderbolt, not connecting drives to TrueNAS.
iSCSI is a network protocol. Sure, you can have a Thunderbolt NIC that uses iSCSI to access a share. But that is not the same as accessing an external SSD over Thunderbolt.
1 Like
Yes, good for me that it works right now. This is just a test machine to learn a bit about TrueNAS while I build a better system.
Unfortunately, your exact quote was, “TrueNAS does not support Thunderbolt.” which is demonstrably false since I’m using Thunderbolt on TrueNAS, despite what you say.
Again, you’re twisting your words in order to prove that you’re ‘not wrong’. Your words were (regarding mounting iSCSI on MacOS), “No support for iSCSI, so I would go with SMB or NFS.”
I can tell you straight up that I have an iSCSI share on my TN unit mounted as a volume on my desktop Mac, they are connected only via Ethernet.
It’s okay to be wrong. We all are occasionally. It’s best to just admit it and move on.
No. TrueNAS does not support Thunderbolt. Linux supports Thunderbolt and that is why you “use” Thunderbolt on your system.
Ok, here I could have been more specific. There is no official iSCSI support from macOS. Sure you can still mount iSCSI on macOS by installing some random github project. I would still not recommend it.
1 Like
Ah I see. And you can use TrueNAS without Linux? Good to know. The average person is not going to differentiate between TN and Linux in this sense.
Thanks for clearing up what you originally meant, but really, it’s a distinction without a difference.
macOS is a product from Apple Inc, so you meant to say “no official iSCSI support from Apple.” And that’s really not a problem, there are a lot of things that aren’t supported out of the box on any OS you can think of. Thank goodness for 3rd party developers or we’d all be SOL now and then when we wanted to do something that the OS maker decided wasn’t worth their time to implement.
Here’s your ‘random GitHub project’. Oh the horror…
Ever heard of iTunes? Here’s it’s origin, it came from a small unknown company.
Well, I would. BTW, great use of emoji. It really makes your points for you.
A: I never maid that claim
B: But yeah you can, it is called CORE
Dude, I know you want to split hairs because you have a wonky setup by attacking drives over TB. Good for you. But OP was talking about accessing the NAS over Thunderbolt. Something like this:
You are spot on!
It is, if you want support.
Nice, daemon tools! I have not seen that software for ages! Loved it in the good old days when Windows and macOS were not able to mount ISOs!
Thank you for bringing up memories 
Cheers 
Dude-ette, you’re the one splitting hairs. You can’t run TrueNAS without an underlying OS, and the same thing applies that I already said.
Maybe your maid made that claim and I just misunderstood.
Quite wonky, but as I said. I set it up to test as I’m new to TrueNAS and it’s the only hardware I had available that wasn’t doing anything else.
I definitely never attacked my drives though, that would be self defeating.
That’s fine, but don’t make a blanket statement that Thunderbolt doesn’t work.
You get support from the company that made the product. That’s how it’s always worked.
And you can love it now for it’s flawless support of iSCSI on macOS.
Isn’t it fun arguing over the Internet? You’ve changed my mind, I’ve changed yours… Right?
That is right, and the underlying OS of CORE is not Linux.
Again, there are two parts to Thunderbolt.
Accessing TrueNAS as a DAS over thunderbolt or attaching drives to TrueNAS.
First one isn’t supported, second one also not supported, since TrueNAS uses ZFS and ZFS does want direct access to your drives. If you wanna hear it from someone else to belive it.
OP is talking about editing videos on macOS. I would only do that over deamon tools iSCSI, if you love jank and dislike performance.
It really has been fun, since you are really funny. And I love to change my mind about stuff or learn something new when presented with new evidence. That wasn’t the case here, but I still had fun. Take care!
Yes, and it seems even based on your own link that CORE works better with TB than Scale, but you’re right, I should have been more clear with my original statement. Same thing I was asking of you.
And supported or not, sometimes TB works, so don’t make a blanket statement that it doesn’t work
Oh I see, you make opinions couched as statement of fact of other’s experiences even when you admit you haven’t used a solution for “ages.”
Then you’re being intentionally close minded. Thanks for showing me I’m wasting my time.
ISCSI Disk with Chelsio Optimization
For Target side see “iSCSI PDU Offload Target” in:
https://service.chelsio.com/beta/drivers/ChelsioUwire-FBSD-3.3.0.1/Chelsio-UnifiedWire-FreeBSD-UserGuide.pdf
For Initiator side see “iSCSI PDU Offload Initiator” in:
https://service.chelsio.com/store2/T5//Unified%20Wire%20(NIC,TOE,vNIC,iWARP,iSCSI,FCoE,WD-UDP)//Linux//ChelsioUwire-3.19.0.3/Chelsio-UnifiedWire-Linux-UserGuide.pdf
Random Read IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
iops-test-job: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [r(4)][3.3%][r=563MiB/s][r=144k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [r(4)][100.0%][r=568MiB/s][r=145k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=12772: Sat Mar 1 13:49:04 2025
read: IOPS=147k, BW=572MiB/s (600MB/s)(67.1GiB/120002msec)
slat (usec): min=2, max=10413, avg=25.84, stdev=166.73
clat (usec): min=525, max=33047, avg=6961.23, stdev=2721.05
lat (usec): min=531, max=33057, avg=6987.06, stdev=2729.07
clat percentiles (usec):
| 1.00th=[ 2245], 5.00th=[ 3195], 10.00th=[ 3818], 20.00th=[ 4686],
| 30.00th=[ 5342], 40.00th=[ 5932], 50.00th=[ 6587], 60.00th=[ 7308],
| 70.00th=[ 8029], 80.00th=[ 9110], 90.00th=[10552], 95.00th=[11994],
| 99.00th=[14877], 99.50th=[16188], 99.90th=[19006], 99.95th=[20055],
| 99.99th=[22938]
bw ( KiB/s): min=498584, max=700624, per=100.00%, avg=586417.29, stdev=7931.13, samples=956
iops : min=124646, max=175156, avg=146604.31, stdev=1982.78, samples=956
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.51%, 4=11.44%, 10=74.97%, 20=13.02%, 50=0.05%
cpu : usr=3.31%, sys=45.37%, ctx=14821884, majf=0, minf=1068
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=17584952,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=572MiB/s (600MB/s), 572MiB/s-572MiB/s (600MB/s-600MB/s), io=67.1GiB (72.0GB), run=120002-120002msec
Disk stats (read/write):
sdd: ios=17552230/7, sectors=140641768/104, merge=27991/0, ticks=26082816/243, in_queue=26083059, util=74.17%
Random Write IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [w(4)][2.5%][w=640MiB/s][w=164k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [w(4)][100.0%][w=647MiB/s][w=166k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=12843: Sat Mar 1 13:51:56 2025
write: IOPS=165k, BW=644MiB/s (675MB/s)(75.5GiB/120001msec); 0 zone resets
slat (usec): min=2, max=53774, avg=22.25, stdev=88.58
clat (usec): min=41, max=220514, avg=6187.46, stdev=2314.89
lat (usec): min=80, max=220536, avg=6209.71, stdev=2318.05
clat percentiles (usec):
| 1.00th=[ 3195], 5.00th=[ 3982], 10.00th=[ 4424], 20.00th=[ 4948],
| 30.00th=[ 5342], 40.00th=[ 5669], 50.00th=[ 5997], 60.00th=[ 6325],
| 70.00th=[ 6718], 80.00th=[ 7177], 90.00th=[ 7963], 95.00th=[ 8717],
| 99.00th=[10814], 99.50th=[13173], 99.90th=[35390], 99.95th=[44827],
| 99.99th=[66323]
bw ( KiB/s): min=426320, max=769760, per=100.00%, avg=659770.61, stdev=9002.01, samples=956
iops : min=106580, max=192440, avg=164942.67, stdev=2250.51, samples=956
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.06%, 4=5.11%, 10=93.15%, 20=1.42%, 50=0.18%
lat (msec) : 100=0.03%, 250=0.01%
cpu : usr=4.46%, sys=54.71%, ctx=11109860, majf=0, minf=48
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,19785363,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=644MiB/s (675MB/s), 644MiB/s-644MiB/s (675MB/s-675MB/s), io=75.5GiB (81.0GB), run=120001-120001msec
Disk stats (read/write):
sdd: ios=0/19745764, sectors=0/158130864, merge=0/20593, ticks=0/18509676, in_queue=18512748, util=69.69%
Random Read/Write IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [m(4)][2.5%][r=291MiB/s,w=289MiB/s][r=74.4k,w=74.0k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [m(4)][100.0%][r=299MiB/s,w=299MiB/s][r=76.6k,w=76.5k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=12884: Sat Mar 1 13:54:50 2025
read: IOPS=76.2k, BW=298MiB/s (312MB/s)(34.9GiB/120002msec)
slat (usec): min=2, max=1602, avg=11.19, stdev=10.21
clat (usec): min=10, max=20747, avg=6054.96, stdev=2138.96
lat (usec): min=41, max=20768, avg=6066.15, stdev=2139.54
clat percentiles (usec):
| 1.00th=[ 1004], 5.00th=[ 2769], 10.00th=[ 3523], 20.00th=[ 4293],
| 30.00th=[ 4883], 40.00th=[ 5407], 50.00th=[ 5932], 60.00th=[ 6456],
| 70.00th=[ 7046], 80.00th=[ 7767], 90.00th=[ 8848], 95.00th=[ 9765],
| 99.00th=[11731], 99.50th=[12387], 99.90th=[14091], 99.95th=[14877],
| 99.99th=[16581]
bw ( KiB/s): min=113336, max=421264, per=100.00%, avg=304911.77, stdev=9588.66, samples=956
iops : min=28334, max=105316, avg=76227.94, stdev=2397.16, samples=956
write: IOPS=76.1k, BW=297MiB/s (312MB/s)(34.9GiB/120002msec); 0 zone resets
slat (usec): min=2, max=7019, avg=34.70, stdev=153.63
clat (usec): min=36, max=409295, avg=7343.01, stdev=8231.25
lat (usec): min=60, max=409331, avg=7377.71, stdev=8234.22
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7],
| 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 10], 95.00th=[ 12],
| 99.00th=[ 30], 99.50th=[ 46], 99.90th=[ 123], 99.95th=[ 159],
| 99.99th=[ 300]
bw ( KiB/s): min=120216, max=425848, per=100.00%, avg=304713.74, stdev=9558.36, samples=956
iops : min=30054, max=106462, avg=76178.43, stdev=2389.59, samples=956
lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.06%, 500=0.18%
lat (usec) : 750=0.18%, 1000=0.17%
lat (msec) : 2=0.74%, 4=11.84%, 10=80.10%, 20=5.66%, 50=0.84%
lat (msec) : 100=0.15%, 250=0.07%, 500=0.01%
cpu : usr=4.06%, sys=48.28%, ctx=11507255, majf=0, minf=54
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=9142109,9136556,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=298MiB/s (312MB/s), 298MiB/s-298MiB/s (312MB/s-312MB/s), io=34.9GiB (37.4GB), run=120002-120002msec
WRITE: bw=297MiB/s (312MB/s), 297MiB/s-297MiB/s (312MB/s-312MB/s), io=34.9GiB (37.4GB), run=120002-120002msec
Disk stats (read/write):
sdd: ios=9128982/9123506, sectors=73074368/73030624, merge=5314/5322, ticks=9235140/9294797, in_queue=18530666, util=70.35%
Sequential Read IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=read --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=1811MiB/s][r=464k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [R(4)][100.0%][r=1849MiB/s][r=473k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=12951: Sat Mar 1 13:57:50 2025
read: IOPS=468k, BW=1826MiB/s (1915MB/s)(214GiB/120001msec)
slat (usec): min=2, max=4945, avg= 7.44, stdev= 9.44
clat (usec): min=32, max=18034, avg=2182.23, stdev=389.20
lat (usec): min=54, max=18037, avg=2189.67, stdev=389.85
clat percentiles (usec):
| 1.00th=[ 1352], 5.00th=[ 1582], 10.00th=[ 1729], 20.00th=[ 1893],
| 30.00th=[ 1991], 40.00th=[ 2073], 50.00th=[ 2180], 60.00th=[ 2278],
| 70.00th=[ 2376], 80.00th=[ 2474], 90.00th=[ 2638], 95.00th=[ 2769],
| 99.00th=[ 3097], 99.50th=[ 3294], 99.90th=[ 4015], 99.95th=[ 4555],
| 99.99th=[ 7504]
bw ( MiB/s): min= 1559, max= 2016, per=100.00%, avg=1826.45, stdev=14.87, samples=960
iops : min=399260, max=516120, avg=467571.43, stdev=3806.07, samples=960
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.06%
lat (usec) : 1000=0.24%
lat (msec) : 2=31.66%, 4=67.94%, 10=0.10%, 20=0.01%
cpu : usr=5.80%, sys=69.84%, ctx=5037909, majf=0, minf=1070
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=56108574,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=1826MiB/s (1915MB/s), 1826MiB/s-1826MiB/s (1915MB/s-1915MB/s), io=214GiB (230GB), run=120001-120001msec
Disk stats (read/write):
sdd: ios=16832368/6, sectors=448498032/72, merge=39229944/0, ticks=3819754/210, in_queue=3819964, util=63.92%
Sequential Write IOPS
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=write --bs=4k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --size=10G
iops-test-job: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [W(4)][2.5%][w=1696MiB/s][w=434k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [W(4)][100.0%][w=1906MiB/s][w=488k IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=13048: Sat Mar 1 14:00:50 2025
write: IOPS=478k, BW=1866MiB/s (1957MB/s)(219GiB/120001msec); 0 zone resets
slat (usec): min=2, max=2858, avg= 7.08, stdev= 9.90
clat (usec): min=46, max=470094, avg=2135.63, stdev=708.64
lat (usec): min=68, max=470128, avg=2142.71, stdev=709.09
clat percentiles (usec):
| 1.00th=[ 1205], 5.00th=[ 1500], 10.00th=[ 1631], 20.00th=[ 1778],
| 30.00th=[ 1876], 40.00th=[ 1975], 50.00th=[ 2057], 60.00th=[ 2147],
| 70.00th=[ 2245], 80.00th=[ 2442], 90.00th=[ 2737], 95.00th=[ 2999],
| 99.00th=[ 3654], 99.50th=[ 4293], 99.90th=[ 7439], 99.95th=[ 9110],
| 99.99th=[13829]
bw ( MiB/s): min= 1550, max= 2195, per=100.00%, avg=1866.91, stdev=32.46, samples=956
iops : min=397044, max=562140, avg=477929.31, stdev=8309.52, samples=956
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.03%, 750=0.12%
lat (usec) : 1000=0.29%
lat (msec) : 2=43.10%, 4=55.82%, 10=0.59%, 20=0.03%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=7.31%, sys=66.52%, ctx=7479668, majf=0, minf=44
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=0,57337172,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=1866MiB/s (1957MB/s), 1866MiB/s-1866MiB/s (1957MB/s-1957MB/s), io=219GiB (235GB), run=120001-120001msec
Disk stats (read/write):
sdd: ios=0/12888160, sectors=0/458279064, merge=0/44397163, ticks=0/4114222, in_queue=4127126, util=66.62%
Random Read Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randread --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=randread, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
throughput-test-job: Laying out IO file (1 file / 102400MiB)
Jobs: 4 (f=4): [r(4)][3.3%][r=4217MiB/s][r=16.9k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [r(4)][100.0%][r=4383MiB/s][r=17.5k IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=13112: Sat Mar 1 14:04:26 2025
read: IOPS=17.3k, BW=4320MiB/s (4530MB/s)(506GiB/120016msec)
slat (usec): min=10, max=29937, avg=21.75, stdev=136.61
clat (usec): min=328, max=80366, avg=14787.36, stdev=7702.61
lat (usec): min=349, max=80389, avg=14809.11, stdev=7704.95
clat percentiles (usec):
| 1.00th=[ 1352], 5.00th=[ 2868], 10.00th=[ 4293], 20.00th=[ 6980],
| 30.00th=[ 9503], 40.00th=[12125], 50.00th=[14746], 60.00th=[17433],
| 70.00th=[19792], 80.00th=[22414], 90.00th=[25035], 95.00th=[26608],
| 99.00th=[29492], 99.50th=[31065], 99.90th=[36439], 99.95th=[41157],
| 99.99th=[53216]
bw ( MiB/s): min= 3398, max= 4834, per=100.00%, avg=4321.19, stdev=34.95, samples=959
iops : min=13594, max=19338, avg=17284.74, stdev=139.79, samples=959
lat (usec) : 500=0.01%, 750=0.09%, 1000=0.28%
lat (msec) : 2=2.14%, 4=6.37%, 10=22.86%, 20=38.61%, 50=29.65%
lat (msec) : 100=0.02%
cpu : usr=0.85%, sys=8.85%, ctx=2086106, majf=0, minf=16432
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=2073990,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=4320MiB/s (4530MB/s), 4320MiB/s-4320MiB/s (4530MB/s-4530MB/s), io=506GiB (544GB), run=120016-120016msec
Disk stats (read/write):
sdd: ios=2062370/11, sectors=1061314560/136, merge=10566/0, ticks=30343146/294, in_queue=30343440, util=74.60%
Random Write Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randwrite --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=randwrite, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [w(4)][2.5%][w=2509MiB/s][w=10.0k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [w(4)][100.0%][w=1699MiB/s][w=6797 IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=13401: Sat Mar 1 14:07:18 2025
write: IOPS=8864, BW=2216MiB/s (2324MB/s)(260GiB/120031msec); 0 zone resets
slat (usec): min=12, max=112441, avg=365.37, stdev=1628.54
clat (usec): min=547, max=203162, avg=28510.84, stdev=19850.31
lat (usec): min=565, max=218525, avg=28876.21, stdev=20075.38
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 6], 10.00th=[ 9], 20.00th=[ 13],
| 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 24], 60.00th=[ 29],
| 70.00th=[ 35], 80.00th=[ 43], 90.00th=[ 55], 95.00th=[ 67],
| 99.00th=[ 96], 99.50th=[ 110], 99.90th=[ 140], 99.95th=[ 153],
| 99.99th=[ 180]
bw ( MiB/s): min= 825, max= 4060, per=100.00%, avg=2217.40, stdev=159.97, samples=959
iops : min= 3302, max=16242, avg=8869.62, stdev=639.89, samples=959
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.21%, 4=2.18%, 10=11.63%, 20=26.72%, 50=46.54%
lat (msec) : 100=11.93%, 250=0.79%
cpu : usr=1.92%, sys=5.10%, ctx=1317041, majf=0, minf=51
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,1064006,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=2216MiB/s (2324MB/s), 2216MiB/s-2216MiB/s (2324MB/s-2324MB/s), io=260GiB (279GB), run=120031-120031msec
Disk stats (read/write):
sdd: ios=0/1062353, sectors=0/544362800, merge=0/916, ticks=0/21768470, in_queue=21799479, util=87.66%
Random Read/Write Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randrw --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [m(4)][2.5%][r=1731MiB/s,w=1725MiB/s][r=6922,w=6898 IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [m(4)][100.0%][r=1422MiB/s,w=1407MiB/s][r=5689,w=5629 IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=13499: Sat Mar 1 14:10:25 2025
read: IOPS=5955, BW=1489MiB/s (1561MB/s)(175GiB/120021msec)
slat (usec): min=12, max=709, avg=19.14, stdev= 8.65
clat (usec): min=210, max=234212, avg=20615.00, stdev=14780.94
lat (usec): min=224, max=234236, avg=20634.14, stdev=14780.85
clat percentiles (usec):
| 1.00th=[ 1565], 5.00th=[ 3523], 10.00th=[ 5407], 20.00th=[ 8717],
| 30.00th=[ 11863], 40.00th=[ 14877], 50.00th=[ 17695], 60.00th=[ 20841],
| 70.00th=[ 24511], 80.00th=[ 29754], 90.00th=[ 39060], 95.00th=[ 48497],
| 99.00th=[ 71828], 99.50th=[ 82314], 99.90th=[109577], 99.95th=[121111],
| 99.99th=[160433]
bw ( MiB/s): min= 581, max= 2181, per=100.00%, avg=1489.24, stdev=78.33, samples=960
iops : min= 2324, max= 8724, avg=5956.95, stdev=313.31, samples=960
write: IOPS=5966, BW=1492MiB/s (1564MB/s)(175GiB/120021msec); 0 zone resets
slat (usec): min=12, max=72315, avg=433.98, stdev=2468.25
clat (usec): min=403, max=234269, avg=21872.04, stdev=15447.39
lat (usec): min=420, max=235489, avg=22306.02, stdev=15781.99
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 10],
| 30.00th=[ 13], 40.00th=[ 16], 50.00th=[ 19], 60.00th=[ 22],
| 70.00th=[ 26], 80.00th=[ 32], 90.00th=[ 42], 95.00th=[ 52],
| 99.00th=[ 75], 99.50th=[ 87], 99.90th=[ 115], 99.95th=[ 128],
| 99.99th=[ 167]
bw ( MiB/s): min= 572, max= 2284, per=100.00%, avg=1491.82, stdev=78.59, samples=960
iops : min= 2290, max= 9136, avg=5967.27, stdev=314.35, samples=960
lat (usec) : 250=0.01%, 500=0.02%, 750=0.06%, 1000=0.11%
lat (msec) : 2=1.01%, 4=4.03%, 10=17.44%, 20=33.00%, 50=39.38%
lat (msec) : 100=4.74%, 250=0.20%
cpu : usr=1.43%, sys=6.88%, ctx=2022259, majf=0, minf=57
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=714832,716070,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1489MiB/s (1561MB/s), 1489MiB/s-1489MiB/s (1561MB/s-1561MB/s), io=175GiB (187GB), run=120021-120021msec
WRITE: bw=1492MiB/s (1564MB/s), 1492MiB/s-1492MiB/s (1564MB/s-1564MB/s), io=175GiB (188GB), run=120021-120021msec
Disk stats (read/write):
sdd: ios=713917/715095, sectors=365705216/366293696, merge=370/365, ticks=9758343/10935775, in_queue=20739185, util=80.66%
Sequential Read Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=read --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=read, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=5374MiB/s][r=21.5k IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [R(4)][100.0%][r=6061MiB/s][r=24.2k IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=13541: Sat Mar 1 14:13:29 2025
read: IOPS=23.0k, BW=5750MiB/s (6030MB/s)(674GiB/120011msec)
slat (usec): min=11, max=217836, avg=18.19, stdev=171.56
clat (usec): min=289, max=992429, avg=11110.33, stdev=27638.30
lat (usec): min=313, max=992441, avg=11128.53, stdev=27638.82
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7],
| 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 10], 60.00th=[ 11],
| 70.00th=[ 11], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 13],
| 99.00th=[ 22], 99.50th=[ 228], 99.90th=[ 472], 99.95th=[ 550],
| 99.99th=[ 860]
bw ( MiB/s): min= 2956, max= 8710, per=100.00%, avg=5790.25, stdev=576.99, samples=947
iops : min=11826, max=34840, avg=23160.99, stdev=2307.96, samples=947
lat (usec) : 500=0.01%, 750=0.02%, 1000=0.04%
lat (msec) : 2=0.47%, 4=1.82%, 10=52.24%, 20=44.32%, 50=0.49%
lat (msec) : 100=0.03%, 250=0.22%, 500=0.29%, 750=0.05%, 1000=0.02%
cpu : usr=0.77%, sys=10.34%, ctx=1614344, majf=0, minf=16431
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=2760367,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=5750MiB/s (6030MB/s), 5750MiB/s-5750MiB/s (6030MB/s-6030MB/s), io=674GiB (724GB), run=120011-120011msec
Disk stats (read/write):
sdd: ios=1385449/6, sectors=1411862016/72, merge=1372294/0, ticks=15384147/153, in_queue=15384302, util=77.86%
Sequential Write Throughput
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=write --bs=256k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --size=100G
throughput-test-job: (g=0): rw=write, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=64
...
fio-3.37
Starting 4 processes
Jobs: 4 (f=4): [W(4)][2.5%][w=2362MiB/s][w=9449 IOPS][eta 01m:57s]
...
Jobs: 4 (f=4): [W(4)][100.0%][w=2576MiB/s][w=10.3k IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=13621: Sat Mar 1 14:16:49 2025
write: IOPS=12.5k, BW=3130MiB/s (3282MB/s)(367GiB/120012msec); 0 zone resets
slat (usec): min=10, max=71711, avg=21.88, stdev=167.22
clat (usec): min=272, max=2245.1k, avg=20426.75, stdev=61018.13
lat (usec): min=294, max=2245.1k, avg=20448.63, stdev=61018.32
clat percentiles (usec):
| 1.00th=[ 1532], 5.00th=[ 3326], 10.00th=[ 4817],
| 20.00th=[ 6652], 30.00th=[ 11994], 40.00th=[ 13304],
| 50.00th=[ 14091], 60.00th=[ 15139], 70.00th=[ 16909],
| 80.00th=[ 20055], 90.00th=[ 25560], 95.00th=[ 35390],
| 99.00th=[ 162530], 99.50th=[ 434111], 99.90th=[ 918553],
| 99.95th=[1233126], 99.99th=[1702888]
bw ( MiB/s): min= 105, max= 8888, per=100.00%, avg=3322.56, stdev=430.22, samples=898
iops : min= 422, max=35554, avg=13290.25, stdev=1720.86, samples=898
lat (usec) : 500=0.01%, 750=0.14%, 1000=0.28%
lat (msec) : 2=1.01%, 4=6.25%, 10=19.45%, 20=52.51%, 50=17.72%
lat (msec) : 100=1.20%, 250=0.65%, 500=0.35%, 750=0.25%, 1000=0.08%
lat (msec) : 2000=0.09%, >=2000=0.01%
cpu : usr=2.10%, sys=4.18%, ctx=524654, majf=0, minf=46
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,1502329,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=3130MiB/s (3282MB/s), 3130MiB/s-3130MiB/s (3282MB/s-3282MB/s), io=367GiB (394GB), run=120012-120012msec
Disk stats (read/write):
sdd: ios=0/755141, sectors=0/768494416, merge=0/746034, ticks=0/15427883, in_queue=15453641, util=76.70%
Random Read Latency
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randread --bs=4k --ioengine=libaio --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=latency-test-job --eta-newline=1 --size=100G
latency-test-job: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.37
Starting 1 process
Jobs: 1 (f=1): [r(1)][2.5%][r=73.3MiB/s][r=18.8k IOPS][eta 01m:57s]
...
Jobs: 1 (f=1): [r(1)][100.0%][r=74.1MiB/s][r=19.0k IOPS][eta 00m:00s]
latency-test-job: (groupid=0, jobs=1): err= 0: pid=13690: Sat Mar 1 14:19:48 2025
read: IOPS=19.1k, BW=74.8MiB/s (78.4MB/s)(8975MiB/120001msec)
slat (usec): min=4, max=595, avg= 6.41, stdev= 1.84
clat (nsec): min=821, max=10769k, avg=45330.39, stdev=21870.11
lat (usec): min=35, max=10775, avg=51.74, stdev=22.03
clat percentiles (usec):
| 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 41],
| 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 44], 60.00th=[ 44],
| 70.00th=[ 45], 80.00th=[ 45], 90.00th=[ 47], 95.00th=[ 50],
| 99.00th=[ 159], 99.50th=[ 180], 99.90th=[ 190], 99.95th=[ 223],
| 99.99th=[ 388]
bw ( KiB/s): min=55264, max=84680, per=100.00%, avg=76613.22, stdev=3629.29, samples=239
iops : min=13816, max=21170, avg=19153.31, stdev=907.32, samples=239
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=95.05%
lat (usec) : 100=3.31%, 250=1.61%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 10=0.01%, 20=0.01%
cpu : usr=1.74%, sys=14.29%, ctx=2300473, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2297525,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=74.8MiB/s (78.4MB/s), 74.8MiB/s-74.8MiB/s (78.4MB/s-78.4MB/s), io=8975MiB (9411MB), run=120001-120001msec
Disk stats (read/write):
sdd: ios=2295505/6, sectors=18364040/72, merge=0/0, ticks=107792/50, in_queue=107841, util=88.33%
Random Write Latency
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=latency-test-job --eta-newline=1 --size=100G
latency-test-job: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.37
Starting 1 process
Jobs: 1 (f=1): [w(1)][2.5%][w=61.8MiB/s][w=15.8k IOPS][eta 01m:57s]
...
Jobs: 1 (f=1): [w(1)][100.0%][w=61.0MiB/s][w=15.6k IOPS][eta 00m:00s]
latency-test-job: (groupid=0, jobs=1): err= 0: pid=14125: Sat Mar 1 14:26:07 2025
write: IOPS=16.5k, BW=64.6MiB/s (67.8MB/s)(7755MiB/120001msec); 0 zone resets
slat (usec): min=5, max=932, avg= 7.01, stdev= 2.20
clat (nsec): min=1062, max=9977.5k, avg=52874.86, stdev=36252.85
lat (usec): min=38, max=9985, avg=59.89, stdev=36.43
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 47],
| 30.00th=[ 48], 40.00th=[ 49], 50.00th=[ 50], 60.00th=[ 51],
| 70.00th=[ 52], 80.00th=[ 53], 90.00th=[ 67], 95.00th=[ 75],
| 99.00th=[ 94], 99.50th=[ 112], 99.90th=[ 223], 99.95th=[ 388],
| 99.99th=[ 1991]
bw ( KiB/s): min=54944, max=81144, per=100.00%, avg=66196.75, stdev=4342.12, samples=239
iops : min=13736, max=20286, avg=16549.19, stdev=1085.53, samples=239
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=57.22%
lat (usec) : 100=42.04%, 250=0.65%, 500=0.05%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=1.89%, sys=15.09%, ctx=1991425, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1985191,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=64.6MiB/s (67.8MB/s), 64.6MiB/s-64.6MiB/s (67.8MB/s-67.8MB/s), io=7755MiB (8131MB), run=120001-120001msec
Disk stats (read/write):
sdd: ios=0/1983432, sectors=0/15867456, merge=0/0, ticks=0/100848, in_queue=101140, util=82.55%
Random Read/Write Latency
root@fedora:~# fio --directory=/mnt/ --filename=test --direct=1 --rw=randrw --bs=4k --ioengine=libaio --iodepth=1 --runtime=120 --numjobs=1 --time_based --group_reporting --name=latency-test-job --eta-newline=1 --size=100G
latency-test-job: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.37
Starting 1 process
Jobs: 1 (f=1): [m(1)][2.5%][r=35.1MiB/s,w=34.8MiB/s][r=8987,w=8912 IOPS][eta 01m:57s]
...
Jobs: 1 (f=1): [m(1)][100.0%][r=33.4MiB/s,w=34.2MiB/s][r=8544,w=8751 IOPS][eta 00m:00s]
latency-test-job: (groupid=0, jobs=1): err= 0: pid=14168: Sat Mar 1 14:29:59 2025
read: IOPS=8907, BW=34.8MiB/s (36.5MB/s)(4175MiB/120001msec)
slat (usec): min=4, max=619, avg= 7.03, stdev= 2.18
clat (nsec): min=902, max=1849.1k, avg=44498.37, stdev=7899.77
lat (usec): min=35, max=1883, avg=51.53, stdev= 8.45
clat percentiles (usec):
| 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 42],
| 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 44], 60.00th=[ 45],
| 70.00th=[ 45], 80.00th=[ 47], 90.00th=[ 50], 95.00th=[ 56],
| 99.00th=[ 68], 99.50th=[ 73], 99.90th=[ 95], 99.95th=[ 121],
| 99.99th=[ 269]
bw ( KiB/s): min=29232, max=41840, per=100.00%, avg=35630.47, stdev=2019.14, samples=240
iops : min= 7308, max=10460, avg=8907.62, stdev=504.79, samples=240
write: IOPS=8901, BW=34.8MiB/s (36.5MB/s)(4172MiB/120001msec); 0 zone resets
slat (usec): min=5, max=872, avg= 7.36, stdev= 2.34
clat (nsec): min=942, max=5483.5k, avg=52248.23, stdev=37189.11
lat (usec): min=41, max=5491, avg=59.61, stdev=37.37
clat percentiles (usec):
| 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 47],
| 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 50], 60.00th=[ 51],
| 70.00th=[ 52], 80.00th=[ 53], 90.00th=[ 57], 95.00th=[ 68],
| 99.00th=[ 88], 99.50th=[ 103], 99.90th=[ 277], 99.95th=[ 519],
| 99.99th=[ 2114]
bw ( KiB/s): min=30192, max=42288, per=100.00%, avg=35604.70, stdev=2030.22, samples=240
iops : min= 7548, max=10572, avg=8901.18, stdev=507.56, samples=240
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=72.24%
lat (usec) : 100=27.44%, 250=0.25%, 500=0.03%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=2.11%, sys=15.71%, ctx=2143013, majf=0, minf=15
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1068914,1068141,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=34.8MiB/s (36.5MB/s), 34.8MiB/s-34.8MiB/s (36.5MB/s-36.5MB/s), io=4175MiB (4378MB), run=120001-120001msec
WRITE: bw=34.8MiB/s (36.5MB/s), 34.8MiB/s-34.8MiB/s (36.5MB/s-36.5MB/s), io=4172MiB (4375MB), run=120001-120001msec
Disk stats (read/write):
sdd: ios=1067973/1067147, sectors=8543784/8537176, merge=0/0, ticks=49329/53154, in_queue=102611, util=83.91%
Best Regards,
Antonio
With a system big as yours, which performance it can reach?
Best Regards,
Antonio
Well, yeah. Doing it on the command line.
You need to make it load the nvme, nvmet and nvmet_rdma or nvmet_tcp kernel modules, e.g. via /etc/modules-load.d.
And then install nvmetcli for easy setup and reloading on boot, unless you want to configure it via /sys. And remember to backup /etc/nvmet/config.json every TrueNAS update.
1 Like
Can you please explain what you are talking about here? If this is a 100GbE card then it can do 100GbE (with the correct switch and PCIe slot). The SerDes might be 4x28G lanes, but the ethernet does not see that. SerDes is NOT a LAG.