Mysterious read speed cap around 300MB/s on both pools (4xNVME stripe, 24xHDD striped mirrors) in beefy server

TrueNAS General
, ,
15

OK. Just a quick update. But an important one. Installed TrueNAS Core 13.3. Imported the same pool.
And here we go. Full read speeds:

root@truenas[~]# fio --name=benchmark --rw=read --bs=1M --ioengine=posixaio --numjobs=1 --iodepth=32 --size=50G --filename=/mnt/work/work_mirrored/fio.2
benchmark: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=32
fio-3.36
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=1952MiB/s][r=1951 IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=1): err= 0: pid=4056: Mon Sep 30 13:40:47 2024
  read: IOPS=1808, BW=1808MiB/s (1896MB/s)(50.0GiB/28314msec)
    slat (nsec): min=1467, max=48610, avg=5777.63, stdev=1624.25
    clat (usec): min=130, max=559228, avg=17661.54, stdev=20579.07
     lat (usec): min=134, max=559233, avg=17667.31, stdev=20579.08
    clat percentiles (usec):
     |  1.00th=[   297],  5.00th=[   545], 10.00th=[   725], 20.00th=[  2147],
     | 30.00th=[  4948], 40.00th=[  8979], 50.00th=[ 13173], 60.00th=[ 18220],
     | 70.00th=[ 23725], 80.00th=[ 29492], 90.00th=[ 38011], 95.00th=[ 45876],
     | 99.00th=[ 78119], 99.50th=[116917], 99.90th=[235930], 99.95th=[291505],
     | 99.99th=[379585]
   bw (  MiB/s): min=  224, max= 1976, per=100.00%, avg=1810.34, stdev=335.40, samples=55
   iops        : min=  224, max= 1976, avg=1809.95, stdev=335.45, samples=55
  lat (usec)   : 250=0.44%, 500=3.53%, 750=6.61%, 1000=3.15%
  lat (msec)   : 2=5.56%, 4=7.88%, 10=15.32%, 20=21.04%, 50=32.78%
  lat (msec)   : 100=3.06%, 250=0.55%, 500=0.09%, 750=0.01%
  cpu          : usr=1.50%, sys=2.50%, ctx=48918, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=72.1%, 32=27.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=93.7%, 8=4.7%, 16=1.5%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=51200,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1808MiB/s (1896MB/s), 1808MiB/s-1808MiB/s (1896MB/s-1896MB/s), io=50.0GiB (53.7GB), run=28314-28314msec

1800 Megabytes/second - as one would expect from a pool of 24 mirrors. ARC set to metadata only. I also see via iostat that it’s actually reading from disks with each disk at around 80MB/s.

So this is a massive bug in TrueNAS Scale!!!

Late here. Will do some more testing later.

7 Likes