Direct IO fio testing

Hi there, I deployed TrueNAS SCALE fangtooth BETA in a VM way.

Here is the fio test results:

zfs-direct.always

sudo fio --name=NORMAL_TEST --size=16G --filename=testfile.img --buffer_compress_percentage=0 --bs=512K --nrfiles=1 --rw=write --ioengine=io_uring --direct=1 --iodepth=8 --numjobs=8
NORMAL_TEST: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=io_uring, iodepth=8
...
fio-3.33
Starting 8 processes
NORMAL_TEST: Laying out IO file (1 file / 16384MiB)
Jobs: 7 (f=7): [W(3),_(1),W(4)][97.1%][w=2151MiB/s][w=4301 IOPS][eta 00m:01s]
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7121: Tue Feb  4 18:17:46 2025
  write: IOPS=1021, BW=511MiB/s (536MB/s)(16.0GiB/32074msec); 0 zone resets
    slat (nsec): min=399, max=11536k, avg=6397.57, stdev=100647.15
    clat (usec): min=11, max=87354, avg=7820.45, stdev=7682.34
     lat (usec): min=35, max=87359, avg=7826.84, stdev=7684.47
    clat percentiles (usec):
     |  1.00th=[  343],  5.00th=[  914], 10.00th=[ 1401], 20.00th=[ 2114],
     | 30.00th=[ 2802], 40.00th=[ 3425], 50.00th=[ 4621], 60.00th=[ 7439],
     | 70.00th=[10421], 80.00th=[12518], 90.00th=[17171], 95.00th=[21890],
     | 99.00th=[35390], 99.50th=[42206], 99.90th=[58983], 99.95th=[68682],
     | 99.99th=[83362]
   bw (  KiB/s): min=243712, max=1069643, per=12.85%, avg=525499.60, stdev=281955.25, samples=63
   iops        : min=  476, max= 2089, avg=1026.13, stdev=550.55, samples=63
  lat (usec)   : 20=0.01%, 50=0.09%, 100=0.12%, 250=0.22%, 500=1.39%
  lat (usec)   : 750=1.87%, 1000=2.28%
  lat (msec)   : 2=11.73%, 4=27.80%, 10=21.86%, 20=26.17%, 50=6.23%
  lat (msec)   : 100=0.24%
  cpu          : usr=0.22%, sys=0.29%, ctx=32581, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7122: Tue Feb  4 18:17:46 2025
  write: IOPS=1020, BW=510MiB/s (535MB/s)(16.0GiB/32109msec); 0 zone resets
    slat (nsec): min=400, max=7129.3k, avg=5609.43, stdev=58953.13
    clat (usec): min=9, max=89041, avg=7830.77, stdev=7659.79
     lat (usec): min=35, max=89045, avg=7836.38, stdev=7661.98
    clat percentiles (usec):
     |  1.00th=[  343],  5.00th=[  947], 10.00th=[ 1434], 20.00th=[ 2114],
     | 30.00th=[ 2769], 40.00th=[ 3425], 50.00th=[ 4490], 60.00th=[ 7439],
     | 70.00th=[10552], 80.00th=[12649], 90.00th=[17433], 95.00th=[22152],
     | 99.00th=[34866], 99.50th=[42206], 99.90th=[57934], 99.95th=[63701],
     | 99.99th=[84411]
   bw (  KiB/s): min=231984, max=1087496, per=12.83%, avg=524510.06, stdev=283218.32, samples=63
   iops        : min=  453, max= 2124, avg=1024.21, stdev=553.06, samples=63
  lat (usec)   : 10=0.01%, 50=0.13%, 100=0.09%, 250=0.20%, 500=1.30%
  lat (usec)   : 750=1.71%, 1000=1.95%
  lat (msec)   : 2=12.85%, 4=27.95%, 10=20.67%, 20=26.63%, 50=6.29%
  lat (msec)   : 100=0.25%
  cpu          : usr=0.13%, sys=0.40%, ctx=32878, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7123: Tue Feb  4 18:17:46 2025
  write: IOPS=1020, BW=510MiB/s (535MB/s)(16.0GiB/32105msec); 0 zone resets
    slat (nsec): min=400, max=5907.7k, avg=5724.68, stdev=54605.04
    clat (usec): min=11, max=83334, avg=7831.03, stdev=7568.12
     lat (usec): min=35, max=83338, avg=7836.76, stdev=7570.83
    clat percentiles (usec):
     |  1.00th=[  388],  5.00th=[  979], 10.00th=[ 1450], 20.00th=[ 2147],
     | 30.00th=[ 2737], 40.00th=[ 3458], 50.00th=[ 4621], 60.00th=[ 7570],
     | 70.00th=[10552], 80.00th=[12649], 90.00th=[17171], 95.00th=[22152],
     | 99.00th=[34866], 99.50th=[40633], 99.90th=[56361], 99.95th=[63701],
     | 99.99th=[82314]
   bw (  KiB/s): min=235049, max=1102067, per=12.83%, avg=524632.08, stdev=283512.70, samples=63
   iops        : min=  459, max= 2152, avg=1024.43, stdev=553.61, samples=63
  lat (usec)   : 20=0.01%, 50=0.05%, 100=0.12%, 250=0.31%, 500=1.03%
  lat (usec)   : 750=1.35%, 1000=2.30%
  lat (msec)   : 2=12.60%, 4=28.08%, 10=20.97%, 20=26.70%, 50=6.26%
  lat (msec)   : 100=0.22%
  cpu          : usr=0.12%, sys=0.40%, ctx=33006, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7124: Tue Feb  4 18:17:46 2025
  write: IOPS=1027, BW=514MiB/s (539MB/s)(16.0GiB/31884msec); 0 zone resets
    slat (nsec): min=400, max=31855k, avg=8315.09, stdev=244994.12
    clat (usec): min=9, max=90004, avg=7771.18, stdev=7604.98
     lat (usec): min=33, max=90008, avg=7779.50, stdev=7617.52
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  963], 10.00th=[ 1434], 20.00th=[ 2114],
     | 30.00th=[ 2769], 40.00th=[ 3490], 50.00th=[ 4555], 60.00th=[ 7635],
     | 70.00th=[10421], 80.00th=[12387], 90.00th=[16909], 95.00th=[21890],
     | 99.00th=[34866], 99.50th=[41681], 99.90th=[57934], 99.95th=[71828],
     | 99.99th=[86508]
   bw (  KiB/s): min=245269, max=1083000, per=12.89%, avg=527185.65, stdev=281227.96, samples=63
   iops        : min=  479, max= 2115, avg=1029.46, stdev=549.18, samples=63
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.07%, 100=0.12%, 250=0.25%
  lat (usec)   : 500=1.11%, 750=1.73%, 1000=2.07%
  lat (msec)   : 2=12.42%, 4=27.83%, 10=21.79%, 20=26.37%, 50=6.00%
  lat (msec)   : 100=0.24%
  cpu          : usr=0.16%, sys=0.38%, ctx=32752, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7125: Tue Feb  4 18:17:46 2025
  write: IOPS=1021, BW=511MiB/s (536MB/s)(16.0GiB/32076msec); 0 zone resets
    slat (nsec): min=399, max=14856k, avg=5792.81, stdev=97343.30
    clat (usec): min=8, max=83394, avg=7823.46, stdev=7644.02
     lat (usec): min=34, max=83398, avg=7829.25, stdev=7646.84
    clat percentiles (usec):
     |  1.00th=[  347],  5.00th=[  914], 10.00th=[ 1418], 20.00th=[ 2147],
     | 30.00th=[ 2769], 40.00th=[ 3425], 50.00th=[ 4555], 60.00th=[ 7701],
     | 70.00th=[10552], 80.00th=[12649], 90.00th=[17171], 95.00th=[22152],
     | 99.00th=[35390], 99.50th=[41157], 99.90th=[58459], 99.95th=[66323],
     | 99.99th=[81265]
   bw (  KiB/s): min=242203, max=1092670, per=12.85%, avg=525515.98, stdev=282687.91, samples=63
   iops        : min=  473, max= 2134, avg=1026.19, stdev=552.03, samples=63
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.08%, 100=0.13%, 250=0.21%
  lat (usec)   : 500=1.30%, 750=1.94%, 1000=2.15%
  lat (msec)   : 2=12.26%, 4=28.20%, 10=20.75%, 20=26.58%, 50=6.12%
  lat (msec)   : 100=0.27%
  cpu          : usr=0.21%, sys=0.32%, ctx=32765, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7126: Tue Feb  4 18:17:46 2025
  write: IOPS=1021, BW=511MiB/s (536MB/s)(16.0GiB/32075msec); 0 zone resets
    slat (nsec): min=400, max=15240k, avg=6408.20, stdev=137175.56
    clat (usec): min=6, max=92610, avg=7823.15, stdev=7674.54
     lat (usec): min=37, max=92616, avg=7829.56, stdev=7678.47
    clat percentiles (usec):
     |  1.00th=[  363],  5.00th=[  914], 10.00th=[ 1401], 20.00th=[ 2147],
     | 30.00th=[ 2802], 40.00th=[ 3458], 50.00th=[ 4555], 60.00th=[ 7635],
     | 70.00th=[10552], 80.00th=[12518], 90.00th=[17171], 95.00th=[22152],
     | 99.00th=[35390], 99.50th=[41157], 99.90th=[60031], 99.95th=[70779],
     | 99.99th=[89654]
   bw (  KiB/s): min=244247, max=1098011, per=12.85%, avg=525551.60, stdev=282750.83, samples=63
   iops        : min=  477, max= 2144, avg=1026.22, stdev=552.08, samples=63
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.06%, 100=0.15%, 250=0.20%
  lat (usec)   : 500=1.21%, 750=1.89%, 1000=2.38%
  lat (msec)   : 2=11.86%, 4=28.11%, 10=21.11%, 20=26.66%, 50=6.12%
  lat (msec)   : 100=0.22%
  cpu          : usr=0.21%, sys=0.31%, ctx=32798, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7127: Tue Feb  4 18:17:46 2025
  write: IOPS=998, BW=499MiB/s (523MB/s)(16.0GiB/32823msec); 0 zone resets
    slat (nsec): min=399, max=13286k, avg=6686.76, stdev=104513.84
    clat (usec): min=8, max=90244, avg=8002.86, stdev=7362.48
     lat (usec): min=34, max=90249, avg=8009.55, stdev=7365.39
    clat percentiles (usec):
     |  1.00th=[  461],  5.00th=[ 1106], 10.00th=[ 1680], 20.00th=[ 2474],
     | 30.00th=[ 3163], 40.00th=[ 3949], 50.00th=[ 5014], 60.00th=[ 8225],
     | 70.00th=[10552], 80.00th=[12518], 90.00th=[17171], 95.00th=[21890],
     | 99.00th=[34341], 99.50th=[39584], 99.90th=[57934], 99.95th=[66847],
     | 99.99th=[87557]
   bw (  KiB/s): min=244247, max=1080778, per=12.44%, avg=508883.75, stdev=268330.41, samples=65
   iops        : min=  477, max= 2110, avg=993.65, stdev=523.91, samples=65
  lat (usec)   : 10=0.01%, 50=0.05%, 100=0.11%, 250=0.16%, 500=0.80%
  lat (usec)   : 750=1.44%, 1000=1.67%
  lat (msec)   : 2=9.18%, 4=27.15%, 10=26.17%, 20=27.10%, 50=5.98%
  lat (msec)   : 100=0.18%
  cpu          : usr=0.16%, sys=0.39%, ctx=34981, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7128: Tue Feb  4 18:17:46 2025
  write: IOPS=1021, BW=511MiB/s (536MB/s)(16.0GiB/32067msec); 0 zone resets
    slat (nsec): min=400, max=34242k, avg=7220.49, stdev=214434.09
    clat (usec): min=8, max=85509, avg=7818.89, stdev=7633.99
     lat (usec): min=35, max=85513, avg=7826.11, stdev=7639.06
    clat percentiles (usec):
     |  1.00th=[  367],  5.00th=[  873], 10.00th=[ 1369], 20.00th=[ 2114],
     | 30.00th=[ 2769], 40.00th=[ 3490], 50.00th=[ 4555], 60.00th=[ 7570],
     | 70.00th=[10421], 80.00th=[12518], 90.00th=[17171], 95.00th=[22152],
     | 99.00th=[35390], 99.50th=[42730], 99.90th=[55837], 99.95th=[66323],
     | 99.99th=[82314]
   bw (  KiB/s): min=242203, max=1080109, per=12.85%, avg=525621.37, stdev=282370.32, samples=63
   iops        : min=  473, max= 2109, avg=1026.35, stdev=551.35, samples=63
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.08%, 100=0.13%, 250=0.21%
  lat (usec)   : 500=1.35%, 750=2.06%, 1000=2.23%
  lat (msec)   : 2=12.40%, 4=26.72%, 10=22.07%, 20=26.14%, 50=6.42%
  lat (msec)   : 100=0.18%
  cpu          : usr=0.15%, sys=0.36%, ctx=32740, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=3993MiB/s (4187MB/s), 499MiB/s-514MiB/s (523MB/s-539MB/s), io=128GiB (137GB), run=31884-32823msec

zfs-direct.disabled

sudo fio --name=NORMAL_TEST --size=16G --filename=testfile.img --buffer_compress_percentage=0 --bs=512K --nrfiles=1 --rw=write --ioengine=io_uring --direct=1 --iodepth=8 --numjobs=8
NORMAL_TEST: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=io_uring, iodepth=8
...
fio-3.33
Starting 8 processes
Jobs: 8 (f=8): [W(8)][100.0%][w=6290MiB/s][w=12.6k IOPS][eta 00m:00s]
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7256: Tue Feb  4 18:19:08 2025
  write: IOPS=1631, BW=816MiB/s (855MB/s)(16.0GiB/20090msec); 0 zone resets
    slat (nsec): min=400, max=31961k, avg=6709.49, stdev=225183.81
    clat (usec): min=13, max=61779, avg=4884.77, stdev=4777.01
     lat (usec): min=36, max=61783, avg=4891.48, stdev=4783.77
    clat percentiles (usec):
     |  1.00th=[  375],  5.00th=[  971], 10.00th=[ 1516], 20.00th=[ 2245],
     | 30.00th=[ 2835], 40.00th=[ 3326], 50.00th=[ 3851], 60.00th=[ 4490],
     | 70.00th=[ 5145], 80.00th=[ 6128], 90.00th=[ 8094], 95.00th=[11469],
     | 99.00th=[28705], 99.50th=[36439], 99.90th=[45351], 99.95th=[48497],
     | 99.99th=[58983]
   bw (  KiB/s): min=650986, max=922007, per=12.52%, avg=835846.80, stdev=56397.40, samples=40
   iops        : min= 1271, max= 1800, avg=1632.22, stdev=110.17, samples=40
  lat (usec)   : 20=0.01%, 50=0.06%, 100=0.13%, 250=0.27%, 500=1.06%
  lat (usec)   : 750=1.65%, 1000=2.09%
  lat (msec)   : 2=10.94%, 4=36.35%, 10=41.08%, 20=4.34%, 50=2.00%
  lat (msec)   : 100=0.04%
  cpu          : usr=0.20%, sys=0.46%, ctx=24493, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7257: Tue Feb  4 18:19:08 2025
  write: IOPS=1630, BW=815MiB/s (855MB/s)(16.0GiB/20095msec); 0 zone resets
    slat (nsec): min=400, max=11784k, avg=5007.72, stdev=86012.56
    clat (usec): min=11, max=63113, avg=4885.46, stdev=4851.91
     lat (usec): min=37, max=63117, avg=4890.46, stdev=4853.61
    clat percentiles (usec):
     |  1.00th=[  355],  5.00th=[ 1057], 10.00th=[ 1532], 20.00th=[ 2245],
     | 30.00th=[ 2868], 40.00th=[ 3326], 50.00th=[ 3851], 60.00th=[ 4424],
     | 70.00th=[ 5080], 80.00th=[ 6063], 90.00th=[ 8160], 95.00th=[11338],
     | 99.00th=[29230], 99.50th=[36963], 99.90th=[45876], 99.95th=[49021],
     | 99.99th=[61604]
   bw (  KiB/s): min=636678, max=922007, per=12.51%, avg=835572.03, stdev=59782.81, samples=40
   iops        : min= 1243, max= 1800, avg=1631.70, stdev=116.80, samples=40
  lat (usec)   : 20=0.01%, 50=0.08%, 100=0.15%, 250=0.32%, 500=1.15%
  lat (usec)   : 750=1.30%, 1000=1.51%
  lat (msec)   : 2=11.53%, 4=36.86%, 10=40.73%, 20=4.25%, 50=2.06%
  lat (msec)   : 100=0.04%
  cpu          : usr=0.23%, sys=0.44%, ctx=24625, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7258: Tue Feb  4 18:19:08 2025
  write: IOPS=1630, BW=815MiB/s (855MB/s)(16.0GiB/20093msec); 0 zone resets
    slat (nsec): min=400, max=7218.8k, avg=5515.66, stdev=69487.58
    clat (usec): min=9, max=61564, avg=4886.35, stdev=4687.26
     lat (usec): min=35, max=61568, avg=4891.87, stdev=4689.10
    clat percentiles (usec):
     |  1.00th=[  396],  5.00th=[ 1020], 10.00th=[ 1532], 20.00th=[ 2245],
     | 30.00th=[ 2835], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4490],
     | 70.00th=[ 5211], 80.00th=[ 6194], 90.00th=[ 8160], 95.00th=[11600],
     | 99.00th=[27919], 99.50th=[35390], 99.90th=[43254], 99.95th=[46400],
     | 99.99th=[61080]
   bw (  KiB/s): min=647168, max=928127, per=12.51%, avg=835464.33, stdev=59098.62, samples=40
   iops        : min= 1264, max= 1812, avg=1631.55, stdev=115.38, samples=40
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.07%, 100=0.10%, 250=0.22%
  lat (usec)   : 500=1.10%, 750=1.35%, 1000=1.94%
  lat (msec)   : 2=11.18%, 4=36.26%, 10=41.05%, 20=4.82%, 50=1.87%
  lat (msec)   : 100=0.02%
  cpu          : usr=0.26%, sys=0.43%, ctx=24612, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7259: Tue Feb  4 18:19:08 2025
  write: IOPS=1630, BW=815MiB/s (855MB/s)(16.0GiB/20093msec); 0 zone resets
    slat (nsec): min=400, max=5621.6k, avg=5561.22, stdev=71340.49
    clat (usec): min=11, max=54792, avg=4885.76, stdev=4711.22
     lat (usec): min=38, max=54796, avg=4891.32, stdev=4712.83
    clat percentiles (usec):
     |  1.00th=[  379],  5.00th=[  947], 10.00th=[ 1450], 20.00th=[ 2278],
     | 30.00th=[ 2835], 40.00th=[ 3359], 50.00th=[ 3884], 60.00th=[ 4490],
     | 70.00th=[ 5145], 80.00th=[ 6128], 90.00th=[ 8225], 95.00th=[11600],
     | 99.00th=[28967], 99.50th=[35390], 99.90th=[43254], 99.95th=[46400],
     | 99.99th=[53216]
   bw (  KiB/s): min=642048, max=923648, per=12.52%, avg=835676.93, stdev=59142.43, samples=40
   iops        : min= 1254, max= 1804, avg=1631.93, stdev=115.49, samples=40
  lat (usec)   : 20=0.01%, 50=0.08%, 100=0.13%, 250=0.22%, 500=1.31%
  lat (usec)   : 750=1.53%, 1000=2.21%
  lat (msec)   : 2=10.64%, 4=35.90%, 10=41.58%, 20=4.45%, 50=1.90%
  lat (msec)   : 100=0.02%
  cpu          : usr=0.20%, sys=0.45%, ctx=24509, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7260: Tue Feb  4 18:19:08 2025
  write: IOPS=1630, BW=815MiB/s (855MB/s)(16.0GiB/20094msec); 0 zone resets
    slat (nsec): min=399, max=30248k, avg=5782.18, stdev=174050.74
    clat (usec): min=8, max=54223, avg=4881.58, stdev=4765.57
     lat (usec): min=38, max=54226, avg=4887.36, stdev=4769.45
    clat percentiles (usec):
     |  1.00th=[  408],  5.00th=[ 1004], 10.00th=[ 1516], 20.00th=[ 2278],
     | 30.00th=[ 2835], 40.00th=[ 3359], 50.00th=[ 3884], 60.00th=[ 4424],
     | 70.00th=[ 5145], 80.00th=[ 6128], 90.00th=[ 8029], 95.00th=[11469],
     | 99.00th=[29492], 99.50th=[36439], 99.90th=[45351], 99.95th=[47973],
     | 99.99th=[52167]
   bw (  KiB/s): min=640766, max=913408, per=12.52%, avg=835730.78, stdev=57061.58, samples=40
   iops        : min= 1251, max= 1784, avg=1632.03, stdev=111.48, samples=40
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.07%, 100=0.11%, 250=0.27%
  lat (usec)   : 500=0.92%, 750=1.56%, 1000=2.01%
  lat (msec)   : 2=11.33%, 4=36.02%, 10=41.25%, 20=4.35%, 50=2.06%
  lat (msec)   : 100=0.02%
  cpu          : usr=0.22%, sys=0.48%, ctx=24449, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7261: Tue Feb  4 18:19:08 2025
  write: IOPS=1631, BW=816MiB/s (855MB/s)(16.0GiB/20090msec); 0 zone resets
    slat (nsec): min=400, max=28277k, avg=7078.07, stdev=214157.17
    clat (usec): min=10, max=56487, avg=4882.03, stdev=4717.89
     lat (usec): min=37, max=56491, avg=4889.11, stdev=4722.87
    clat percentiles (usec):
     |  1.00th=[  396],  5.00th=[  971], 10.00th=[ 1500], 20.00th=[ 2245],
     | 30.00th=[ 2802], 40.00th=[ 3326], 50.00th=[ 3884], 60.00th=[ 4424],
     | 70.00th=[ 5145], 80.00th=[ 6194], 90.00th=[ 8225], 95.00th=[11338],
     | 99.00th=[27919], 99.50th=[35390], 99.90th=[44827], 99.95th=[47973],
     | 99.99th=[55313]
   bw (  KiB/s): min=717824, max=927107, per=12.52%, avg=835619.95, stdev=53722.45, samples=40
   iops        : min= 1402, max= 1810, avg=1631.83, stdev=104.92, samples=40
  lat (usec)   : 20=0.01%, 50=0.07%, 100=0.16%, 250=0.21%, 500=1.10%
  lat (usec)   : 750=1.56%, 1000=2.08%
  lat (msec)   : 2=11.11%, 4=36.09%, 10=41.12%, 20=4.52%, 50=1.95%
  lat (msec)   : 100=0.02%
  cpu          : usr=0.20%, sys=0.46%, ctx=24508, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7262: Tue Feb  4 18:19:08 2025
  write: IOPS=1630, BW=815MiB/s (855MB/s)(16.0GiB/20102msec); 0 zone resets
    slat (nsec): min=400, max=25693k, avg=7126.61, stdev=205596.94
    clat (usec): min=11, max=57366, avg=4886.05, stdev=4782.79
     lat (usec): min=34, max=57370, avg=4893.18, stdev=4789.87
    clat percentiles (usec):
     |  1.00th=[  326],  5.00th=[  922], 10.00th=[ 1434], 20.00th=[ 2245],
     | 30.00th=[ 2835], 40.00th=[ 3359], 50.00th=[ 3916], 60.00th=[ 4490],
     | 70.00th=[ 5211], 80.00th=[ 6194], 90.00th=[ 8160], 95.00th=[11338],
     | 99.00th=[29230], 99.50th=[36439], 99.90th=[45876], 99.95th=[51119],
     | 99.99th=[55313]
   bw (  KiB/s): min=648192, max=918948, per=12.51%, avg=835160.27, stdev=57284.23, samples=40
   iops        : min= 1266, max= 1794, avg=1630.92, stdev=111.92, samples=40
  lat (usec)   : 20=0.01%, 50=0.11%, 100=0.19%, 250=0.40%, 500=1.20%
  lat (usec)   : 750=1.63%, 1000=2.11%
  lat (msec)   : 2=11.00%, 4=35.29%, 10=41.53%, 20=4.57%, 50=1.89%
  lat (msec)   : 100=0.06%
  cpu          : usr=0.22%, sys=0.45%, ctx=24482, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8
NORMAL_TEST: (groupid=0, jobs=1): err= 0: pid=7263: Tue Feb  4 18:19:08 2025
  write: IOPS=1630, BW=815MiB/s (855MB/s)(16.0GiB/20098msec); 0 zone resets
    slat (nsec): min=400, max=12624k, avg=5617.01, stdev=95704.61
    clat (usec): min=8, max=62076, avg=4887.68, stdev=4762.93
     lat (usec): min=37, max=62079, avg=4893.30, stdev=4765.32
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[ 1004], 10.00th=[ 1516], 20.00th=[ 2278],
     | 30.00th=[ 2835], 40.00th=[ 3359], 50.00th=[ 3884], 60.00th=[ 4490],
     | 70.00th=[ 5145], 80.00th=[ 6128], 90.00th=[ 8094], 95.00th=[11469],
     | 99.00th=[28967], 99.50th=[35914], 99.90th=[45351], 99.95th=[51119],
     | 99.99th=[60031]
   bw (  KiB/s): min=642048, max=919552, per=12.51%, avg=835555.00, stdev=57611.90, samples=40
   iops        : min= 1254, max= 1796, avg=1631.70, stdev=112.55, samples=40
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.06%, 100=0.16%, 250=0.20%
  lat (usec)   : 500=1.11%, 750=1.41%, 1000=1.97%
  lat (msec)   : 2=11.53%, 4=35.65%, 10=41.41%, 20=4.56%, 50=1.86%
  lat (msec)   : 100=0.06%
  cpu          : usr=0.25%, sys=0.44%, ctx=24502, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,32768,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=6520MiB/s (6837MB/s), 815MiB/s-816MiB/s (855MB/s-855MB/s), io=128GiB (137GB), run=20090-20102msec

My testing is based on a Hyper-V virtual machine with only 4GB RAM and 64G virtual disk. TrueNAS VM recognized that 64GB virtual disk as an HDD.

Why does direct_IO perform poorly compared with buffered io?

I also test the 4K random writing. Those two write models are close in iops.

You set direct=always for the dataset? You are expressly telling ZFS to bypass all standard caching mechanisms and read directly from disk.

You are also virtualizing this disk which for a whole host of reasons is bad

Unless you have an NVME pool with multiple VDEVs this not not a knob you should be turning.

I did these tests under a VM because I have no other available machine. My productive machine is a physical host.

Sure, but you’re testing a performance oriented setting not designed or intended for the system you are running it on. What are you trying to accomplish?