NVMe pool slower than HDD pool | What am I missing here?

The-NinthWave · August 3, 2026, 8:09pm

MB: X10SRi-F
PCIe 3 slot 5 x8

My setup has two pools:

Data pool is comprised of 4x 4TB HDD in RAIDZ2

2x Seagate IronWolf ST4000VN006-3CW104
1x WDC WD40EFPX-68C6CN0
1x WDC WD40EFZX-68AWUN0

machines pool is comprised of 2x 240GB NVMe M.2 on PCIe converter card in x8 slot bifurcated in BIOS in mirror:

SAMSUNG MZVPW256HEGL-000L7
Patriot M.2 P300 256GB

I ran some tests and unless I am mistaken, the HDD pool gives significant write performance than the SSD pool and HDD reads as fast as the SSD pool, maxing theoritical SATA 6GB

How is this possible ?

HDD pool write

truenas_admin@truenas[/mnt/Bassin/testpool]$ sudo fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=write --name=testfile
testfile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
testfile: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=235MiB/s][w=235 IOPS][eta 00m:00s]
testfile: (groupid=0, jobs=1): err= 0: pid=2899517: Wed Jul 29 12:25:45 2026
  write: IOPS=254, BW=255MiB/s (267MB/s)(14.9GiB/60015msec); 0 zone resets
   bw (  KiB/s): min=34816, max=362496, per=100.00%, avg=260829.66, stdev=68093.48, samples=120
   iops        : min=   34, max=  354, avg=254.59, stdev=66.47, samples=120
  cpu          : usr=0.97%, sys=9.30%, ctx=15619, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,15280,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=255MiB/s (267MB/s), 255MiB/s-255MiB/s (267MB/s-267MB/s), io=14.9GiB (16.0GB), run=60015-60015msec

HDD pool read

truenas_admin@truenas[/mnt/Bassin/testpool]$ sudo fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=read --name=testfile
testfile: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][90.9%][r=6560MiB/s][r=6559 IOPS][eta 00m:02s]
testfile: (groupid=0, jobs=1): err= 0: pid=2903196: Wed Jul 29 12:27:17 2026
  read: IOPS=6082, BW=6083MiB/s (6378MB/s)(85.6GiB/14406msec)
   bw (  MiB/s): min= 2924, max= 6675, per=99.83%, avg=6072.30, stdev=1205.59, samples=28
   iops        : min= 2924, max= 6675, avg=6072.21, stdev=1205.65, samples=28
  cpu          : usr=0.77%, sys=99.19%, ctx=85, majf=0, minf=39
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=87626,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=6083MiB/s (6378MB/s), 6083MiB/s-6083MiB/s (6378MB/s-6378MB/s), io=85.6GiB (91.9GB), run=14406-14406msec

SSD pool write

truenas_admin@truenas[/mnt/machines/testpool]$ sudo fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=write --name=testfile
[sudo] password for truenas_admin: 
testfile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
testfile: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=215MiB/s][w=215 IOPS][eta 00m:00s]
testfile: (groupid=0, jobs=1): err= 0: pid=2909168: Wed Jul 29 12:31:46 2026
  write: IOPS=184, BW=185MiB/s (194MB/s)(10.8GiB/60090msec); 0 zone resets
   bw (  KiB/s): min=133120, max=233939, per=100.00%, avg=189280.16, stdev=25145.60, samples=120
   iops        : min=  130, max=  228, avg=184.77, stdev=24.54, samples=120
  cpu          : usr=0.76%, sys=6.74%, ctx=11367, majf=0, minf=38
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,11107,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.8GiB (11.6GB), run=60090-60090msec

SSD pool read

truenas_admin@truenas[/mnt/machines/testpool]$ sudo fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=100G --runtime=60s --readwrite=read --name=testfile
testfile: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][90.5%][r=6543MiB/s][r=6543 IOPS][eta 00m:02s]
testfile: (groupid=0, jobs=1): err= 0: pid=2913140: Wed Jul 29 12:33:28 2026
  read: IOPS=6505, BW=6505MiB/s (6821MB/s)(85.3GiB/13430msec)
   bw (  MiB/s): min= 4889, max= 6632, per=100.00%, avg=6507.02, stdev=333.38, samples=26
   iops        : min= 4889, max= 6632, avg=6506.96, stdev=333.53, samples=26
  cpu          : usr=0.59%, sys=99.38%, ctx=34, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=87367,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=6505MiB/s (6821MB/s), 6505MiB/s-6505MiB/s (6821MB/s-6821MB/s), io=85.3GiB (91.6GB), run=13430-13430msec

MBILC · August 4, 2026, 2:06pm

How is your CPU usage when running the test?

floz · August 4, 2026, 2:29pm

Did you confirm the commands ran on the intended disks? This can be done in the charts under ‘Reporting’.

How is the SSD pool configured?

Why did you select those parameters for fio? Have you compared your fio result with actual performance such as copying a large file in shell?

Your reads are not the speed of SATA. 6-7 GB/s is more than 8 times faster than SATA (6Gb/s). So either youre testing ARC or the wrong drive(s). (see Common pitfall when benchmarking ZFS with fio | pV-tech )

Please don’t cite AI slop screenshots.

The-NinthWave · August 4, 2026, 3:38pm

I don’t

I typed a query in google, it returned this result with the “fs.com” as a source. I went to the website and they sell HW and they have a FAQ that I read. Maybe they have written this using AI but I am not going 2 level down.

The-NinthWave · August 4, 2026, 3:43pm

This was done a few days ago. I’ll rerun the test and check.

Forgot to specify in OP like I did for HDD pool. It’s a mirror.

I just followed some reference for which I did not keep the exact source, only a screenshot. IIRC, it’s from the legacy TN forum.

Glitch01 · August 4, 2026, 3:51pm

I’m with floz, running the same test will read/write directly to arc (ram) unless you disable arc before testing.

The-NinthWave · August 4, 2026, 4:17pm

Si I read this ( I think I can understand the main idea) and ran this

fio --rw=read --bs=1m --direct=1 --ioengine=libaio --size=10G\
  --group_reporting --filename=/mnt//Bassin/machines/bucket --name=job1 --offset=0G\
  --name=job2 --offset=10G --name=job3 --offset=20G --name=job4 --offset=30G\
  --name=job5 --offset=40G --name=job6 --offset=50G --name=job7 --offset=60G\
  --name=job8 --offset=70G

got this

              capacity     operations     bandwidth 
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
Bassin      6.70T  7.85T     17      3  7.71M   236K
boot-pool   6.78G  52.2G      0      4  11.6K  73.3K
machines    43.7G   192G      0     14  85.6K   223K
----------  -----  -----  -----  -----  -----  -----
Bassin      6.70T  7.85T     33      0  2.12M      0
boot-pool   6.78G  52.2G      0     38  12.8K   601K
machines    43.7G   192G      8  1.02K  32.8K   131M
----------  -----  -----  -----  -----  -----  -----
Bassin      6.70T  7.85T     32      0  2.02M      0
boot-pool   6.78G  52.2G      0      0  95.9K      0
machines    43.7G   192G      8    762  32.8K  95.3M
----------  -----  -----  -----  -----  -----  -----
Bassin      6.70T  7.85T     31      0  2.00M      0
boot-pool   6.78G  52.2G      0      0      0      0
machines    43.7G   192G      6    480  85.5K  60.1M
----------  -----  -----  -----  -----  -----  -----
Bassin      6.70T  7.85T     29      0  1.87M      0
boot-pool   6.78G  52.2G      0      0  10.4K      0
machines    43.7G   192G      8    490  67.9K  61.3M
----------  -----  -----  -----  -----  -----  -----
Bassin      6.70T  7.85T     29      0  1.87M      0
boot-pool   6.78G  52.2G      0      0  74.3K      0
machines    43.7G   192G      8    989  68.7K   124M

CPU usage stayed on the floor

somethingweird · August 4, 2026, 4:38pm

Might be silly, but make sure it’s in the correct PCIe slot? (it can mistakenly happen in the wrong slot)

X10SRi-F
1 PCI-E 3.0 x16
1 PCI-E 3.0 x4 (in x8 slot),
2 PCI-E 3.0 x8 ← guess one of these
1 PCI-E 2.0 x4 (in x8 slot)
1 PCI-E 2.0 x2 (in x8 slot)

just covering every possible case -

Farout · August 4, 2026, 5:25pm

I think you are reading from the ARC.

A reboot will clear it.

But dont expect miracles from your NVMEs when it comes to writes. Consumer NVMEs use a pseudo SLC cache. And once that is full, they have to write directly to the TLC NAND - and this can be terribly slow.

Glitch01 · August 4, 2026, 5:47pm

This is a really good point especially when using desktop boards for a NAS. There’s just an extremely limited amount of pci-e lanes for use from the cpu & high end board will have dedicated controller to allocate additional lanes for extra m.2 slots. Can’t just slap an addon card to a free pci-e slot and expect to get pci-e lanes required.

Constantin · August 4, 2026, 8:38pm

Yup, there can be a world of difference in write speed between what goes into the cache up front vs. the permanent flash in the back. No different than real CMR hard drives vs. shingled magnetic recording drives that use a CMR cache.

Once the “fast” cache is full, performance craters.

Topic		Replies	Views
Another Slow nvme Write Speed Thread (Slower than SATA HDDs) TrueNAS General Hardware	4	186	July 1, 2025
SSD pool extremely slow TrueNAS General SCALE , Performance , ZFS	8	248	September 23, 2025
Transfer speeds caps to 20mb/s on NVME TrueNAS General Hardware , Performance	16	1103	August 8, 2025
Inconsistent write speeds on 2.5 Gb/s link TrueNAS General SCALE , Networking , ZFS	20	993	May 15, 2024
Dissapointing iSCSI and NVMEOf Performance :( How to optimise disk / pool performance? TrueNAS General SCALE , ZFS	1	158	January 16, 2026

NVMe pool slower than HDD pool | What am I missing here?

Related topics