Performance issues with new SCALE 25.10.0 system [slow], compared to CORE 13.0-u6.2 [fast]

TrueNAS General
,
1

I’m upgrading old hardware and Truenas (Core → Scale). Both systems have 5 spinning disks, RAIDZ2 config, and both with 500 GB SSD for log and 500 GB SSD for cache. Both datasets use “STANDARD” sync.

Both systems have 10Gb SFP+ to aggregation switch, and iperf3 results show link is fully functional.

Remote VM on a Proxmox system has two NFS mounts, one to CORE and one to SCALE, but limited to 2.5 Gb/s link.

FIO test script, shows markedly different results between the two NFS mounts: (script does random and sequential read/write)

CORE : ~ 145 MB/s write and 265 MB/s read, Average IOPS 31,543

SCALE: ~ 15 MB/s write and 39 MB/s read, Average IOPS 3654

Running FIO command for random write directly on SCALE system shows 98 MB/s write and average IOPS 23,875

Any pointers where to start looking and finding the issue for discrepancy?

2

Could you please provide a detailed list of your hardware components and the exact FIO commands you used?

3

OK, I’ve reverted to 25.04 because of this post: 25.10 drive performance has dropped significantly

I’m trying to make sure I have the right “tools” and understanding to evaluate performance and other appropriate settings.

This is the “basic” format of my FIO test. My previous post, I have a script with runs through RanW, SeqW, RanR and SeqR.

sudo sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test_rand_write --filename=test_rand_write --numjobs=32 --bs=4k --iodepth=256 --size=200M --readwrite=randwrite --ramp_time=4 --group_reporting

Testing with a single device pool with a 500 GB SSD (using above), directly on Truenas system

Random write: 1092 MB/s, IOPS: 27,9650

However, from remote VM (with 2.5 Gb/s link), with NFS share and using the same command as above,

Random write: 11.5 MB/s, IOPS: 2,953

I’m not understanding the 95% drop in performance due to “NFS and 2.5 GbE link” interconnect.

So, questions:

  1. Am I running a “resonable” test FIO to evaluate?
  2. Why the 95% reduction in BW between server and VM (i.e. 2.5 GbE should produce 250 MB/s throughput)

My Truenas system:

MB: X11SPH-nCTPF (built in SATA and 2 10G SFP+), BIOS version 4.7, FIRMWARE version 1.74.17. Processor: Intel Xeon 3204 (6 core 1.9 GHz typ)

Memory: 32 GB

1 Like