Hi guys!
I’m in the process of configuring my first TrueNAS server and although I’m a computer enthusiast, I’m definitely a TrueNAS novice.
I’m running SCALE 25.10.1 on bare metal with the following HW:
Intel 14500T
64GB DDR5 ECC RAM
Boot drive: 250GB Intel SATA SSD
HBA: 9500-8i
SATA drive: WDC WD8005FRYZ 8TB CMR drive (a temporary drive for testing and copying back and forth)
SAS drives: 3x WUH721818AL5204 18TB CMR 4Kn drives
NIC1&2: Intel X710-DA2 10Gb/s
NIC3: Intel i219LM 1Gb/s
NIC4&5: Intel i226LM 2.5Gb/s
The SAS drive has a sustained write throughput about 250MiB/s, but I’m getting 8MiB/s writes through SMB from a Win10 machine with a 2.5Gb/s NIC. The speed doesn’t change much between 1, 2 or 3 disk stripes or RAIDZ1. Record size 128K or 512K, SYNC Always or DISABLED, COMPRESSION LEVEL OFF or LZ4 doesn’t change the result much.
Using SMB to read data from TrueNAS single SAS drive pool to the Win10 machine, I get 250MiB/s.
If I copy data from SATA drive to the SAS drive via shell & mc, I’m getting about 200-250MiB/s.
If I upload data from the Win10 machine to Immich running in a docker container, I get about 100-110MiB/s.
iperf3 shows speeds near max. throughput for both 2.5Gb/s and 10Gb/s (depending on which NIC I currently use).
During copying of the data to the TrueNAS via SMB, the drive sound comes on every zfs_txg_timeout seconds, and iostat confirms it. The following output is for a single SAS drive pool during copying a single large file from the Win10 and then running following shell command during it:
dd if=/dev/zero of=/mnt/hgst/wd2/test6.img bs=1M count=2048 root@truenas[/sys/module/zfs/parameters]# zpool iostat hgst 0.5 capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- hgst 93.2G 16.3T 0 9 209K 1.52M hgst 93.2G 16.3T 0 63 0 32.0M hgst 93.2G 16.3T 0 298 0 27.0M hgst 93.2G 16.3T 0 0 0 0 hgst 93.2G 16.3T 0 0 0 0 hgst 93.2G 16.3T 0 0 0 0 hgst 93.2G 16.3T 0 0 0 0 hgst 93.2G 16.3T 0 0 0 0 hgst 93.2G 16.3T 0 0 0 0 hgst 93.2G 16.3T 0 0 0 0 hgst 93.2G 16.3T 0 0 0 0 hgst 93.2G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 363 0 57.0M hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 426 0 84.9M hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 375 0 59.0M hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 343 0 47.0M hgst 93.3G 16.3T 0 7 0 32.0K hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 0 0 0 hgst 93.3G 16.3T 0 429 0 207M hgst 93.3G 16.3T 0 523 0 262M capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- hgst 93.3G 16.3T 0 543 0 272M hgst 93.3G 16.3T 0 543 0 272M hgst 93.3G 16.3T 0 543 0 272M hgst 93.3G 16.3T 0 543 0 272M hgst 94.1G 16.3T 0 537 0 149M hgst 94.1G 16.3T 0 586 0 293M hgst 94.1G 16.3T 0 533 0 267M hgst 94.1G 16.3T 0 543 0 272M hgst 94.1G 16.3T 0 542 0 271M hgst 94.1G 16.3T 0 545 0 273M hgst 94.1G 16.3T 0 507 0 254M hgst 94.1G 16.3T 0 543 0 272M hgst 94.1G 16.3T 0 553 0 277M hgst 94.1G 16.3T 0 613 0 276M hgst 95.4G 16.3T 0 319 0 70.6M hgst 93.4G 16.3T 0 229 0 18.8M hgst 93.4G 16.3T 0 0 0 0 hgst 93.4G 16.3T 0 0 0 0 hgst 93.4G 16.3T 0 0 0 0 hgst 93.4G 16.3T 0 0 0 0 hgst 93.4G 16.3T 0 0 0 0 hgst 93.4G 16.3T 0 0 0 0 hgst 93.4G 16.3T 0 0 0 0 hgst 93.4G 16.3T 0 0 0 0 hgst 93.4G 16.3T 0 0 0 0 hgst 93.5G 16.3T 0 369 0 82.8M hgst 93.5G 16.3T 0 0 0 0 hgst 93.5G 16.3T 0 0 0 0 hgst 93.5G 16.3T 0 0 0 0 hgst 93.5G 16.3T 0 0 0 0 hgst 93.5G 16.3T 0 0 0 0
SMB write speed to the SATA drive is the same. The Win10 machine is copying data from a NVMe drive and can easily max out 1Gb/s connections of other Win computers or Synology NAS in the same LAN. If there is empty TrueNAS RAM ZFS cache (probably ARC?) after a fresh boot, I get 250MiB/s for the first 4-5GiB and then it drops to 8MiB/s.
Enabling “multichannel” for the SMB service doesn’t change the speeds significantly.
The drive, SAS HBA, NIC, CPU and Win10 machine are more than capable to handle much higher throughput. I really tried googling and even tried Claude AI to get the answer to my questions, but nothing helpful came up, probably I need to tune the ZFS params, but I’m not familiar with them. But it also feels like SMB is writing somehow differently than Immich or dd or copying through mc. Any help would be appreciated.