I wanted to learn if there was appreciable differences in performance of RUST, SSD, and NVME pools on a True NAS server with a large enough cache to handle all the writes that were being thrown at it. My server is mostly used for backups so write performance is of most interest to me. The servers i a Supermicro X11SRL PCIE3 with 128 GB of RAM, Pool Tank is 3 Seagate Ironwolf 8TB Z1, Pool Speedy is 4 Kingston DC600M SSD’s 4TB each, Z!, Pool Alice is 2 Kingston KC3000 1TB (with ram cache) as a mirror on an Asus HyperCard 4.0. The network is 10 gb ethernet Intel 710 nics all multimode fiber, short distance, one switch. None of the pools have sync turned on. Conclusion: With large files (all over 1 GB) and such a large amount of memory on the sever, there was no more than a 10% difference among the pools when using Cobian backup, the same was true using Crystal Disk Mark at 128 MB. For all three pools, I was able to achieve over 1100MB/sec sequential writes thus saturating the network connection. All testing was done over SMB. The performance with small files was minimally investigated, but it was rather poor. I need to dig further to find out why.
Small files means more IOPS required, hence SSDs will perform better than spinning rust.
Suggested reading are iX's ZFS Pool Layout White Paper and possibily Dig the IOPS - Resources - TrueNAS Community Forums.