Hey,
writing at my TrueNAS Scale 24.10. via SMB gets slow after several (around 35) Gigabytes. Gigabit Network, starts at expectable 110-115 Megabytes/s Then idles around 10-25 Megabytes/s. I use Windows 11 to write the files and it is a single file.
Hardware: i5-7400T, Asus H110I-Plus, 64 GB DDR4 (upgraded from 16 GB), 2 x Intenso 128GB SSD Sata 3 Boot drive, Intenso SSD SATA 3, 2 TB as pool. Fritzbox 7490, cheap 5 Port Gigabit Switch.
What I tried so far:
-upgrade from TrueNAS Core to Scale
-upgrade to Electric Eel
-delete pool from “mirror” 2 x 2 TB to 2 x 2 “stripe” (now 4 TB available), gave advantage that my “test” File of 50 Gigabyte wrote through with Gigabyte speed, then the next file got slow at the very start.
-Network troubleshoot: TrueNAS and Win 11 via new Cat6 cables direct in FritzBox to avoid the switch (didnt change anything)
My thouhts so far: the RAM doesnt get allocated right.
vm.kmem_size, vm.kmem_size_max increase Variablen in deiner /boot/loader.conf
Maybe should I try this (from german FAQ in old forums)
You should think carefully about using a stripe which has zero redundancy - it is your choice but one of the big benefits of TrueNAS is having ZFS to give you extremely good redundancy protection for your data.
As I think you may already know, writing to your TrueNAS over SMB create asynchronous writes to disk i.e. they are cached in memory and a positive response is immediately sent back to Windows to send the next block - so to start off with you use full network bandwidth.
However ZFS has a maximum amount of memory it can use for all pending writes in order to avoid writes trashing the ARC - which by default is 10% of your total memory. (Open a shell and run cat /sys/module/zfs/parameters/zfs_dirty_data_max_percent to see what your memory percentage is). This is further limited in order to preserve memory for synchronous writes by zfs_vdev_async_write_active_max_dirty_percent which defines how much of zfs_dirty_data_max_percent can be used for async writes. (cat /sys/module/zfs/parameters/zfs_vdev_async_write_active_max_dirty_percent) So, for your system staged async writes are limited to 0% of 6.4GB which is c. 3.6GB.
Once the async write memory is full, then acknowledgements back to Windows to send the next packet are delayed until some writes have been completed to bring the pending writes back below this limit.
If writes are more important to you than preserving your ARC, then after boot you can increase this to (by default) up to 25% by running sudo echo xxxxxxxx > /sys/module/zfs/parameters/zfs_dirty_data_max (not persistent after reboot - you can add it to Init Scripts in System Settings/Advanced to apply it after every reboot).
NOTE 1: I just tried to update zfs_dirty_data_max and zfs_vdev_async_write_active_max_dirty_percent on my own system and (despite these pseudo files apparently being r/w for root) they both failed with a permission error.
NOTE 2: ZFS default tuning settings are generally considered to be a good balance delivering excellent overall performance, and tweaking them can have unintended consequences that actually reduce overall performance. But everyone’s normal workload is different and if yours is something wildly different from a typical NAS server, then this may be beneficial for you overall.
Do NOT try this. Core and Scale are based on very different operating systems.
cat /sys/module/zfs/parameters/zfs_dirty_data_max and multiply the result by 2.5 and round down to an integer.
However as I said, ls -l /sys/module/zfs/parameters/ shows zfs_dirty_data_max to have r/w access by root, however when I tried this on my own system I got a permissions error despite doing it as root (using sudo) - and I have no idea why but it may be TrueNAS’ protection (like preventing use of apt).
EDIT: Whilst it doesn’t work if you do sudo echo ... it works if you do sudo -i and then once you have root you do echo ... (probably because the echo is run as root but the write to the file is non-root).
Assuming that the files you are reading are not in ARC, then read speeds are at the slowest of network and disk speed. Because of ARC and Prefetch, reads are actually more complex to understand than Writes.
The datasheet doesn’t terribly help here, but it does indicate that it’s using “Advanced 3D NAND technology with SLC data turbo cache” (machine translated) - so it’s likely that once you run out of the “data turbo cache”, you’re seeing the actual sustained write speeds of the underlying NAND, and it isn’t specified which type.
So - drives say they have a data transfer rate of “up to 500MB/s” - which is faster than 1Gb/s network speed, so if the disk write was sustainable you should be able to write from memory to disk faster than you can send data over the network. So, most likely it is as @HoneyBadger surmised.
