I have a few NAS servers running TrueNAS (core + scale) and a few non-ZFS based NAS systems, plus virtue NAS systems (on VirtueBox and PVE), over different hardware systems and a 1Gbps network.
I have not noticed any performance difference between the NAS systems on ZFS and non-ZFS NAS systems - the network is my binding constraint.
I have also changed my hardware configurations, with RAM from 4GB to 128GB. No difference for ZFS or non-ZFS systems.
I have also played with hardware settings for NAS systems on VB/PVE. no difference as well.
I have noticed that some NAS systems (fnOS for example) do seem to use RAM as buffers. but without noticeable performance gains.
So, has anyone done analysis or aware of analysis that shows that big RAM helps materially for ZFS?
First, ZFS is not the fastest file system, logical volume manager and RAID solution. It’s original design was to replace Solaris UFS with something that solved other problems. For example, like long boot time file system checks. Or improved data integrity.
Having larger RAM helps for specific cases, like repeated access to files or directory metadata. Yet, not reading the same files or directories over again simply means the ARC, (aka RAM cache / L1ARC), does not really contain useful entries. In fact, it may not populate easily.
However, if you are already network bound, then that is the area to improve.
Whence you have faster network, you can consider a Metadata only L2ARC device. Or possibly a Special Allocation vDev for Metadata and small blocks. But, the average home user generally does not need those, which complicates ZFS pool management.
