My write speeds should reach a theoretical maximum of 125MB/s due to having a gigabit-NIC.
I have TrueNAS Scale installed on Proxmox with a HBA card passed through. The HBA connects to the drives via SAS to SATA breakout cables. TrueNAS has 8GB RAM assigned to it and 8 CPU cores.
Over SMB, I get 45-55MB/s write speed. Would increasing the RAM assigned to TrueNAS help here? I’m new to ZFS so don’t really understand aside from the fact that RAM is used as cache.
Also worth noting that for the initial few seconds of the transfer it actually reaches over 125MB/s… I don’t know why it goes over but I guess it’s filling the cache up?
So you are saying, you have slow disks, the absolute minimum memory possible allocated to a memory hungry platform, using the slowest file transfer in protocol in the world and it is slow? Wow, color me shocked. Your configuration is TERRIBLE for performance. At a minimum, you need more memory on TN for any sort of performance, virtualizing also slows you down as you are going through 2 network stacks and SMB is garbage. If you want to test true speed, use something like iperf, which gives you real data.
That all depends on what you are doing with it. If you are doing a lot of reads and writes and want it to be performant, as much as possible is the best recommendation. My smaller system has 64GB of memory and my main has 96GB and it performs quite well across the board. I use mine for work files, VMs, and tv/movie recordings. I run bare metal and have mini pc’s virtualizing most of my workload.
That highly depends on what pool you have and what write you do.
A sequential write could be at 125MB/s.
Random 4k you could get something like 1MB/s.
IMHO the word “cache” should never be used in the ZFS context, since it only confuses people with their assumptions of what a cache is.
For sync writes, there is no “cache”. You can offload ZIL to a SLOG.
For async writes, data is written into TXGs. TXGs are on RAM.
TrueNAS aims at 5 second TXGs.
So for an async write and your 1GBit NIC, you could write 625MB/s into RAM, before the performance will drop to your pool speed.
I am unsure of what you are trying to do or what you are unhappy with.
Just my two cents:
RAIDZ1 is pretty dangerous. I would go with RAIDZ2 or with mirrors. You could make a 3 way mirror.
Don’t use zvols for Data. Unlike datasets which have a max recordsize value, zvols have a static block size value. This comes with huge performance implications. So only put stuff on zvols that really needs zvols (VMs) and not data (movies, music).
