I have my motherboard maxed at 128gb ddr5 ecc ram. I have one main storage pool of twelve 24tb harddrives in Raidz3. This pool serves anything from Plex to Adobe photo and video storage. I have seen the 5x rule and others say 3x to 8x but some say rules have changed and some articles were years ago. What the general consensus in 2025? And should I mirror them or raidz? I have plenty of space in my tower.
How critical is the data and how good is your backup strategy? Also, how performant does the share have to be? 12x24tb is a BIG pool, but the z3 helps mitigate that. 12 is a funky number, I would probably go raidz2x2 pools (that is still ~88TB per pool). I would not go mirrors as it can actually be less redundant and unless you are doing VMs or need really performant drives, I would not bother. I also would not bother with L2Arc unless you know you are over running your memory perry consistently. Truenas does not use a lot of memory on its own and you like have 100GB or so of cache with much faster memory.
1 Like
Some of it is critical as far as photos and video but I have a second offsite truenas server which I’m just setting up and have an external hard drive which contains another copy. I went with the raidz3 12 x 24 tb drives just because of the ease of a single pool but I have spares ready to go if needed. My system uses every gb of ram in there…that’s why I was wondering if I should use an nvme to help with plex or even the most edited or viewed photos in lightroom. All video editing with be done on a different all flash pool. What sizes and configurations is everyone using for L2ARC?
You are asking about L2ARC yet you haven’t posted any plain, ARC statistics? ARC and L2ARC are all about repeated reads on your system.
Start with the basics to understand ZFS and how it uses memory. If you just want to add in L2ARC and play with it, go ahead. You can remove it from your system without damaging your pools.
BASICS
iX Systems pool layout whitepaper
Special VDEV (sVDEV) Planning, Sizing, and Considerations
2 Likes
I was just trying to get a community consensus of what everyone is running. I have read a ton of info but in the future I’ll save all forum input for chastising beginners for not reading every article and actually understanding it.
A lot depends on the use case. For example, my pool is mostly WORM, where few things change except the time machine backups. The pool is about 1/4 full and I can achieve 400MB/s transfer speeds pretty consistently with a single HDD VDEV and a sVDEV.
The sVDEV is not the only reason I have been able to increase write speeds from about 250MB/s to 400MB/s. Another reason was @winnielinnie, who helped me understand the benefits of larger recordsizes for datasets that mostly hold large files (i.e. images, videos, and linux ISOs, for example).
Large recordsizes (i.e. 1M) not only allow faster writes of such files into the pool, they also significantly cut down the metadata needed vs. when default 128kb recordsizes are used for a dataset. That in turn speeds things up. Record sizes should be tuned on a per-dataset basis. Datasets that are meant to store databases and like small files should feature smaller recordsizes.
A sVDEV can increase speeds two further ways, as long as the underlying disks are fast and redundant. Usually, a sVDEV consists of multiple, mirrored SSDs. It stores metadata and small files in separate partitions. Both metadata and small file writes benefit a lot from SSD use, so the pool in aggregate speeds up a lot, especially during writes.
A L2ARC can be helpful in similar ways by storing metadata and/or frequently-accessed files. Unlike sVDEVs, a L2ARC is expendable but it will take some time to fill up with stuff the ARC is kicking out or files that are accessed all the time. So it’s benefit will take a few rsyncs or like operations.
2 Likes
Awesome thank you. Any Size recommendation?
Hi Erik,
L2ARC is one of the most complex areas of ZFS and also one of the most debated. RAM (ARC) is significantly faster than disk be it HDD or SSD so it makes sense that ideally you want your reads coming from RAM where possible. I appreciate you’ve maxed out your RAM options on your mobo but that alone doesn’t automatically mean you would benefit from an L2ARC, in fact it could make things a little worse. I always suggest people considering L2ARC first take a look at arc_summary and feel free to post the output here. This will show you how effective your ARC is atm. Very roughly speaking I’d say a hit ratio of mid-high 90% means L2 is probably not worth it.
4 Likes
That. Assuming that your system has been up for some time, begin by posting the output of arc_summary (using formatted text, </> or two lines of three backquotes).
With 128 GB RAM, you may well find out that you do not need a L2ARC.
2 Likes