I currently have 8x14TB setup of spinning rust. It is getting full and new HDs has been quite pricey. The current setup is a mirrored raid, so 4 mirrored sets in a stripe. I have been thinking about redoing this to free up some space.
Idea 1:
4 x raidz
4 x raidz
stripe on top
Idea 2:
8 x raidz2
This results in the same space. Which is going to be better? I care about read more than writes. Is there any real difference in these 2 setups?
I have both a log and L2ARC SSDs for use in this setup.
Raidz2 is safer.
The question is what is the pool used for. If itās just SMB/NFS shares, and you used mirrors for flexibility, go ahaed with raidz#. If you have zvols or iSCSI shares, you used mirrors for a good reason.
L2ARC and SLOG are irrelevant to this discussionā¦ but beg the question whether you actually need them.
If you donāt need mirrored/striped datasets (see @etorix), I suggest the Z2 for general WORM use. I run a z3 here and on a 1GbE network, the network is the gating factor. On a 10GbE connection, itās the Z3 but then things are so zippy, I donāt really notice.
When I had a similar set, I went raidz2 for general storage/shares. As others stated, the bottleneck will be the network connection and internal performance will generally be for copying data from A to B within the NAS itself unless you are using a client to perform the operations. Raidz2 is a good balance of performance & data protection.
Iām guessing idea #1 is 2 stripped vdev of raidz1? A bit more risky with higher iops.
Purpose matters as others stated. If this is your only NAS for data retention, raidz2 hands down.
Hereās an old video from Lawrence Systems where they did the benchmarking for the layouts for you. The major difference in performance is in random read/writes for the different layouts.
Clients are 1/2.5/10G. A mix of MacBook Pros (with thunderbolt dock for networking), AppleTVs, etc. In the current system I can push ~2.5G from a desktop to the NAS while moving files from local NVMe storage.
A SLOG will help for the sync writes that Time Machine imposes. It does not have to be a fast SSD, however, I would try to use something that has power loss protection (PLP) such as Optane based SSDs. I use a P4801x but thatās likely overkill. A PCIE 3.0x2 based 32GB Optane module is likely perfectly fine for Time Machine use and very cost-efficient for boot drives also.
Plus, unlike my SATADOMS, those optane modules should NEVER wear out.
The system is direct from TrueNAS. The SLOG and L2ARC are
Micron_5300_MTFDDAK480TDS that are underprovisioned. I use to use NFS so the SLOG helped however I switched to SMB everywhere (MacOS SMB is terrible, but what can you do). The only thing the SLOG helps with now, as you pointed out is TimeMachine backups, which I do use from 3 systems.
I have my media store on NVMe on a local system. I make changes there and rysnc those up to the NAS for the other devices to use. I already find the speed limits of the spinning rust when moving a 50GB movie frustrating and moving to RAIDZ2 seems like that would make it worse. I wonder if dropping the 2 SSDs as SLOG/L2ARC and mirroring them and using them as the special metadata device on a 8 drive RAIDZ2 would making things usable.
I have a feeling I am going to need to backup and just test out a few things.
I had planned on dealing with all of this a year ago but put it off. Given the price increase, that was a costly delay. Oh well.
I purchased this system at the end 2021 and have gotten very good use out of it. I really wish they would release a new Mini XL that had a better CPU.
I submitted a resource page re: sVDEV a while ago. Some advance planning is advisable.
I get decent transfer performance to and from the NAS with it, though I use three mirrored data center SSDs for the task. Keep in mind: Unlike an optional metadata-only L2ARC, the loss of the sVDEV altogether will lead to pool loss.
I only really notice speed differences re: the Z3 when Iām transferring to the pool from an SSD via 10GbE. MacOS is super annoying re: its networking in that you can set network adapter preferences and said preferences are cheerfully ignored.
Itās only when you force a particular connection (ie make WiFi inactive) that you can be sure for a given interface is actually used.
I donāt think there are any CPU / motherboard improvements to be had. I mentioned the āreplacement MBsā that servethehome did articles on in a different thread. More expensive replacements more trade offs for NAS use.
The last 10 years have basically led to the extinction of SATA-port-plentiful motherboards with a focus on NVME and edge computing. Whatever that is.
A few SATA-plentiful boards are still available with the slight bonus that their memory is a little less dear than DDR5. Or go with the Asrock dual-Oculink mini-itx we showcased here that can run NVME and SATA for a total of 9 SATA ports. Still not a 2750d4i but not bad.
Add a low power Ryzen 5 Pro -G or -GE (unlocked!!!) and you have a lot of compute in a very compact and efficient package. The only downside as an end user is that Asrock Rack end user support is uneven, to put it mildly. I luckily had ixsystems in my corner!
If you are using SMB for the NAS connection open /etc/nsmb.conf and add āmc_on=yes & āmc_prefer_wired=yesā. It does not matter if you are using multichannel in this case, it just seems to result in the ethernet being the path used by the SMB service.
