Best ZFS Raidz config (vdev size per pool) and upgrade possibilities later.
I used Freenas years ago then switched to Windows storage spaces. I have been using that as a home NAS for years on various Dell servers. I more recently picked up a bunch of 4tb SATA SSDs and was going to install them. Then I came across a system I was going to setup as a compute cluster. I am limited to the amount of 2.5inch slots and the w/r will be low for consumer drives. I did some hunting and got what I think is a good deal on a Gigabyte Epyc server with 24 NVMe SSDs. This fixes a number of things for me that I do not have to worry about the consumer reliability and do not have to mess around with NVMe cache cards. I am just trying to decide if to make the 24 1tb drives into one vdev and do Raidz2 or going smaller. I have read posts saying to go three 8 drive vdevs in Z2. I have also heard 8 vdevs in Z1. I have also considered two 12drive vdevs in Z3. I am considering ease of upgrade later because at current pricing I would replace some of those drives with say 3.2 TB drives in batches. However that does me nothing unless I replace every drive in the pool or copy all the data onto a backup server, rebuild, and copy back. Currently, I have two 16tb externals that I backup to often and then unplug and put in the safe. IT would take a while but maybe not as long as individually replacing 24 disks and waiting for each to resilver and then expand. Just my thought.
I am looking for opinions. Never used 24 drives at one time in this use before.
Thanks.
In general, ZFS RAID-Zx, (1/2/3), should not be too wide. Not even for NVMe or SATA SSDs. The number normally used as the suggested maximum is 12 storage devices. But, it’s not a hard and fast rule. Depends on the hardware and application.
Note that is PER ZFS vDev. You can have multiple ZFS vDevs in a pool.
You only need to replace all the storage drives in a vDev, for that specific ZFS vDev to grow in size. So if your pool had; 3 x 8 NVMe in RAID-Z2
Then you only need to replace 8 NVMe at a time to grow that vDev, which grows the pool. Another reason not to make too wide of a RAID-Zx vDev.
As for what is best, that is really up to you and depends on your application. Using 12 wide RAID-Zx vDevs are great for bulk and archival storage but not really suitable for VM storage. Nor are they good for small files.
I think I have come up with a good solution. I have a small 1U NAS that I built years back on a small Atom board. I just need to come up with a sync schedule and put in two large drives in a mirrored config. Then figure out a way how to get it to sync changes nightly at a specific time. Then continue with the offline backup on either a monthly or quarterly basis. Can anyone point me in the direction of a not always on planned backup? In that case I would run two RAIDz1s all day everyday.
