This is my first truenas build and need som adwise on what the best vdev config will be.
Setup:
Supermicro X12SCZ-TLN4F
Nic 10gbe onboard
11th Gen Intel(R) Core™ i5-11600 @ 2.80GHz
128gb ram
Controller
1x LSI 9400-16I
1x IBM M1210
Pool boot -Mirror
vdev - 2TB SATA SSD
Pool data raidz2
vdev1 - 9x 4tb SSD SAS/SATA
vdev2 - 9x 4tb SSD SAS/SATA
SLOG - Mirror
1x Crucial T705 2TB PCIe Gen5 NVMe M.2 SSD
1x Samsung 9100 PRO SSD - 2TB
Spare
1-2x 4tb sata/ssd
This is my setup for now and that gives me 14x4TB=56TB (two vdev inraidz2)
I understand that two vdev is better performance and faster rebuild that with the SSD is hours not days, but as my limits is the CPU and the 10gbe nic will it not be better to make one 18x 4tb raidz2 ? i mean that if rebuild is fast and i have spares, then i can get 8TB extra spare because of one vdev vs two vdev
this Is a real waste. A bit less for the mirror layout (can be usefull in specific use cases, but Is useless in many), but a lot more for the size of the disks: boot pool will use <10 gb, and the rest of the space will remain unused.
If your use case not cover Sync write, they can be useless
Is a lot dangerous, ix reccomend 10-12 disks max per vdev, in your place i would use those
Since you’re going for a Supermicro board, it’s a pity not to pair it with a Xeon W-1200/1300 and ECC UDIMM.
Why two HBAs? You can use SATA ports or use an expander to save PCIe lanes.
The boot drives are vastly oversized (128 GB would be fine), and the mirror is quite possibly overkill.
As for SLOG… What’s the use case for a SLOG to begin with? Especially with a SSD pool, which has the potential to beat its SLOG. Then, assuming there’s a use case, the drives are vastly oversized (SLOG is NOT a write cache, you’re never going to use more than two transaction groups worth of data, i.e. 10 seconds at 10 Gb/s), the mirror is overkill (unless this is business-critical system aiming for “five-nines”, and then you should be talking to a salesperson from iX) and I doubt that your drives are suitable for SLOG to begin with (you should use Data Centre SSD with Power-Loss Protection).
It can be. After about 12 storage devices in RAID-Zx, things get a bit strange. The wider the worse it can get. While ZFS can under allocate the full 18 wide stripe for a file block that does not need the full 18 wide, this can, over time, lead to fragmentation of the free space.
One result of this fragmentation is slower writes, and eventually slower reads. The reads get slower because ZFS may have written data in fragments all over instead of a long stream of blocks. This slowness can even impact things like backups or worse, data migration OFF this too wide RAID-Zx vDev.
Some people using wide RAID-Zx vDevs claim to have no problems. And, to be fair, they probably don’t, (or didn’t at the time of writing), have problems.
The main issue with too wide ZFS RAID-Zx vDevs, is that you are running is a less common configuration. When, (or if), problems develop, it may be too late for a straight forward fix. It is of course uncertain if you will have any problems. But the millions of hours of use for RAID-Zx vDevs that are 12 disks or less, has shown this to be a very reliable configuration.
For example, one problem is that it might take weeks to copy all the data off, even though original full backs used to take days. Then you are stuck. Continue using the pool. Or full backup, reconfigure the pool and restore.
it is an pro home lab setup where i am replacing some old Synology nas and i do work with forensics so are often moving/processing multi tb data over my homelab there is full 10gbe
Primary need is…
Low heat, noise and wattages
NFS storages for my 2 ESX server
NAS storages for data
fast storages for forensics data and work
And then it is an fun project where i have most of the parts and some 4tb ssd, so some parts and selection is based on what i had and then trying to combinde that into an very cool truenas server
okey that make good sense, i was just unsure how much writing that was to the OS disk and to make sure there was an long life time then i just selected some big disk
it was the plan to set sync on the dataset for VMware VMs
The case is an 45drive HL15 where the plan was to have the disk primary in the 15 disk slots in front and then i plan to 3d print an 4 drive holder to be placed in the case near the mailboard and then connect them to the 4 port hba.
so the idea right now is likely 2x vdev raidz2 of 9x 4tb drives, that will give 18 disk and then i have an slot exstra for spare disk or i like the idea to have one free slot if i need to replace one disk then can add that and then do an safe replace
This to me says a minimum of “use a Xeon and ECC” or perhaps even more to the point “contact us at TrueNAS about a commercially supported solution for use in a business environment.”
SLOG will be useful in this case then, but both of the SSDs listed are “pro-sumer” grade. Would suggest a couple of Optane P4801X cards if you are limited to M.2 slots.
A 2x9wZ2 is much more reasonable than 1x18wZ2 for the reasons of stripe width as mentioned above by @Arwen and you will also be able to benefit from better behavior with the SPA routines having two vdevs to work with vs just the one.
I understand what your saying, that is possible but as you say along many others that it is an bad idea to go this less safe route and even that i lose some disk space it is better to have 2x vdev raidz2 of 9wide 4tb ssd as that safe and tested design.
The setup is not standalone, it will be the primary storages and then i use the old Synology NAS that it is replaceing, they will be replica copys, the design will like be 3 tiar and offline backup
From the Hardware Guide section of the current Fangtooth 25.04
section Hybrid Storage & Flash Cache (SLOG/ZIL/L2ARC)
SLOG devices do not need a large capacity since they only need to service five seconds of data writes delivered by the network or a local application. A high-endurance, low-latency device between 8 GB and 32 GB is adequate for most modern networks, and you can strip or mirror several devices for either performance or redundancy. Pay attention to the published endurance claims for the device since a SLOG acts as the funnel point for most of the writes made to the system.
SLOG devices also need power protection. The purpose of the ZFS intent log (ZIL), and thus the SLOG, is to keep sync writes safe during a crash or power failure. If the SLOG is not power-protected and loses data after a power failure, it defeats the purpose of using a SLOG in the first place. Check the manufacturer specifications for the device to ensure the SLOG device is power-safe or has power loss/failure protection.
An UPS is not a proper substitute for PLP.
You’d be better putting VMs on these NVMe drives, mirorred, and leaving the raidz2 to bulk storage (SMB/NFS shares).
okey yes and i think that i will test if it is okey or have have to change that, but i hope that i can combine the VM stuff there is mainly for test lab vm.
I have an another ESX with internal SSD that i use the important stuff, so the VM on the NAS… it is okey if it is not the fastest one.
The design of the NAS is manily to get alot of space on SSD and if i have to make an mirror then it will cost some drive bays and SSD.
what is the best block size for the dataset with VM?
One last comment, (because you mention “45drive HL15”).
It is possible and at times practical to have asymmetrical vDevs. For example, if you went with a 12 disk RAID-Z2 using the conventional maximum width, then added a 6 disk RAID-Z2, that can be grown later. If you get more 4TB SSDs, (or larger), you can expand your 6 disk RAID-Z2 a disk at a time. Up until it gets to about 12 disks. Then you simple add another RAID-Z2 vDev.
The reason why I say it is practical to have asymmetrical vDevs, is that the current RAID-Zx expansion does not change the existing data’s to parity ratio. So starting off with one vDev at the conventional / conservative maximum of 12 disks keeps any data on it, with it’s best data to parity ratio, (10 data to 2 parity for full width writes).
But, that is getting deep into the weeds most people don’t need to know about.
Lots of great info here. For the OP, yes, one giant vdev is a bad idea. Lots of much smaller vdevs working together in a pool can be magical. Striping VDEVs, etc. for badass read times. But it also comes down to how much space you’re willing to sacrifice; it’s why I like raidz1 and odd numbered pools (3 disks, 5 disks, each pool losing a drive to parity).
Other have covered better than i could certain topics, on this
yep, totally legit, big disks have more tbw… but consider
with the mirror setup, you degrade 2 disks at same time
if the possibility to have a limited powerdown do to a boot pool failure is not a concern, at least just keep one of the disk as spare outside the server, and use in case of need
IMHO part of a good strategy (mirror or not) is to implement one of Joe’s utility, to always have fresh config to restore in case of need
small Intel Optane NVME disks are “terribly” cheap, 16gb especially are available for less than 5€ each and despite theyr limited size should resist well as boot pool
This is correct - power-loss prevention here is referring to the drives themselves. A UPS will not protect data in the case of a software crash or internal component (motherboard/PSU) failure.
