I’m building a new filserver for my home since my 13 year old TrueNAS server (with Core) is getting old. I’m going to use TrueNAS Scale on the new server.
The system at a glance:
AMD Epyc 4245P CPU
Asrock B650D4U motherboard
Kingston DDR5 ECC 48 GB RAM
Broadcom 9500-8i storage controller
4x Seagate Exos 16TB spinning drives
2x Kioxia Exceria G4 2TB Gen 5 NVMe SSDs
I’m thinking of different konfigurations to use with this hardware. I want to build for long time storage that will last at least 10 years (just like my old machine). I have thought about doing a metadata vdev for small files since lots of my backup that will be stored is small photos, documents and other small files. I have 10 GbE to the machine and really want to use the performance to do backups across the network (i have a Synology RS1219+ and a Ugreen NASync 4800 Plus in the network as well).
Is using the Kioxia NVMe SSDs for a metadata vdev a viable option you think? Should i purchase two more inexpensive SATA SSDs for this purpose instead? Or is the idea stupid?
Is your backup just a bunch of files, or do you use some kind of backup solution?
Perhaps they are ok. Some users do recommend using different SSD models for the sVDEV mirror. Thus eliminating the risk of simultaneous death from simultaneous write resource depletion and/or bugs in firmware.
Are SATA SSDs cheaper?
Also, just in case – a special VDEV is a “full-fledged” VDEV. If you lose this, your pool is gone.
I did something similar for my RAIDZ2 pool.
Few things I learned:
I used way more storage for my s vdev than originally intended. This might have to do with tails. A file that is 10GB, can fit into multiple 1MB record sizes and have a for example 80k tail. If you have set s vdev to 128k, that 80k tail of that 10GB file would end up on the s vdev, because it is smaller than 128k.
besides metadata, there is no point in storing small files like docs on s vdev, since that is even read fast from a HDD RAIDZ.
The previous point and the fact that a s vdev is critical for the pool, leads others here to recommend L2ARC over s vdev. L2ARC uses very little RAM and is even reboot persistent now. It still has some small drawbacks. It can’t “cache” metadata writes. It needs to warm up.
For vdev you can use the worst, cheapest SSD, since the workload is almost nothing and even the poorest SSD will outperform your HDDs. Just make sure to get two different brands/controllers and probably avoid QLC drives. If your Kioxia has a firmware bug and both drives act up, all your data is gone. If you have to different vendors, you avoid the “bad batch” risk.
Watch out, I have heard many people complain that mobo going bad after a few months.
Why so little?
No matter if you decide to go with L2ARC or s vdev, these are overkill and too expensive IMHO. Rather spend it on more RAM.
Very debatable statement. I think it depends on the use case.
Anyway, if @Durandal plans to only use it for metadata, then 2TB is way too much (for a 48TB pool). I think that even 240GB would be ok (especially if big media datasets would use 1-4MB recordsize).
I’m kinda on a budget here, but i plan to increase this to at least 96 GB or 192 GB in the future with more 48 GB DIMMs.
Mainly a backup of files, but i plan to use more backup apps in the future.
I did not know that, thanks for the info.
Alright, but for big file listings via SMB (lets say a folder with 10 000 pictures), storing the small files on an NVMe drive over 10 GbE must be faster, right?
Yep, that the svdev is critical is one of the points for me not using it for my pool. My current pool (9x4 TB) was created over 10 years ago and some disks have died during these years
Valid point, i have read alot of stuff for this build but i will probably read even more about the components
Thought i had read everything there is to know about this mobo, do you have a link perhaps?
Yeah my plan was to use the svdev for files smaller than lets say 5 MB, but having a separate metadata pool just for metadata could maybe speed things up as well.
Different backup solutions use different storage layouts. For example, vanilla windows backup&restore backup consists of many ~200MB zip archives; the actual files sizes don’t matter much.
Yeah, listing would be faster. Even though having 10k pictures in one directory is usually not the greatest idea. Also, sVDEV should be enough for just listing speed-up.
It doesn’t work that way. It has nothing to do with the file size; it is about block size, and (big) files usually consist of multiple blocks. Also, the maximum value of special_small_blocks for the dataset is 1M (at least in the GUI).
Perhaps this thread can be useful for explanations:
Wow, i was bad at doing my research for this motherboard i see. A simple search showed alot problems with some revisions of the motherboard. Too bad there’s a huge problems getting affordable AM5 socket motherboards with AMD Epyc 4005-support in my region (the nordics). We’ll see if i change motherboard or not.
A comment on using 2 way Mirrored sDev if using RAID-Z2/3 for the main HDD vDev. Using RAID-Z2 gets you 2 disks of redundancy before data loss, and RAID-Z3 gets you 3 disks of redundancy.
Using just 2 way Mirror for critical special vDevs, (Metadata, De-Dup, etc…), means if you lost both, the pool is lost. So, in some ways, using a 3 way Mirror for critical special vDevs when using RAID-Z2 makes sense. (Or using 4 way Mirror with RAID-Z3…)
I also agree that using 2, (or 3), different brands for the critical special vDevs is also suggested when designing a long lived NAS.
Note that the following are not critical vDevs:
Hot Spares
L2ARC
SLOG
Loss of any during normal use is a non-issue. Simply deal with them before shutdown or rebooting. Loss of a SLOG after crash or power loss, and before pool import is the only time you can loose data. Thus, for Enterprise use, they may Mirror or 3 way Mirror their SLOG(s).
You mention 10Gbit/ps Ethernet. While a special Metadata vDev can speed up access to the Metadata, it does not generally help with spinning disk access. That said, modern HDDs can in theory saturate 1Gbit/ps Ethernet. So jumping beyond Gigabit makes sense in that regard. Just warning you that some things just won't saturate 10Gbit/ps Ethernet when HDDs are used.
No, that is a metadata only task. This could be done with a s vdev and special_small_blocks=0
Disks dying isn’t a problem, there only should be less disks dying than what your redundancy can stomach I mean, the same is true for your HDD mirror or RAIDZ
But like swc-phil said, it is pretty important what kind of backup you do.
A RAIDZ2 with 4 drives will not only perform worse than striped mirrors, it will also offer a little bit less storage for smaller files because of padding.
Performance is not my worry if you use it for large files for something like syncthing.
For zvols or VMs on the other hand, it is pretty bad.
Probably irrelevant for your workload, if you don’t use 16k zvols or many small files.
RAIDZ can have a “padding downside” for small files that traditional RAIDs don’t have and you end up with less usable storage than expected.
Then your backups would be the same as your files. My main point was that despite a backup source consisting of thousands of small files, some backup solutions could make it into several big archive files. So the actual files-size histogram could be irrelevant (but that is not your case, as syncthing doesn’t change backed files).
Using a separate metadata VDEV can speed up directory-heavy workloads, especially with lots of small files. But it’s only worth it if your dataset actually benefits from faster metadata access. I apply the same logic in lead management—Phonexa helps me isolate performance bottlenecks and optimize the parts of the system that matter most, not just throw hardware at the problem.
I mean, the same is true for your HDD mirror or RAIDZ