i have built one previous truenas scale system, learned from mistakes, but I still consider myself a novice and am seeking some advice regarding the design for a second system.
Usecase: main storage for home, some apps
The hardware of the new system consists of:
Dell R540, 64Gb RAM, HBA355i with 8 disk slots of which 2xSSD (960GB) and 6xCMR HDD (14TB), Dell Boss Card with 2xNVMe (240GB), integrated NIC 1GB.
The 6 HDD will make up the main storage pool but I am unsure about ARC2, System drive, and App drive and their redundancy.
My current thinking: 2 NVMe mirrored → ARC2, 1 SSD → system, 1 SSD → APPS, 6 x HDD storage pool.
Is an ARC2 overkill for my home use case, especially since I only have a 1GB NIC at the moment? Would it make sense to have once I have a 10GB NIC?
For the system drive, should I have it mirrored instead of just one drive? Would it be better on NVMe or SSD?
How would someone more experienced set it up for a home usecase for max reliability?
Just one (relatively inexperienced) user’s personal opinions (get other opinions before making a decision)…
IMO L2ARC is overkill for a home server and it takes memory away from the primary ARC in RAM. Try it without and check your Reports/ZFS to see what your ARC hit rate is. You can try an L2ARC and then remove it if it is not beneficial.
An L2ARC can be metadata and data OR it can be just metadata-only - a metadata-only L2ARC may take less memory away from primary ARC. You can also try metadata-only and see if you can spot any performance improvement. Ditto for try-before-buy.
If you decide to have an L2ARC there is no need to mirror it.
10Gb NIC is only going to give you something if your LAN bandwidth is 10Gb end-to-end. So if you have a 1Gb NIC already and your LAN isn’t 1Gb end-to-end then spend the money on upgrading your LAN.
For 6x14TB drives, you should really use RAIDZ2 or RAIDZ3.
The apps pool should be used for both apps themselves and (where possible due to size) the app data. If you are using it for data, ideally it would be mirrored (if not replicate it to your HDD pool).
NVMe is a LOT faster than SATA SSD - if you have no other uses for NVMe consider using them for the apps pool (mirrored). Perhaps save an SSD for something else and buy an extra 14TB drive.
Mirroring the system drive is less important providing that you keep a copy of the system configuration file. Implement @joeschmuck 's Multi-Report script so you get sent this by email once per week.
IMO performance of boot is less important because once booted, most stuff needed is already executing or in ARC.
Reliability - Stress test before production use, redundancy, UPS, SMART monitoring, SMART tests, Snapshots, backups.
Thank you, @Protopia! Following your comments, i would:
skip L2arc
you suggested to use NVMe for APP and APP DATA because it is faster, but I did some more digging (link), and apparently, the DELL BOSS-S1 card is not meant to be a fast storage device. It’s meant to give you redundant boot storage, without losing storage at the front or back of the server. Therefore SATA is fine and the card can’t handle NVMe speeds anyway. If this is true, I would prefer to use it as intended - a mirrored boot drive as it is only 240Gb.
use the two SSD in mirror for APP and DATA (1TB)
6 x 14TB in z2 for storage
Implement Multi-Report script by @joeschmuck (will research it)
I really have to wonder if this is true of anyone these days. GbE has been the standard for 20 years or so at this point. Certainly, if you have any “Fast Ethernet” gear, get rid of it and switch to Gigabit, but I’m thinking this would be a very rare situation these days.
