General recommendations and sanity check on hardware

TrueNAS General
, ,
1

Greetings community.

Another noob looking for expert insight - my sincerest apologies, I’m sure this type of thread has been done to death a million times over.

A while back I yolo’ed a TN Scale setup and somehow lost two disks in a vdev
That was a learning experience.

Since I’ve had other things to do bu have tried to research ZFS (on TN scale) some more and have purchased a lot of goodies with intend on rebuilding bigger than last time.

I have some questions I’ll post at the bottom but here is the plan/setup (build still TBD these are just the parts i have)

Server:
Motherboard: ASUS Z270-A
CPU: Intel i7-7700
RAM: 2x CORSAIR Vengeance DDR4 16GB 2400MHz CL16
Fans: 2x 120mm, 2x 80mm, 4x 40mm Noctua fans
M.2 nve: Samsung 970 pro
2.5" SSD: Innodisk 2TB 3TG6-P
2.5" bay: 2x Icy Dock ExpressCage MB038SP-B (16 bays total)
Case: Silverstone RM42-502
PSU: Corsair VS550
HBAi: LSI 9300-i16 IT mode
HBAe: LSI 9300-8E
Rack: Caymon SPR642 (600mm deep so network rack)

Because the rack is shallow I opted to try my hand at adding a JBOD/DAS to the server

JBOD:
Controller: CSE-PTJBOD-CB2 Power board for JBOD
Expander: Dell JBOD 24 internal 12 external lane SAS2 6Gbps
PSU: CORSAIR RM Series RM650
Case: EDENSE ED424H48

I also have:
22x ADATA ISSS316 2.5" SATA SSDs
6x WD Red Plus WD120EFBX

The intent:
Store all my linux distros and personal data (with intent to figure out a off-site backup of the personal data) and be off-site data backup for friends and family.
Possibly also store computer snapshots/backups as well has hosting various services example like HA, Unifi, pihole, tailscale

The plan (so far):
Buy 8 more WD disks - I’d like to get 2 full vdevs running and have 2 drives to spare. Also buy 2x16GB extra stick of memory ( the board supports 64GB max)
Get the jbod built, get proxmox up and running again (something about 8.2 now having updates? maybe) get Truenas up and running again if it isn’t already on the original install.
Then start over completely on the ZFS config, shares, IAM, apps etc.
Since the jbod has 6x 4 SATA backplanes I was thinking of spreading vdevs across the backplanes so that i can lose 2 entire backplanes with Z2. Sound neat not sure it it makes sense though.

ZFS plan
12x12TB WDs in 2 RAID-Z2vdevs - 2x2TB as spare (or future expansion)
4x1TB Adata SSDs for L2ARC (because i have an abundance of them)
SLOG: Unsure - thinking about buying 2x Kingston DC600M SSD 1920GB for mirror because they have PLP and I reckon the Adatas aren’t good enough?
Metadata: Unsure again - maybe more DC600M’s? Maybe I can add these down the line if i need performance?
6x 1TB Adata in RAID-Z2 for some faster storage for maybe VMs

Questions:

  1. First of all I’d be very excited for any input or recommendations baring in mind that besides a few extra drives I have most of the hardware on hand in a big pile - might share a picture
  2. Lightly worried that my 6x WDs are serial sequence from same vendor and if I should distribute the 6 drives over 6 vdevs with Z2 to minimize risk.
  3. I’m worried about adding 1 vdev at a time that the data isn’t equally distributed across all vdevs over time and that will come back and haunt me somehow. I’ve looked at redistribution scripts but still feel weary.
  4. I’m unsure about adding/expanding L2ARC, ZIL and Metadata vdevs over time if it will need to be designed for today and if I should just do ot not
  5. Uncertainty about using these Adata drives for anymore more than L2ARC and fast storage array
  6. Uncertain about the need or benefit og metadata special vdev and if (assuming the 3% rule) if i need to calculate for that today of can grow into it easily. at 24 disks in 4 cedecs of Z2 that’s long term 192TB meaning 5.76GB which is a lot esp if it needs to run in a mirror too. Maybe I could do 2x2TB now and add additional 2x2TB later and stripe the mirrors for 4TB?

Overall I think I just need some assurance or guidance so that I’m not painting myself into a corner - now is when I have time to change direction or swap strategy.
What would you do with my intent and available hardware?

Long-long term additons I’ve considered:
10Gbe (but i have a 4x1bge card for now I could look into teaming if i need more than 1 gbps)
Used quadro or similar to transcode with
Adding 12 more HDDs
Swapping mb/cpu to something more reliable with ecc support.
Swapping ye 'ol PSU to another corsair RM600

2

Strongly consider changing motherboard to get ECC as you are also limiting yourself to 64GB on current choice.

With 2 HBA, check what is required for 10Gb network cards, might be out of slots already.

Drop the L2ARC plans. 32 to 64GB is starting point for considering that per the Scale documentation hardware guide. View your ARC hit ratios first. Your 4TB would use 80GB of RAM per the memory sizing guidelines

2 Likes
3

Awesome feedback - tyvm, I totally missed the L2ARC on memory consumption.

Might have to look into a new board, cpu and memory. If you have any recommendations they’d be welcome. If costs on get out of hand I might have proceed with no special vdevs and a memory upgrade for now.

4

I’m a fan of 6, 7 or 8-way RAIDZ2.

Your current plan fits with that, although I’m not 100% certain what you mean by “2x2TB as spare”

Do you mean 2x12TB? If so, just use them as part of your 12TB RaidZ2s. I don’t think it makes a lot of sense to use spares with RaidZ2… I use a spare with mirrors… but with RaidZ2 I normally don’t bother, and instead can get a replacement drive delivered within a day or two… and I still burn in that replacement before I put it into service.

I’d skip the L2ARC… check your ARC hit ratio… if its 99+% then at most you’re only going to service 1% of hits from L2ARC.

BUT maybe you could use the 3 of those drives as a metadata device… triple mirrored. Maybe using one of them as an an L2ARC?

For SLOG, I still think one of the best options is an Optane drive (not the very very low end ones, but something like a p4801. yes. 2nd hand… but they’re virtually indestructible.

And then your flash pool. A good idea.

Ideally you would use a mainboard that supports ECC, but if you can’t, the oh well.

As you know by now, RAID is not a backup, and you should still have a separate backup for anything you don’t want to lose… I use another TrueNAS system for the backup… and replicate to it. Ie when you upgrade… then the older system becomes the backup system :wink:

2 Likes
5

Okay I think I need to reshuffle the budget slightly and drop going for 2 vdevs of 6 HDDs for now and just stick to 1 vdev in Z2 that opens up my budget significantly

For SLOG, I still think one of the best options is an Optane drive (not the very very low end ones, but something like a p4801. yes. 2nd hand… but they’re virtually indestructible.

Like this Intel OPTANE P4801X?
Should I get 2 and mirror? I think I’ll get a PCIe 3.0 x8 to 2x M.2

I’d skip the L2ARC… check your ARC hit ratio… if its 99+% then at most you’re only going to service 1% of hits from L2ARC.

Gotcha - I’ll park that until i can see the ARC hit ratio, makes sense!

Ideally you would use a mainboard that supports ECC, but if you can’t, the oh well.

I find it hard both financially and logistically (I’m scandinavian) to find good servergrade hardware that is affordable.
I found this slightly expensive ROMED8-2T I could pair with a 2nd hand EPYC
Would there be any issues with mixing OPTANE and EPYC?

I understand EPYC can’t du hardware a accelarated transcoding so I’m thinking of adding a cheap quadro or similar which I can fit in the budget if the EPYC is not new.

Also not sure if I should go for ROMED8-2T or ROMED8-2T/BCM.
It seems to come down to Intel X550-AT2 vs Broadcom BCM57416 and I’m leaning more heavily towards Intel - they have served me right in the past.

6

SLOG is only for sync writes, which I do not see with your stated uses. So drop that.
If you do have a use for it, mirrorring is not required: SLOG is not pool-critical.

No L2ARC before you know you’ll benefit from one—and then, you’d need 64 GB RAM or more, so definitively a motherboard change.

If you go for EPYC, you may just use the CPU for transcoding. How many concurrent streams, and how often?
And defintively Intel for the NIC. Broadcom support is a bit of an unknown.

As said, if you’re using a spare with raidz2 you can as well go straight for raidz3. (If you have many vdevs, a handful of collective spares would make sense, but then you’re also in dRAID territory.)

As for metadata vdev, mind that this is pool-critical so you’d rather want 3-way mirror or more—especially as you seem to be quite cautious.

As an alternative to dual HBAs, a single 9300/9305 HBA and an AEC-82885T expander could do. The expander only draws power from the PCIe slot, and can even be powered from 4-pin Molex.

7

Your (all of you’s) feedback is appreciated! Thank you so much. :heart:

Understood thanks - just read a little up on sync writes and I agree I plan to run my VM from flash and back them up to HDD array.
The intent was only to speed up file transfers but it might be better dealt with using more RAM/ARC.

Understood, as hammered in from the previous responses. I’ll wait to check ARC hit ratios. :sweat_smile:
I’ll grab the ROMED8-2T with the Intel NIC and figure out a CPU + RAM. Pricing on 64GB LRDIMM ECC sticks is cheap enough I might grab two for 128 total - esp since I’m dropping the SLOG.

To begin with 2 max concurrent but maybe up to 5 or 6 eventually if things go well. as for how often I’d say daily usage of 2-3 hours depending

yes, just forget about that nonsense i didnt explain myself very well. The point was just that I wanted to keep a few spares on hand on a shelf in case.

Interesting.Yes Most of the data is reproducable but some it wont be (and ripping a sea of bluerays - while easy is not fun to redo) so overall I’m being cautios.
With a potential 196TB max and 3% rule that’s 5.75TB of meta data if full.
Though with ZFS I believe the rule of thumb is to keep it under 80% capacity so more like 4.61TB. I’ll have to calculate it - Wendell from L1T has a video about calculating metadata space requirements. I’ll start with no metadata drive for now and add if needed down the line.

I already have the two HBAs sitting. So unless you see an issue with running two that’s going to be it for now :rofl:

Great answers and guidance so far - I’ll take as much more as you’ll part with. I think my main concern with adding every kind of speciality vdev fro mthe get-go was concerns of designing something without the flexibillity to grow into.

I already feel much more at ease following your advice. Investing in a board and ECC RAM (esp above 64GB) will benefit me more than an i7 on an old gamer board.

8

Indeed.

Mind that losing the special vdev means loosing the whole pool, all 200 TB of it, not just the metadata and small files.
Metadata vdev is best introduced at pool creation, since old data will NOT be moved. With raidz#, it is also an irreversible operation. So, with an EPYC board and plenty of RAM and PCIe lanes, your best option is a persistent L2ARC for metadata.

1 Like
9

Understood and I can always add that later. I realize this is an impossible question answer but how much L2ARC per TB of storage? I realize it might depend mostly on how much memory I have and what data is accessed but might there be a rule-of-thumb thing to go by?

SmallBarky already made me aware of the memory to L2ARC ratio of 5x to 20x or at least 1GB of memory per GB of L2ARC.

10

Should be 1GB per 50GB of L2ARC. (Scale docs, hardware section on mem sizing)

Rule of thumb for L2ARC is going by the ARC hit ratio. You don’t expect rarely used data to be in ARC, nor L2ARC. Those caches are more meant for data that is read often.
Sort of like if you stored daily newspapers. You would expect the cache to have today and maybe yesterdays paper in the cache but if I was to try to read news from 5 months ago, it would have to get it from the disk.

1 Like
11

Thanks!

And all on the thread who have helped steer me right - I took the plunge at bought a new asrock rack ROMED8-2T and a used EPYC 7502P.
I also treated myself to a PNY Quadro RTX5000, I’m told it might be possible to do HDR to SDR tonemapping during transcoding so keen to see if that can work.

1 Like