TrueNAS build for JBODs

[over9000]

I am currently a Synology user with 24 drives. I can add another 12 drives, but then my NAS is maxed out.

Hence I am looking at alternatives.
Currently my plan is to

• build a 2U server - no need for drive bays in the server itself
• buy a large JBOD (at least 24 drives)
• True NAS Core

My rack depth is limited to approximately 70cm; thus the very large JBODs don’t fit.

I am considering these components for a new build for the server and wanted to get your thoughts.

rough prices in USD for a total of 1814.

• 375 case - CSE-523L-505B
• 526 mainboard - B650D4U
• 313 ram - 48G ksm56e46bd8km-48hm
• 310 CPU - AMD EPYC 4245P
• 63 boot SATA SSD - kingston a400
• 228 HBA 9500-8e
• tbd cpu cooler

I think many people buy used hardware (Dell,…) which comes in much cheaper. I am a bit hesitant to buy a x20, x30 or x40 gen Dell. I fear the use a TON of power. The newer generations x50 and x60 seem to be on the pricier side (comparable to what the components above cost).

I wanted to get your feedback on this.

[TheColin21]

I don’t have much experience with JBOD chassis for TrueNAS but there are some other things that caught my eye here:

• I wouldn’t use TrueNAS CORE for a new build anymore. It’s development is basically over. It will only get some security or major bug fixes in the future. Use TrueNAS SCALE/CE instead.
• the memory is not on the boads QVL - I’d choose RAM that’s listed there for a NAS/server build.
• As I live in the EU I am not sure about the pricing but if I understand your list correctly you want to use just 1 single 48GB module of RAM? I’d go for two modules for dual channel. Single channel is not as bad with DDR5 as it was previously but still a substential downgrade comapred to dual channel. Also for at least 24 drives I’d use more than 48GB of RAM (assuming you use relatively large drives, maybe 16-20TB)
• Again - I don’t know US prices but a Ryzen 9600 would probably be much cheaper than an EPYC 4245P and still supports ECC memory. I don’t see any reason to use the EPYC here.
• Your boot SSD has no DRAM cache and uses QLC modules - that’s bad for performance and longevity. TrueNAS has no need for especially fast or resiliant boot drivers but with small SATA SSD prices I don’t see a reason to use a bad one when you can have a good one for nearly the same price. Also 63USD seems pricey for one of those so I hope you mean two of them for mirroring? Using anything more than the smallest 120/240GB variants would be a waste of money but mirroring would be worth it imho.

[over9000]

thanks. I am actually in Europe (in a non EUR countery) I just converted prices to USD.

• yes, the price is for two SATA ssds
• I wasn’t quite sure about Ryzen and ECC. But I guess they do support ECC (though there are mixed reports online)
• as for RAM, the QVL list only has a single 48GB ECC stick - SR6G7UD5385MB01. I cannot find this anywhere. I never had issues with Kingston RAM - I may just stick with the one I listed above.
• Indeed I can get a Ryzen 9600 for 255 USD, I may consider that (I have little knowledge about PCIE lanes, I think the epyc has 4 more that are useable).
• if you can recommend a small boot SATA SSD, I am all ears! I looked up a few enterprise drives, but most were much pricier than what I listed above (63 USD for TWO drives)

[TheColin21]

@over9000

• yes, the price is for two SATA ssds

Thats good, I’d still choose another SSD though.

@over9000

• I wasn’t quite sure about Ryzen and ECC. But I guess they do support ECC (though there are mixed reports online)

It depends on the model. AM4 non-Pro chips with iGPU (APUs) didn’t but I think nearly all AM5 models do.

@over9000

as for RAM, the QVL list only has a single 48GB ECC stick - SR6G7UD5385MB01. I cannot find this anywhere. I never had issues with Kingston RAM - I may just stick with the one I listed above.

Understandable. You are going to use more than 1 DIMM though right?

@over9000

Indeed I can get a Ryzen 9600 for 255 USD, I may consider that (I have little knowledge about PCIE lanes, I think the epyc has 4 more that are useable).

As far as I could see both have 28 lanes with 24 usable.

@over9000

if you can recommend a small boot SATA SSD, I am all ears! I looked up a few enterprise drives, but most were much pricier than what I listed above

If you have some PCIe lanes left: small Optane modules are very cheap on Aliexpress and such. You could also use these in USB enclosures.
For SATA SSDs: Sandisk Ultra, Samsung 870 Evo, WD Blue can all be had under 50€ for the 250-256GB versions in Germany and have DRAM cache, at least 100TBW and use TLC modules.

[over9000]

thanks.
Just now I purchased two used Intel SSDPro 5400s 240GB for 17 EUR each

I may eventually look into an Optane usnig m.2 to u.2 (for SLOG). But I first need to procure the hardware.

[TheColin21]

@over9000

I may eventually look into an Optane usnig m.2 to u.2 (for SLOG).

Just to be sure: do you actually have lots of synchronous writes?

[over9000]

I actually don’t know.

here is last’ weeks IOPS from my Synology.

The blue curve is not too relevant (I have an SSD pool which will likely stay on the Synology for a while)
the red curve is the HDD pool

at which point is an optane meaningful?

… .imgur …
com/a/BSh4Y2z
(I am not allowed to post links…? )

[etorix]

Take the tutorial by the forum bot to post images.

And please do not convert prices to USD if you’re in Europe, use your local currency—and give us a clue where you are, for appropriate local advice if we can.

[over9000]

thanks
I am in Switzerland (CHF).
I usually try to post in EUR or USD as this will be somewhat simpler to work with for most readers.
will look into the tutorial now

[over9000]

[Album] imgur.com

here is the IOPS link of my current NAS (red curve is the HDD pool).
I am not quite sure if this helps to assess my need for an Optane/SLOG device

[Farout]

I’m from Switzerland aswell. I get my server stuff usually used from ebay (china), serverschmiede.com or servershop24.de. You will have to pay import duties.

SLOG is not a write cache. You only need it for sync writes. So databases, Block storage (iSCSI, zvols for VMs), and NFS.

[over9000]

thanks.
Well, I don’t use iSCSI, I don’t use block storage.
But I have a lot of NFS clients (ubuntu vm and such). As in /etc/fstab has a nfs mount command. Does this mean I could benefit from SLOG ? I don’t think that my NFS mounts have any ‘sync’ option though.

[Farout]

If you are using NFS without sync, then you wont benefit from a slog.

[Stux]

@over9000

• build a 2U server - no need for drive bays in the server itself
• buy a large JBOD (at least 24 drives)

Have you considered simply building a 4U 24 bay system?

Build Report: Norco RPC-4224, SuperMicro X10-SRi-F, Xeon E5-2699A v4 Part 2

Continuing on from the Part 1 on the old forum. [image] I recently upgraded the CPU from E5-1650 v4 to the Unicorn model… the E5-2699A v4. This is an upgrade from 6 cores to 22! with a pretty good base clock. Broadwell era Xeons are cheap now and I get a kick out of dropping in a CPU with an RRP of 10K. With scale I now have a bunch of windows VMs etc running on this 7 year old system Previously, I was planning on virtualizing FreeNAS on top of ESXi, and as part of that I ob…

[over9000]

No I have not.
The supermicro JBOD I ordered has 45 bays. I am not quite sure with how many bays I want to start, but I want flexibility (spare room) to go beyond 24. I am well aware that a 24 bay server can easily be expanded with more JBODs.
But in my case drives and server are cleanly separated and I only have 1 4U JBOD.

[over9000]

if anyone is interested here is what I settled on:

• case - CSE-523L-505B
• mainboard - B650D4U
• 96 GB of ECC RAM - 48G ksm56e46bd8km-48hm
• CPU - AMD EPYC 4245P
• mirrored boot SATA SSD - used Intel (possibly 5400 pro)
• HBA - LSI 9500-8e
• Mellanox 25G NIC

plus a 45 bay Supermicro JBOD (it has two of the flat SAS connectors).
I have not installed this yet. I gather going for TrueNas Scale is the better choice these days (vs Core)