Feedback or Tips/Ideas for my setup

TrueNAS General
1

Greetings everyone,

I was hoping to get some feedback on my setup for TrueNAS and possible upgrades or solutions.

(Long post, sorry about that!)

This has been a year long project, I purchased the first part last year October.

It still isn’t up as I need a new PSU.

I originally had a 5800x for it since I wanted gameservers but I really wanted ECC ram and found a great deal on Supermicro/Xeon combo (power consumption is fine).

Fractal Define 7XL

Supermicro X9DR3-F

64gb DDR3 ECC

x2 E5-2670 (16c/32t) 2.6GHz-3.3GHz

Quadro P400 (for plex)

2.5gbe NIC to 2.5gbe switch to 2.5gbe NIC on main computer

LSI 9300 with 2x4 SATA cables

M.2 x16 PCIE adapter 4 slots

2x240gb (mirrored boot)

6x8TB Exos Raidz2 for 32TB space

1x12TB Exos for Plex

1x1TB NVME L2ARC for main storage

1x1TB NVME for apps (maybe two drives for redundancy?)

With Black Friday coming up, I’ve been wanting to do any possible upgrades then to save a few bucks.

I have a bunch of extra 8tb drives but was thinking having cold spares is better than all that unused space, especially since the drives are from a datacenter.

My main use case is Photography, Video Editing (storage for 4k videos), Family Photos, some Plex, and just general pc storage.

I have a lot of photos which are critical.

I’m putting a Synology NAS at a family members place for off-site redundancy.

I’m also using Google Drive, but I’d rather use the NAS going forward and backing up to BackBlaze for archive.

I have limited experience with TrueNAS and none with BackBlaze.

If anyone got any feedback, ideas/tips I’d love to hear them, any is appreciated!

2

If you are in the US, consider buying used helium filled drives with a 5 year warranty from goharddrive.com. They run cooler, use less power, and are no longer available new. Save a ton of money, put them through the Smart / Badblock wringer, then deploy as needed. I bought a bunch of spares when my he10s (that’s the 10TB version) were available for $76 a pop. You’d want to search for he8 models. Or go denser and reduce the number of drives to reach whatever capacity you’re after.

1 Like
3

Never mind…. Are tariffs driving HDD prices crazy? I haven’t seen prices this high since flooding in Asia knocked out all the factories a few years ago. Who knew I was smart enough to buy an appreciating asset? My drives are now worth 2x what I paid for them. Wow!:grimacing:

1 Like
4

Unfortunately not US, I’m from Norway.

If I wanted to buy new exos drives in same size, I’d have to sell nearly 3 of my harddrives on market just to buy 1, that’d hurt :stuck_out_tongue:

Not a whole lot of refurbished drives in Norway, not really any to be honest, and used exos drives are few.

Homelab/server scene is really small in Norway.

Especially as I have 14 8tb drives and only using 6.

They got around 50k hours on them and only around 6 power on cycles, not sure how long they’d last?

Came straight from a datacentre.

5

Consider not getting L2ARC until you run your system for a bit & confirm it’d be of any benefit. Don’t fall for ‘more hardware = more betterer’. L2ARC, SLOG, and special vdevs have their own uses, costs, and risks. There is a very strong chance that if you cannot explain in detail why they fit your specific usecase, then you don’t actually need them.

That’d also open up a slot for 2 nvme drives for apps & maybe in the future VMs so you can have it mirrored.

I’m also confused why you’d get a single drive to feed Plex. Why not add another 8TB drive (or two) to the main pool & have your Plex media as its own seperate dataset on that pool? That way you don’t have to go through the pain of getting all that media back if a single drive dies?

Otherwise, I don’t see anything wrong with the equipment itself, so my advice would be to burn everything in. Memtests, cpu tests, burn in the drives with backblocks, slap a fan onto the HBA, make sure HBA is running latest firmware, make sure everything has enough cooling, etc. etc.

Oh, maybe a UPS - not often that folks regret getting a UPS.

1 Like
6

>Never mind…. Are tariffs driving HDD prices crazy?

No, these are secondhand drives, imported before the tariffs. It’s the AI craze: they’re buying up HDDs (and lots of other stuff) to build datacenters for AI, because AI models need to store tons of data, and this is constricting the market. Datacenters are keeping drives longer too, further shrinking the supply of secondhand drives. I got my drives about a year ago, just before the prices started shooting way up; I’m so glad I didn’t wait even a day longer.

1 Like
7

Thank you for that clarification. Makes sense that even the secondhand market will get affected by the AI data center craze.

The amount of money that is being burned right now in AI-bubble-land is going to make the 2001 IT-bubble look like child’s play (PE ratios of 250+ for a glorified car company??? Cisco topped out at 105, etc.).

Another big difference: fiber installed back then may have gone dark but eventually was made use of. The data centers being built today are unlikely to ever yield any economic return.

I’m glad I have my spares.

8

My main reason for L2ARC was the amount of photos I have, and the hope to increase thumbnail and file loading as much as possible.
I’m going back and forward quite a lot, so having a cache to speed things up would be really nice especially since I have a spare NVME, if it improves speed any noticeably?

Will a L2ARC drive help with photo editing straight off the NAS as well?

Regarding single drive, I was thinking I didn’t want to take the precious space of the main storage, especially since It’s raid protected, and my movies/series are not important at all.

I’ll definitely get a UPS, and do some testing, luckily the HBA got a fan on it already, does it need more cooling than the little fan?

Hoping to get some good prices on fans on Black Friday, some quiet Noctua would be nice, until then I got some cheap fans laying around.

I have seen some cheap IronWolf Pro 16tbs, and wondering if I should switch out to 3 of those, raidz1, 32tb usable space.
My 6 exos drives got 50,000 hours on them, 6 power cycles, though I’m unsure the chances of failure for them at those hours, any idea?

9

Generally, maxing out available ram will produce better results than L2ARC. If you really want to use it, then test without it installed first. At worst you’ll get worse performance & can remove it.

Maybe consider using the nvme for when you have active projects you’re working on & then the hdds as a more secure cold storage?

At least consider a mirror for the movies :frowning: it ain’t important data, but it’d suck to lose them.

As long as you got a fan slapped onto the HBA, you’ll be fine. Once again though, make sure it is on the latest IT firmware available. sas2flash or sas3flash command will help confirm.

As far as drives? Hard to say, but you mention having cold spares, so you should be fine.

1 Like
10

L2ARC is a pretty risk-free way to see if things get speedier or not. I happen to be a fan of a metadata=only L2ARC because I use rsync for some backups and I find it speeds up directory browsing in general too. For rsync backups, L2ARC sped up backups by 12x for my mostly WORM NAS.
I have since moved on sVDEV - but that is riskier than L2ARC (see my sVDEV resource page) and unlike L2ARC cannot be removed from the pool.

2 Likes