Mooglestiltzkin's Build Log: Truenas build recommendation am5 2024?

[mooglestiltzkin]

ty for sharing

IBM M1015 (LSI9211-8i) IT Mode

is it this one?
https://www.aliexpress.com/item/1005005225192396.html

says sas though. im assuming it works for sata? cauz ur spec say sata hdds.

with this card, can you setup the raidz for the hard drives in truenas? using the selected drives. Does it matter if it’s IT mode or simply no raid?

*update

added some interesting data points so you don’t have to find them

Every SAS-Card I have used has worked with SATA SSDs.

The price difference between a card that does 6gbps and another that does 12gbps is negligible.

Just buy the 12.

IT mode = firmware without RAID, which is what you want when using unraid, as it doesn’t use or need hardware raid.

Yes, HBA is the same as IT-mode

An HBA card isn’t necessarily an improvement to your mobo’s built-in controller. The card simply adds more ports and manages it with its own SATA controller, taking the work load off the processor. If you plan on adding drives, you’ll need an HBA card.

*watching Christian’s video he explained using HBA card you can passthrough the sata drives to truenas easily. But without it (aka u use the mobo sata ports), you have to add the drives in proxmox manually. Christian’s youtube video for proxmox truenas explains how.

Yes, if you need more drives, and you’re out of SATA ports on your mobo, this is what you need.

You will also likely need a breakout cable, since it sounds like you aren’t using a backplane.

???

sauce
https://www.reddit.com/r/unRAID/comments/ychjgn/recommended_hba_it_mode_sas_controller/

[mooglestiltzkin]

ULTIMATE Power Supply Tutorial: NEW PSU standards Explained from NOOB to EXPERT! [What is PSU?]

this video explains a lot about psu. watch this first, so u know what to get (if ur a newbie like me and didnt know anything about it). it’s 10month old but… it’s still alright.

also on the motherboard page, check for compatibility, check that the psu u r getting is listed. I also saw that pcpartpicker also has a compatibility list, but i mostly rely on the motherboard compatibility list first.

Check based on tierlist to know what’s a good psu

PSU Tier List rev. 17.0g - Cultists Network

Best PSU Picks 2024 - Hardware Busters

Use calculator to figure out what you need
https://pc-builds.com/power-supply-calculator

check price, look for a good deal

[etorix]

@mooglestiltzkin

also not sure if he can fit the 4x 3.5’’ hdds ontop of the 3 m.2 ssds i was using. thats why i went with a bigger case, but like he pointed out there are 1u cases with front 3.5 drive bays for 4 or more drives

1U can only take 4 LFF (3.5") bays. And then it’s awfully noisy because you have to cool the server with a wall of high-speed, high-static pressure, 40 mm fans.

@mooglestiltzkin

says sas though. im assuming it works for sata? cauz ur spec say sata hdds.

If you don’t mind to read:

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

PSU sizing for NAS (already posted):

Proper Power Supply Sizing Guidance

[mooglestiltzkin]

i will read ty. i have much to learn.

i’ve heard about lba’s in passing, but never used one personally. i got some understanding now

it’s just… the mobo has 6 sata ports already, and i probably wont go proxmox for now. but i’ll look into it

@etorix

PSU sizing for NAS (already posted):

i also added the recent psu youtube cauz i wish i had posted that earlier but missed it. was useful for new nas/pc builders, cauz it explains quite a lot in whole youtube.

@etorix

1U can only take 4 LFF (3.5") bays. And then it’s awfully noisy because you have to cool the server with a wall of high-speed, high-static pressure, 40 mm fans.

yeah… that sounds awful

i noticed my case has a place for a front fan (fan was not included…)

I checked what fans i had on hand. i had some 120mm gentle typhoons from a water cooling system i no longer use.

But i check the specs, seems i need a 80mm fan.

I want to check first whether i really do need the fan to position center front of case to add more airflow. will check truenas temp readings for cpu and hard drives first before i decide

Common Fan Layout Mistakes You Might be Making with Your PC Build! PC airflow tips and tricks

the sfp+ 10g pcie is almost here. probably receive by tomorrow.

Cooler still stuck in china at this rate maybe get by next week.

[mooglestiltzkin]

Getting the Most Performance out of TrueNAS and ZFS

was re-examining truenas setup specifically optimization.

goal is 10g networking speeds. my networking gear met the requirement. so hoping the new nas does the trick.

but then tim reminded me, for 10g if your drives does 150 MB/s then you’d need 8-9 drives to hit that 10g speed mark.

So if you don’t plan to use that many drives, then the other option is to setup a slog using ssd mirrored.

Is it safe to use slog without a ups? and how fast do ssds wear out when using as a slog?

[Fleshmauler]

@mooglestiltzkin

Is it safe to use slog without a ups? and how fast do ssds wear out when using as a slog?

If I remember slog is only useful for sync writes - if you have those disabled or aren’t making use of them, then slog is a detriment or at best not useful.

Don’t fall into the trap of thinking you need slog/l2arc/meta data vdev thinking that “more hardware = more faster = more better”. Deploy only if you have specific use case that you understand how & why it’d be improved with them. Otherwise they’ll either case additional points of failure with no benefit.

Unexpected power off can cause inflight writes to fail, not sure how much more risky it is with slog vs by default. UPS is generally just a good thing to have.

If in the end you find that your HDDs only (for example) use 6gig max of your 10gig link - be happy. This means you got 4gigs minimum for your VMs & apps which hopefully run off of your M.2s/SSDs without needing a second link.

Also generally expect that writes will be slower than reads.

[Stux]

@mooglestiltzkin

but then tim reminded me, for 10g if your drives does 150 MB/s then you’d need 8-9 drives to hit that 10g speed mark.

That’s right.

I switched from 2x 8-wide Raidz2 to mirrors to hit 10gb.

But, if you’re not bottlenecked by your LAN… then you’re getting the full benefit of your disks

SLOG is not a write cache, but a good SLOG is a great way to solve a slow sync write problem, but only if you have a “slow sync write” problem

[mooglestiltzkin]

@Fleshmauler

Don’t fall into the trap of thinking you need slog/l2arc/meta data vdev thinking that “more hardware = more faster = more better”. Deploy only if you have specific use case that you understand how & why it’d be improved with them. Otherwise they’ll either case additional points of failure with no benefit.

thats what i was worried about

i re-checked the hdd drives i am using. surprisingly no proper reviews about it. t’s 7200 rpm drives with 256mb cache

all i could fine was this, quoting at least 250 MB/s read/write
https://www.reddit.com/r/DataHoarder/comments/1cu0qdc/seagate_exos_refurbished_read_speed_hitting/

as for noise, i got 2 in my nas atm (1 being block burned), its within my tolerance level so not an issue for me.

Backblaze Also Points to Seagate Exos X12 Reliability Challenges

for warranty, mine seems to be in a similar situation as this amazon buy (i sourced mine elsewhere, different ecommerce platform)

These drives (I bought 2) were sold as NEW, with 5 year warranty.

As I typically do, I checked Seagate Warranty as soon as I got these drives…

Seagate says these drives are NOT under warranty, and says that you must deal with the seller who is NOT the manufacturer but a “renewer” of drives for any warranty offered by them.

Seagate says these are DATA CENTER RETURNS and were sold in quantity to a data center.

If you buy these drives and one or more fail, The best you’ll get is another data center swap-out drive if your time for a full refund has expired which is about 30 days.

I was lucky;
The drives failed within 10 hours of spin-up with S.M.A.R.T. error 187 which is a FATAL DRIVE error so I was able to return for a full refund, but you can’t return to AMAZON, no, they make you deal with Maestro who has to authorize the return and refund or replacement…

Notice the picture… 1 error is fatal and these drives, in less than 10 hours of use have over 40 logged, uncorrectable errors.

The following information comes from, Seagate, WesternDigital, ArgusMonitor, Asustor, Google, and almost every DataCenter you research…
S.M.A.R.T. Number 187 reports the number of reads that could not be corrected using hardware ECC. Drives with 0 uncorrectable errors hardly ever fail.
This error code is one of the SMART stats we use to determine hard drive failure;

Every Data Center has a SMART-187 policy that reads: once SMART 187 goes above 0, we schedule the drive for immediate replacement because its total failure in imminent.

as long as i don’t have to claim warranty… should be ok at least i did not see any buyers complain so far for this seller which i bought from them in the first place.

smart test/long test done no issue. block burn at least 3 passes completed so far and still zero issues.

slightly worried about patrick from sth’s report on the x12, but will use these confidently for now and hope for the best

[Stux]

@mooglestiltzkin

all i could fine was this, quoting at least 250 MB/s

Drives read the inner tracks slower than the outer tracks.

So in reality for a non fragmented read you can probably expect somewhere between 250 to 150MB/s, but as the pool fills and the drives get fragmented that might slow down to 80MB/s.

[mooglestiltzkin]

something like if it exceeds 80% then performance takes a hit? from what i read.

i only expect my storage usage to take somewhere between 30-50% for the next 5 years for the total usable space

[Fleshmauler]

@mooglestiltzkin

something like if it exceeds 80% then performance takes a hit? from what i read.

That would be software sides of ZFS. What Stux is saying is a physical property of the HDD. If you look at the reporting of your HDD that is doing badblocks you can actually see the read/write speeds start fast & then drop depending on where the drive is physically reading/writing from. It’ll prolly look something like this \

[Stux]

@mooglestiltzkin

something like if it exceeds 80% then performance takes a hit? from what i read.

More like 95%

But fragmentation increases rapidly as a drive fills, which is why at 95% (iirc) ZFS switches to a more conservative (and slower) allocation strategy.

And as I said, fragmentation causes slower reads (at least on spinning disks)

So, my advise re 80% is when you hit 80% plan an upgrade and have it completed before you hit 90%.

[etorix]

@Fleshmauler

Also generally expect that writes will be slower than reads.

ZFS magic can actually result in the reverse! Consolidated sequential writes may turn out faster than reading files which have been fragmented over multiple transaction groups.

Generally, don’t worry, don’t expect to “saturate 10 Gb/s” with 4 HDDs and be happy if the NAS performs above 1 Gb/s.

[mooglestiltzkin]

@Fleshmauler

ou look at the reporting of your HDD that is doing badblocks you can actually see the read/write speeds start fast & then drop depending on where the drive is physically reading

u were right

so 160-170 MB/s write

@etorix

Generally, don’t worry, don’t expect to “saturate 10 Gb/s” with 4 HDDs and be happy if the NAS performs above 1 Gb/s.

well i probably dont need 10g (though my switch, transceivers all support it).

was just hoping to see if i could get there as a homelab enthusiast was trying to hit those numbers. feels sad after all the expense and time cant hit that

im basically in the situation techno tim was before, he realized u needed to do extra stuff to get those speeds.

actually with my ts-877 the rear/write 2.5g speeds are a bit wonky.

when i copy in one direction its fine, i get solid 2.5 but when i do it in the opposite direction, it crawls to 1g and sometimes hits 2g but doesnt stay there consistently enough. was hoping a new nas would solve that. so at bare minimum i hope the new nas fixes that (hope the issue isnt desktop is the bottleneck )

How Much Network Speed do you NEED?

[Stux]

Zoom out if you can.

[mooglestiltzkin]

u mean this?

[etorix]

@mooglestiltzkin

was just hoping to see if i could get there as a homelab enthusiast was trying to hit those numbers. feels sad after all the expense and time cant hit that

You will get there… occasionally, when your NAS can serve a request from ARC, or for about 10 s when your desktop will send a file from its internal SDD to the NAS.
To be there consistently, you’d need more drives, more vdevs and/or go for an all-flash pool.

[Fleshmauler]

My badblocks output looked something like this (but I’m still on Dragonfish); excuse my mspaint skills - pretend it is a nice graph if you could:

Anyway - this actually showed that the HDD physically slows down depending on how deep the head has to go to read/write on the platters (wrong terms, but hopefully explained the point).

[Fleshmauler]

@etorix

You will get there… occasionally, when your NAS can serve a request from ARC, or for about 10 s when your desktop will send a file from its internal SDD to the NAS.

Something to also keep in mind is the speed of the system you are read from/writing to. Unless your client can keep up with your server - you ain’t seeing 10g consistently I really should update my ingest system…

[mooglestiltzkin]

thats why for test i used gen4 m.2 ssd on pc to transfer from/to.

Crucial P5 Plus 2 TB Specs

didnt help

ill try with new nas see how it goes.