TrueNAS Idle Power Savings - Zen 3 Ryzen CPU's ECC Ram, and Kill-A-Watt - An Adventure Begins

TrueNAS General
,
1

Just upgraded a build for Truenas. Thought it would interesting to share feedback for a relatively high output low power use server build that is working great. I made the quoted power measurements with a Kill-A-Watt.

My old TS140 build finally died and I was slowly adding drives to it to get additional storage. Before I knew it I had 12 4 TB drives in two vdevs, using ghetto style PCI-E sata cards. Never had a problem but the new build I decided to stop being dumb with this.

sata 1
sata 1757×733 325 KB

Got a B550 + 5600x + 32gb ECC udimms+ 550w bronze PSU to start, and got a LSI 9400-16i card to manage the (12) 4tb drives.

I measured the power usage incrementally to get a rough idea of each component measured before and after upgrade. Idle was measured 10 minutes after boot to let TrueNas Dragonscale simmer down after start:

Start System
B550 + 5600x + 32gb + 9400-16i + 12x4tb WD Red + 64gb SSD (OS) + 512gb M2 (non-nvme drive) + old TS140 bronze (?) 550w PSU + 4 120mm system fans + wraith CPU cooler + headless (no gpu):

Idle Power 105w-110w (!)

Same as above but swapped for Superflower Titanium 750w PSU:

Idle Power 90-95w

(I was impressed with this one change - the other PSU was 12+ years old)

Same as above but all rust spinner (12 x 4tb reds) drives disconnected / unpowered. LSI 9400 HBA still plugged in however:

Idle Power 75-80w (I thought 12 drives would use more that 15w spinning - I checked this 3 times over 30 minutes to make sure)

Same as above but all rust spinner (12 x 4tb reds) drives disconnected AND LSI 9400 HBA removed:

Idle Power 65-70w (9400-16i uses around 10w constantly which is what the specs say)

Same as above but all rust spinner (12 x 4tb reds) drives disconnected AND LSI 9400 HBA disconnected in AND Eco-45w enabled in the MSI bios:

Idle Power 50-55w (I consider this the bottom floor of possibility with 5600x+b550 combo - pretty crappy tbh)

Personally I thought a relatively mid tier CPU 5600x to use 50w “no-matter-what” in a headless system to be pretty crappy. Somebody in the know told me that it was due to their chiplet design and their IO CPU chip power demand being impossible to lower any further. I didnt want to lose ECC ram funtionality so in researching, I learned that AMD makes (rare to find) Ryzen Pro 5650G chips that use the monolithic chip technology that get pretty low draw on idle. Ordered from Aliexpress (non-existant in America) for around $110 delivered for a new one that wasnt vendor locked and popped it in (it took a little doing to get the BIOS to use it right - didn’t work with UEFI and needed QSM bios mode) and fired everything up.

5650g

Finished System:
B550 + 5650G Pro + 32gb + 9400-16i + 12x4tb WD Red + 64gb SSD (OS) + 512gb M2 (non-nvme drive) + Titanium 750w PSU + 4 120mm system fans + wraith CPU cooler + integrated GPU + 35w BIOS mode

Idle Power 60-65w (!)

I was pretty impressed and if anybody is going to build a Ryzen based AMD system, I can’t really see a better product than either the 5650g PRO or the 5750G PRO (8 cores). You get an impressive 24 lanes and a pretty decent APU (Vega 7/8) for VM’s and basic UI / Bios access. 60-65w idle for my 12 drive monstrosity that runs a decent suite of Truenas apps (Plex, Qbittorrent, Photoprism, Omada Controller, NGINX PM, Vaultwarden, Webdav, Adguard Home DNS) with ECC ram is fantastic.

Note: I don’t really care about the money per se, I just couldnt justify the 5600x housefire CPU being in a NAS anymore. Maybe people didn’t know about the Zen R5/R7 Pro 5650G being an awesome chip for a home NAS (I sure didn’t). I sold the 5600x for like 90$ so I only paid like +$20 for the 5650G Pro chip all in all.

Thanks for reading my blog.

11 Likes
2

Additional savings are possible with Curve Optimizer. But this is considered “overclocking”.

image
image1325×399 72.9 KB

E.g.
Simple Precision Boost Overdrive (Undervolting) Guide for ASUS B650E-I : r/sffpc (reddit.com)

For my system, going from stock voltage to PBO -30 gained 6.8% in Cinebench 24 peak performance and generally speaking will resulting lower temperatures and higher performance under normal operation.

3

This is helpful. However, because this R5 5650Pro CPU is not used often in the normie market (it was apparently developed specifically for Lenovo Thinkcentre mini pcs) the BIOS was much more restrictive and didn’t let me play around with a lot of the overclocking settings. It did have a MB Power Management setting, and I changed to the 35w (same TDP profile as the 5650GE version of this chip) one because I was chasing lowest power idle settings as my NAS is on 24/7.

I guess I was convinced that only using Intel atom denverton based Truenas systems for low idle power was possible with basic onboard video and ECC ram support. The availability of this chip was news to me, and I can attest it works well in Truenas and the kernel knows it, ranks the memory as ECC and the Vega 7/8 works on both Plex AMD GPU passthrough and a basic Windows 11 VM I was messing around with.

Granted the famous A2sdi board have much better vanilla networing stats out of the box, and IMPI connectivity. But if you are willing to trade that off for much better CPU processing, much better APU, and pop in a 10gb SFP $20 Mellanox 3x card its for much less money, its a fantastic system for Truenas.

4

This is really helpful. I’ve been looking at the AMD Ryzen 5000 Pro series because those (Pro) are the only (5000 series) ones that (officially) support ECC (as you have stated). But wondering about and considering the power draw. Do I really need that much CPU power for TrueNAS SCALE?

iX Systems are still selling the Mini X+ with the 8 core Atom C3758 CPU. Your 5650G is 350% faster. But the C3758 has a TDP of only 25w.

Calculating the power cost. 65w x 24h x 30d = 46.8 KwH x $0.182 (my cost per KwH) = $8.5 / month. At 100w that’s $13.10 per month.

5

I am glad you enjoyed it. The IX systems c3758, for numerous reasons, will be better for a “true” nas as a standalone system. Also, since these Atoms are embedded into the motherboards they will integrate with all the other nice must haves for server use (teaming NICs, IPMI, Aspeed sata controllers, etc ). I believe they have low idle power draw because numerous old school Truenas posters have said as much and I believe them. The only objective measurement I saw was from the servethehome here.

The problem is they are hella expensive even if you build yourself. The lowest c3758 supermicro boards are like $600+ on ebay without memory. And thats from a grey market ebay seller who ain’t calling you back if the thing fails. Normie motherboards like my b550 msi pro-vdh, while cringe to enterprise guys, will actually replace my board via newegg / amazon. Zoomer420 dude on ebay selling this will most likely not. YMMV.

For comparison, the build I have is:

$130 5650g
$70 msi pro vdh mb
$25 mellanox 10g sfp card
$100 hba 9400-16i ( you dont need this, i just had 14 drives to connect)

So for $325 vs $600+ you get better graphics card (atom is not quick sync), better network, worse motherboard (pro vdh is well reviewed [here] mind you (https://www.youtube.com/watch?v=wuPH9pCCK-E)) but functional ECC udimms. The supermicro allow r dimms for slightly cheaper ECC r dimms and potentially higher total memory due to density. Im happy with 32 gigs running a suite of apps no problem and one SteamOS VM. I have my CPU / mb running at 35w eco mode. Not sure if eco mode 35w = 25w tdp but I am certain the Atom will sip power much better 100%.

My back of napkin math with my crude testing setup shows that when fully idle the b550+r5650g+32gb u dimm ram + system fans + 64gb SSD + 512 M2 software drive uses around 25w-30w idle. This is with the following K3s running (nginx, vaultwarden, qbittorrent (idle), omada, photoprism, plex, adguard as DNS and Webdav)

1 Like
6

Curious, what do you use the SteamOS VM for? Playing games?

7

You’re going to laugh Stux, but my kids like crappy games like Overcooked 2 and Castle Crashers. Often I am using my beefy as hell main gaming / home theater PC for selfish reasons, so I use that to stream these crappy games to another TV’s steamlink.

I followed this guide here to do it but I failed miserably repeatedly in the TruenNas environment. https://www.youtube.com/watch?v=NJ2jvTgp4AA. As you know from my stupid questions on the dockge thread, I’m like a plumber doing brain surgery with unix. My neighbor down the street did everything himself and while new to Truenas itself, hes a paid tech bro sysadmin who uses hypervisors for his real job all the time and figured it out. Im not too proud to say he did all the hard work. The Vega 7 chip in my APU plays those games just fine.

1 Like
8

So, the Vega 7 is passed into the VM and then the VM is streaming via steamlink?

Pretty neat.

I think I’d like to play with that too.

My kids play overcooked on the xbox :wink:

9

Its great. I was more or less playing around VM’s on Truenas and it was beyond my abilities. I never got past the “rebuilding” the Steam OS linux distro for my VM instance that the video shows. Probably really easy for a guy like you. If I had to do it again, I would just install a barebone windows 10 LTSC VM (I can do this) and then install Steam on it as normal. I just thought the Steam OS was pretty cool and low overhead for just silly games. Lucky I have a smart tech bro neighbor who lives, eats and breathes terminal commands, VM’s and hypervisors so he made it childs play (literally). He was impressed with Truenas as he had never used it before, so maybe I’ve made a convert down the road :smile:

PS: Normally I let them play on the big gaming rig, but a lot of times I want to use it when they play. Overcooked is a great game for kids; I won’t corrupt them like my upbringin playing Resident Evil in elementary school :frowning:

1 Like
10

I am planning to replace my 5700x with a ryzen pro for a lower idle power draw, just like you did, and repurpose it for a truenas system. The thing is that my msi b450m board marks no ECC support on its spec page, unlike am4 motherboards from other brands. Could you confirm that your ecc udimms are actually working in ecc mode on that msi mb?

11

Depends where you live of course, translate that to the high in some places California price per KWH and the $13.10 could easily be $50/month for your pretty low cost system. Translate that to my cost due to solar (I still pay small fees to electric company) and the cost spread over the lifetime of the system, it would cost me about $1.50/month so I obviously wouldn’t care!

1 Like
12

Hi Molamola. Late response. I have been travelling a lot. To be clear, I have the b550-pro vdh motherboard and not the b450 you are using. I can 100% confirm it supports ECC mode and my truenas dragonscale install says this, as well as verified in shell commands (dmidecode etc.)

I have no idea if your MB supports it. My MB on the website confirms ECC support for “x” and “pro” ryzen CPUs. If the website says it does like this, then you should be good to go. I took a screen from the b45m-pro max mb from MSI. No idea what exact model you have.

Capture
Capture844×763 106 KB

Please refer to the MB you have manual similar to mine.

13

I agree its peanuts for a lot of American home users. Myself included. I just generally like things to run efficiently in life and its not alll about money. For example for a DIY 24/7 server I will ALWAYS buy a titanium branded reputable PSU for the whopping +$30 upcharge. WIll I make the money back? I dont care, I just know that they are built so well at that scale that the peace of mind for the +$30 and warranties is worth every penny. The peace of mind is worth it over cheap chines bronze mystery meat oem psu’s in ebay server gear.

Not a perfect analogy, I re-insulated my entire 100 year old American house and replaced all windows with new german windows myself. Will I make back the heating costs where nat gas in america is practicallly a free waste product? Never. Do I prefer a draft free, high quallty communicating HVAC system keeping my house perfectly at a set temp with no cold rooms? Absolutely.

/end blog post

1 Like
14

I love the journey and the alert to the Ryzen -G and -GE series.

I have been through a similar journey as you re: reducing heat gain / loss in an old girl. Our house is now 150 years old and still features a bunch of wavy glass windows. We lowered heating needs to12 BTU/ft^2 on a design day (i.e. 1/8th the pre-renovation).

Having played around with Emporia power measurement, my next task is reducing the power needs of my ‘datacenter/network’ as it is the largest-power consumer in these months at about 250W continuously. At $0.33/kWh, I have some incentive to do so.