TrueNAS Idle Power Savings - Zen 3 Ryzen CPU's ECC Ram, and Kill-A-Watt - An Adventure Begins

TrueNAS General
,
1

Just upgraded a build for Truenas. Thought it would interesting to share feedback for a relatively high output low power use server build that is working great. I made the quoted power measurements with a Kill-A-Watt.

My old TS140 build finally died and I was slowly adding drives to it to get additional storage. Before I knew it I had 12 4 TB drives in two vdevs, using ghetto style PCI-E sata cards. Never had a problem but the new build I decided to stop being dumb with this.

sata 1
sata 1757Ă—733 325 KB

Got a B550 + 5600x + 32gb ECC udimms+ 550w bronze PSU to start, and got a LSI 9400-16i card to manage the (12) 4tb drives.

I measured the power usage incrementally to get a rough idea of each component measured before and after upgrade. Idle was measured 10 minutes after boot to let TrueNas Dragonscale simmer down after start:

Start System
B550 + 5600x + 32gb + 9400-16i + 12x4tb WD Red + 64gb SSD (OS) + 512gb M2 (non-nvme drive) + old TS140 bronze (?) 550w PSU + 4 120mm system fans + wraith CPU cooler + headless (no gpu):

Idle Power 105w-110w (!)

Same as above but swapped for Superflower Titanium 750w PSU:

Idle Power 90-95w

(I was impressed with this one change - the other PSU was 12+ years old)

Same as above but all rust spinner (12 x 4tb reds) drives disconnected / unpowered. LSI 9400 HBA still plugged in however:

Idle Power 75-80w (I thought 12 drives would use more that 15w spinning - I checked this 3 times over 30 minutes to make sure)

Same as above but all rust spinner (12 x 4tb reds) drives disconnected AND LSI 9400 HBA removed:

Idle Power 65-70w (9400-16i uses around 10w constantly which is what the specs say)

Same as above but all rust spinner (12 x 4tb reds) drives disconnected AND LSI 9400 HBA disconnected in AND Eco-45w enabled in the MSI bios:

Idle Power 50-55w (I consider this the bottom floor of possibility with 5600x+b550 combo - pretty crappy tbh)

Personally I thought a relatively mid tier CPU 5600x to use 50w “no-matter-what” in a headless system to be pretty crappy. Somebody in the know told me that it was due to their chiplet design and their IO CPU chip power demand being impossible to lower any further. I didnt want to lose ECC ram funtionality so in researching, I learned that AMD makes (rare to find) Ryzen Pro 5650G chips that use the monolithic chip technology that get pretty low draw on idle. Ordered from Aliexpress (non-existant in America) for around $110 delivered for a new one that wasnt vendor locked and popped it in (it took a little doing to get the BIOS to use it right - didn’t work with UEFI and needed QSM bios mode) and fired everything up.

5650g

Finished System:
B550 + 5650G Pro + 32gb + 9400-16i + 12x4tb WD Red + 64gb SSD (OS) + 512gb M2 (non-nvme drive) + Titanium 750w PSU + 4 120mm system fans + wraith CPU cooler + integrated GPU + 35w BIOS mode

Idle Power 60-65w (!)

I was pretty impressed and if anybody is going to build a Ryzen based AMD system, I can’t really see a better product than either the 5650g PRO or the 5750G PRO (8 cores). You get an impressive 24 lanes and a pretty decent APU (Vega 7/8) for VM’s and basic UI / Bios access. 60-65w idle for my 12 drive monstrosity that runs a decent suite of Truenas apps (Plex, Qbittorrent, Photoprism, Omada Controller, NGINX PM, Vaultwarden, Webdav, Adguard Home DNS) with ECC ram is fantastic.

Note: I don’t really care about the money per se, I just couldnt justify the 5600x housefire CPU being in a NAS anymore. Maybe people didn’t know about the Zen R5/R7 Pro 5650G being an awesome chip for a home NAS (I sure didn’t). I sold the 5600x for like 90$ so I only paid like +$20 for the 5650G Pro chip all in all.

Thanks for reading my blog.

8 Likes
2

Additional savings are possible with Curve Optimizer. But this is considered “overclocking”.

image
image1325Ă—399 72.9 KB

E.g.
Simple Precision Boost Overdrive (Undervolting) Guide for ASUS B650E-I : r/sffpc (reddit.com)

For my system, going from stock voltage to PBO -30 gained 6.8% in Cinebench 24 peak performance and generally speaking will resulting lower temperatures and higher performance under normal operation.

3

This is helpful. However, because this R5 5650Pro CPU is not used often in the normie market (it was apparently developed specifically for Lenovo Thinkcentre mini pcs) the BIOS was much more restrictive and didn’t let me play around with a lot of the overclocking settings. It did have a MB Power Management setting, and I changed to the 35w (same TDP profile as the 5650GE version of this chip) one because I was chasing lowest power idle settings as my NAS is on 24/7.

I guess I was convinced that only using Intel atom denverton based Truenas systems for low idle power was possible with basic onboard video and ECC ram support. The availability of this chip was news to me, and I can attest it works well in Truenas and the kernel knows it, ranks the memory as ECC and the Vega 7/8 works on both Plex AMD GPU passthrough and a basic Windows 11 VM I was messing around with.

Granted the famous A2sdi board have much better vanilla networing stats out of the box, and IMPI connectivity. But if you are willing to trade that off for much better CPU processing, much better APU, and pop in a 10gb SFP $20 Mellanox 3x card its for much less money, its a fantastic system for Truenas.

4

This is really helpful. I’ve been looking at the AMD Ryzen 5000 Pro series because those (Pro) are the only (5000 series) ones that (officially) support ECC (as you have stated). But wondering about and considering the power draw. Do I really need that much CPU power for TrueNAS SCALE?

iX Systems are still selling the Mini X+ with the 8 core Atom C3758 CPU. Your 5650G is 350% faster. But the C3758 has a TDP of only 25w.

Calculating the power cost. 65w x 24h x 30d = 46.8 KwH x $0.182 (my cost per KwH) = $8.5 / month. At 100w that’s $13.10 per month.

5

I am glad you enjoyed it. The IX systems c3758, for numerous reasons, will be better for a “true” nas as a standalone system. Also, since these Atoms are embedded into the motherboards they will integrate with all the other nice must haves for server use (teaming NICs, IPMI, Aspeed sata controllers, etc ). I believe they have low idle power draw because numerous old school Truenas posters have said as much and I believe them. The only objective measurement I saw was from the servethehome here.

The problem is they are hella expensive even if you build yourself. The lowest c3758 supermicro boards are like $600+ on ebay without memory. And thats from a grey market ebay seller who ain’t calling you back if the thing fails. Normie motherboards like my b550 msi pro-vdh, while cringe to enterprise guys, will actually replace my board via newegg / amazon. Zoomer420 dude on ebay selling this will most likely not. YMMV.

For comparison, the build I have is:

$130 5650g
$70 msi pro vdh mb
$25 mellanox 10g sfp card
$100 hba 9400-16i ( you dont need this, i just had 14 drives to connect)

So for $325 vs $600+ you get better graphics card (atom is not quick sync), better network, worse motherboard (pro vdh is well reviewed [here] mind you (https://www.youtube.com/watch?v=wuPH9pCCK-E)) but functional ECC udimms. The supermicro allow r dimms for slightly cheaper ECC r dimms and potentially higher total memory due to density. Im happy with 32 gigs running a suite of apps no problem and one SteamOS VM. I have my CPU / mb running at 35w eco mode. Not sure if eco mode 35w = 25w tdp but I am certain the Atom will sip power much better 100%.

My back of napkin math with my crude testing setup shows that when fully idle the b550+r5650g+32gb u dimm ram + system fans + 64gb SSD + 512 M2 software drive uses around 25w-30w idle. This is with the following K3s running (nginx, vaultwarden, qbittorrent (idle), omada, photoprism, plex, adguard as DNS and Webdav)

6

Curious, what do you use the SteamOS VM for? Playing games?

7

You’re going to laugh Stux, but my kids like crappy games like Overcooked 2 and Castle Crashers. Often I am using my beefy as hell main gaming / home theater PC for selfish reasons, so I use that to stream these crappy games to another TV’s steamlink.

I followed this guide here to do it but I failed miserably repeatedly in the TruenNas environment. https://www.youtube.com/watch?v=NJ2jvTgp4AA. As you know from my retard questions on the dockge thread, I’m like a plumber doing brain surgery with unix. My neighbor down the street did everything himself and while new to Truenas itself, hes a paid tech bro sysadmin who uses hypervisors for his real job all the time and figured it out. Im not too proud to say he did all the hard work. The Vega 7 chip in my APU plays those games just fine.

1 Like
8

So, the Vega 7 is passed into the VM and then the VM is streaming via steamlink?

Pretty neat.

I think I’d like to play with that too.

My kids play overcooked on the xbox :wink:

9

Its great. I was more or less playing around VM’s on Truenas and it was beyond my abilities. I never got past the “rebuilding” the Steam OS linux distro for my VM instance that the video shows. Probably really easy for a guy like you. If I had to do it again, I would just install a barebone windows 10 LTSC VM (I can do this) and then install Steam on it as normal. I just thought the Steam OS was pretty cool and low overhead for just silly games. Lucky I have a smart tech bro neighbor who lives, eats and breathes terminal commands, VM’s and hypervisors so he made it childs play (literally). He was impressed with Truenas as he had never used it before, so maybe I’ve made a convert down the road :smile:

PS: Normally I let them play on the big gaming rig, but a lot of times I want to use it when they play. Overcooked is a great game for kids; I won’t corrupt them like my upbringin playing Resident Evil in elementary school :frowning:

1 Like