If you want speed and low latency, you forget about HDDs.
Use a single NVMe in a PCI slot, or, if you are still not happy, buy 2 or even 4 in the same card.
( but, if you use striped M.2, provide a proper backup too)
Right… except I already have a box of 1.2TB 15K SAS drives that are fast enough for VM virtual disks, so… I’m good. Thanks!
Well, its up to you!
How about:
- Noise
- electricity bills
- heat/cooling
- room you need for all those drives
- and the expected constant rate of failing drives. (1.2TB SAS drives are really old)
So, I hope, you have a nice backup available.
I tried running TrueNAS on old Dell servers and had a devil of a time getting them to ignore the internal RAID controllers, which cannot be run in IT mode. I put an LSI HBA in there and hooked up the internal drive cages, but then the onboard RAID card errors and stops the system from booting.
I eventually settled on an old HP DL380 (see system specs in my sig) which lets me disable the onboard HBA.
You mention putting a 10GbE NIC in: make sure the mainboard supports a high enough PCIe version, and that you have PCIe slots with enough lanes available to make it worthwhile. Same goes for your HBA: there’s no point having a 10GbE network if you can’t read/write data fast enough from your storage to use it.
That was interesting info. I would like to read a guide for buying used HP servers…
…and unless you don’t use compression. For one, zstd-9 easily eats up almost 100% of my i5-12400 during windows backup&restore. And E5v4 xeons have comparable performance.
Honestly, I went a 1u rack before with only a lowly dual-core i3 (not even the higher power Xeon) and I found the noise was still intolerable even when idling, but particularly atrocious when it is booting up and the fans are at full throttle. The pitch of the noise is also very high pitch in tone, making the noise also extremely annoying to hear all day.
My replacement server is now a DIY standard ATX form factor Supermicro/Xeon Silver in a full tower with many 140mm fans and it provides equivalent airflow at fractions of the noise level (quiet enough to run in the bedroom when idle).
Since then, I swore to myself to never buy enterprise rack servers again unless I’m feeling masochistic and want to torture myself.
One other plus point for used HP servers (imho): it’s not hard to find license keys for the Advanced iLO functionality online ![]()
You’ll get IP-KVM and remote media functionality with that, but ensure it’s a minimum of iLO3, as iLO2 (which is on my G6) only supports IP-KVM on ancient versions of Internet Explorer with Java support. Later iLO versions use HTML5. I keep a Windows 95 VM just for managing this box when I have to ![]()
Tangent: as servers get smaller/more densely packed, the fans get smaller, faster, higher pitched and louder. We have a bunch GPU compute nodes in one of our datacentres now that make so much noise (>85dB) the room has had to become a Mandatory Hearing Protection Zone. No one is allowed to work in there without ear defenders, and noise levels have had to become one of the marking criteria for purchasing/tenders ![]()
Yep, those small Sunon/Delta 20mm screamers are little tornadoes spinning at 15-18k RPM.
The servers arrived today so I unboxed one and had a look.
Seems this was a serious storage machine in its last life. It’s got 12×3.5 bays in the front, 2 in the back and 4 on the internal tray. The boot drive is an NVME SSD on a BOSS card. It’s got a HBA330 in the mini mono form factor so I don’t have to worry about flashing it nor consuming a PCI slot. The network daughter board has 2×10GbE RJ45 and 2×1Gbe so I won’t need to mess with fiber and SFP modules.
No idea on the noise yet, I’ll fire it up once I’ve got it racked. It’s got 6 of the high performance fans (60mm Nidec Ultraflow), up to 19500 rpm but I should be able to run them at far less than that.
I also read through the manual. I can run the PSUs in a host spare configuration with one in a sleep state at lower utilisation.
Just an FYI, my x10 Supermicro with E5-2898v3 runs over the course of a day an average of 100W, so, less than you said. That would be everything, all disks, anything attached to it since it all goes to the same UPS. That includes a few handbrake encodes with no gpu per day lately, converting a bunch of BluRay.
Goes from around 80W much of the day with usage to 140W with Handbrake and other normal loads.
Indeed, that does look ingenious. Exploiting a glaring weakness in human sense of hearing.
The challege is, of course, the metallurgy required to make those fan blades that are strong enough to spin at that insane rate for years. You’re just about looking at the forefront of aviation turbofan engines industry.
![]()
The problem with that plan is, dogs can hear 30KHz, and humans will likely be irritated by the dogs’ reactions (barking?).
Yeah, the datacenter’s bloodhounds won’t be happy. At least crocodiles in the moat will stay calm.
Joking apart, I suspect that high sound pressure levels can damage hearing regardless of the frequency.
I have 3 DL380 Gen9s, an HP 5820X switch and a UPS in a Noren quiet rack with forced cooling, Very, very quiet.
I have had 2 failed SAS drives in the last 10 years, and I have spares. I wouldn’t call that “constant” by any stretch. In any case, they are cheap these days thanks to NVMe’s impact to the HDD market.
I export my VMs weekly to the big CIFS share, then copy to iDrive e2 ($50/year). So I have 2 copies of everything.
Everything costs something. Go big or go home.
Well, I’ve just run into my first noise issue. There’s a minimum fan speed that can’t be lowered in iDRAC. The fans idling at 33% PWM is very audible. I came across a suggestion it’s based on ambient temperature on boot.
It used to be possible to set fan speeds with IPMI but that was removed.
Can’t say we didn’t warn you ![]()
The thermal design of a server is quite different to a PC. They rely more on pressure (given the tight spaces involved) than flow, although flow is still required. Given the geometry constraints they need small diameter fans (compared to 120/140mm units common in PC cases) and to generate that pressure they need high speeds. This gives both the fan tip noise as well as the bulk roar of air moving through small spaces.
Unfortunately you can’t shut the things down because the stuff inside will cook, and some of those thing run toasty already. Not sure about dell, but the disk controller on my DL360 and 380 runs at over 75 Celcius at minimum (It has a critical at 130C so it has lots of room to move, but still requires significant amounts of air to move over it). Same for the voltage regulator modules knocking the 12V supply down to 1V for the CPUs. Massive amounts of heat and they’re usually over 60C in my units. The Dell’s are unlikely to be vastly different.
If you look at an HBA for example, many of the LSI/Broadcom units will state minimum flow rates like 200 linear feet per minute because they’re expected to be cooled for reliability.
You may have some ability to control fan speeds, but don’t get too cocky in believing you know better than the people who programmed it because you may find yourself with an unreliable server or its lifespan won’t justify what you spent on it.
You may, however, be able to find alternative fans with better performance to noise profiles than the stock ones. They’ll be pricey though and probably not give you the answer you hope for.
