DELL T440 deployment, suggestions, critique, warnings

TrueNAS General
1

I was hesitant to drop yet another deployment discussion, but I have done my research and I have the outline for what I intend to do. However, it never hurts to ask for a second opinion before dumping time, money and energy, last but not least data into a (potentially) flawed project.

Hardware: DELL T440, Xeon Silver 10c20t, 64GB RAM, up to 4 NVMes on ASUS hyper M.2 x16 gen 4 (4x4x bifurcation NVMe “riser”), up to 8xLFF 6GB SAS 12gbit HDDs on a HBA mode PERC H730p (with 2 more as cold spares for 10 total). All of that is already available/purchased. Cuurrently looking at NVMe drives to get.

I want this to replace and add some functionality that my current setup which is on an aging DELL T1650 just can’t handle.

Use: Storage, Apps (Plex, Jellyfin, SFTP, etc.), VMs (Work and Testing), VPN (WG) and whatever else crops up.

HDD Storage: I intend to have a RAIDZ2 pool that will be hosting all my different bulk storage needs. At first I intended to have it 8x6TB wide, but since I don’t need more than 10TB atm, I might stick to 4x6TB wide and have the remaining drives as cold spares, to save power and noise. That machine will be living at my house, so swapping in a drive should occur in a matter of hours. I might setup a hot spare depending on opinions and experience shared below. I will have the drives spin down to cut down on power and noise. To that end, there will be no apps, VMs or system logs rolling on the HDD pool.
Questions: adding drives to an existing pool? Hot spare?

SSD Storage: 1 SSD will be dedicated to my boot-pool. It will be cheap and nasty, just because I hate the fact it will be dedicated to a trickling log and lost for any other use. I don’t intend any redundancy or backup for that. My other deployment already torched one SSD in that role (it was old and probably on its last legs anyways) and reinstallation + restore parameters was a fairly easy and straightforward job. I will investigate having the boot-pool on the IDSDM dual (mirrored) micro SD daughterboard (testing ATM with 2x mirrored high endurance U3 micro sd cards).
The rest of the SSD storage will be set up in a pool, probably 2 wide striped or 3 wide RAIDZ1 for my Apps, VM system partitions and “always on” storage (limited temporary storage for stuff I don’t need to keep/will commit to the main storage later - as not to spin up all my dives to copy a temporary file). The SSD storage will be automatically backed up on the HDD pool coordinated with my other replication tasks, as not to have multiple spin ups of the drives.
Questions: anyone with experience on IDSDM? Good M.2 SSDs that can make full use of the 4x PCIe v3? Anyone with the Asus Hyper M.2 x16 - I will be looking at cooling performance, since the airflow of the T440 is reversed for the PCIe slots. Might have better results with the board fan off.

Thanks to those that took the time to read the wall of text.

2

Well, first fail of the plan has happened. Classic RTFM moment. The Asus Hyper M.2 or ANY x16 card CANNOT be used in the Dell T440, if it has only one CPU! The only x16 slot is connected to CPU #2.

Forging ahead, I have ordered a Dell BOSS. Two SSDs are less than 4, but more than zero.

Feeling like talking to myself, but hopefully someone will learn from my mistakes.

2 Likes
3

Felt like dropping an update on the deployment. With the T440 coming at retiring age, I expect more and more to become available at bargain price. And since Truenas is a great option to take advantage of such server, here goes my (almost) half-year of ownership report.

The Good: iDRAC, quiet-ish cooling, good expansion capabilities*, dual PSUs, looks nice
The Bad: power draw, poor PCIe quantity

My setup at time of writing:

Xeon Silver 4114(10c20t)- adequate for my use case. I have a second 4114 + cooler if I decide I need more compute or the PCIE x16 slot, I also have a Xeon Gold 6138 (20c40t) if I need more single socket performance.

64GB (4x16GB) 2666MHZ,

8x 6GB SAS Toshiba (NetApp) 3,5" HDDS in raidz2, no matter what I try I am unable to get those HDDs to spin down. Just recently I was finally able to have them enter some sort of standby with sg_start --pc=3 /dev/sdX so I know the drives and the HBA will allow standby. This experiment however lead to Truenas flagging one of my drives as faulted.

DELL HBA330 the server came with a PERC 730, which I had in HBA mode. I replaced it to see if a proper HBA will solve the inability of the system to get the HDDS to Standby. Also it had the battery ripped off during shipping and so it was making my iDRAC all kinds of angry.

1x512GB NVMe SSD on PCIe x4 riser. I have a DELL BOSS PCIe x8 card for 2x NVMe SSDs. Cannot use it, because of no free x8 PCIe slot.

2x128GB high cycle SD cards in mirror on the DELL SD riser for boot-pool. Truenas is angry at me for those, calling them USB device,

Mellanox ConnectX-3 QSFP+ with QTS adapter and 10Gbps SR SFP+ I first went fof a quad port 10Gb SFP+ Silicom (Intel x520) NIC, but promptly changed it because it was drawing 15W+ while idling. Eats my second PCIe x8 slot. Might get another ConnectX-3 that can go in a x4 slot so I can use the slot for something else.

iDRAC Enterprise I had to buy a license from ebay to upgrade to the Enterprise level. Best money spent on this server, except the server itself.

Google Coral TPU M.2 this is on another PCIe x4 riser and it is not getting recognized by the server. Sadface.

The currrent setup is a bit different from what I initially intended, due to peculiarities of the server. Neverteless, the only area where this setup falls short is GPU. I simply don’t have the need for one and the power draw is already at a level that has me halfway convinced to shut down the server. Power draw is at the 140-150W mark with every service and a few VMs up. When I did the experiment of forcing the HDDs to Standby, power draw dropped to 87W, which is not insignificant, but still almost half of the normal.

Bottom line: if you can live with the number of PCIe slots and no high power GPU, this machine offers good compute and RAM possibilities with excellent IPMI, redundant PSUs with decent efficiency, 8x3,5" hot-swap bays (or 16x2,5"). Just keep in mind the electricity bills and that you need a second CPU for the PCIe x16 slot.