Sorry for the late response, it’s been one of those weeks. Thanks to everyone for the excellent resources and advice, I’ve taken the time to read what everyone shared and do some more research.
I was aware of the 80%-90% rule, but not the fragmentation cost over time. Was surprised that ZFS had no defragmentation built in, but I can imagine the difficulty and trade offs involved in integrating a defragmentation algorithm into a COW file system. Fascinating to learn about; I’m glad I came to the forums.
Based on the resources Essinghigh and Krill shared, I am considering getting a 1x1 1 tb NVMe Mirror (just two drives) for my block storage needs. Figure it’s better to keep unnecessary writes off the primary storage, even if SSDs don’t suffer the same fragmentation issues as HDDs as Stux mentioned (nice bonus for SSDs that I was not aware of). I imagine this should be good for my block storage needs, but let me know if there’s any reason to go with 2 tb drives or a 2x2 mirror (don’t imagine I’ll need too much capacity here).
Thanks for input Stux. To make sure I understand correctly, the fragmentation concerns discussed here by JGreco don’t apply to SSDs, because data on an SSD is effectively “fragmented” by default (dynamic wear leveling acts in a similar manner as COW, to my knowledge, and the SSD microcontroller spreads contiguous data written across multiple chips to even out wear).
So keeping the pool occupancy rate low (around 10%-20%) is not a concern with a SSD pool, from a fragmentation perspective, and you will see no appreciable performance impact until you hit ~90%, when ZFS changes algorithms to a slower allocation mode to save space.
I didn’t even think to check B550 boards, thanks for bringing this board to my attention! Of course just when I’m also considering adding in some NVMe drives for block storage. Maybe I can break up a x16 HBA into 2 NVMe and 8 SSDs. Otherwise I have some decisions to make and trade offs to consider between having IPMI or more PCIe lanes.
Great to have a nice real world estimate, thanks for that.
Absolutely, going with an SFP+ card and a DAC cable for that reason. The switch is mainly RJ45 2.5GBase-T, but has two 10g SFP+ ports as well.
Didn’t even check Solarflare NICs, thanks for the tip. Looks like they came out around the same time as the X710s, so there’s a decent chance the have ASPM support. Another rabbit hole awaits lol.
I’d be surprised if the 5700x doesn’t support the C6 power state, as my understanding was that all Zen 3 CPUs supported it. Though the 5700x is a bit of an odd one, since it was released much later along in the Zen 3 lifecycle.
Not surprising that the lowest it’s gotten is C3 though, as my understanding is that Zen 3 AMD CPUs go C1-C2-C3-C6. (PowerTOP is an Intel utility, so it’s a bit wonky with AMD CPUs, which follow a different power state system than intel CPUs). Could be your use case, or could be some device in your system not supporting ASPM and thus preventing the deep idle (the CPU can idle down to C3 with non-ASPM compliant PCIe devices, but not C6). That seems to be the annoying thing about achieving a C6 power state, as all it takes is one device not supporting ASPM to keep the processor from reaching C6. Trying my best to avoid that fate in the buying phase, but easier said than done.
Lots more to reply to, but I got to get to bed, so I’ll reply to the rest tomorrow. Thanks so much everyone for the excellent thoughts and input. Really helped me hone in and clarify some concepts.
But realistically, the MC12-LE0 is well suited to 6 SATA and 4 NVMe in the bifurcated x16 slot (Asus HyperM2 and similar) OR to a HBA for many SAS/SATA drives but no NVMe (save for the x1 boot drive). 10 GbE NIC in the x4 slot in either case.
For more than 6 SATA and multiple NVMe drives and a NIC, you may look into refurbished Xeon Scalable/EPYC to have more PCIe lanes to begin with—of course, idle power will be higher. Using a 5900X because you have it may not be the best starting point.
Lack of support for C6 may be on the TrueNAS side, not on the hardware side.
