Option C: Use NVMe Retimers
Example: Astera Labs PT4161LR
Option D: Direct U.2 Card
Examples: 10Gtek S94N416
Which out of these options are the best options in terms of performance and as low as possible latency and which one you won’t recommend at all and why.
You need to post your detailed hardware, OS, specs, etc. I suggest putting it in your signature. Posted a code snip to get you started. It hides the signature, click to show the featured.
[details=“Click me to see more”]
TEXT IN SIGNATURE
[/details]
Looks like you have this below for expansion slots on MB. You will have to figure out what you have taken up already. Tri-Mode adapters are probably best since proven but you have PCIe 3.0 slots. You need to check for compatibility and your speed / performance may be limited. I don’t know if anything else on your list will work for NVMe.
PCIe
1 PCIe 3.0 x16,
1 PCIe 3.0 x16 (x16 or x8 ),
1 PCIe 3.0 x8 (x0 or x8 ),
1 PCIe 3.0 x8,
1 PCIe 3.0 x4 (in x8 slot)
M.2
M.2 Interface: PCIe 3.0 x4; SATA
Form Factor: 2280, 22110
Key: M-Key
…to be a nuisance. Tri-Mode runs NVMe drives through a SCSI bus, negating everything NVMe was designed, and forces to use U.3 backplanes which are incompatible with U.2 drives.
Incidentally, the OP’s 3rd gen. Xeon Scalable is PCIe 4.0. (And using a P4801X for boot is just a waste of resources. I boot from OPtane myself, but these are 16 GB M10.)
So, of the options:
A (Tri-Mode) No!
B (switch) Only if you need to… though @NickF1227 had some tests on the old forum with 3.0 switches suggesting that these may actually help by offloading switching work from the CPU.
C (retimer) Retimers are not switch. Add only if you need these to ensure signal integrity—at which point you might as well go straight for a switch.
D Yes. Simple adapters and CPU bifurcation should be the first thing to try. And if these work, stay with that.
Requires the motherboard PCIe x16 slot to support splitting to x4 + x4 + x4 + x4. If the PCIe x16 slot only supports splitting to x8 + x4 + x4, only three disks can be recognized. If the PCIe x16 slot only supports splitting to x8 + x8, only two disks will be recognized.
Hmm. That is understood but then how these OEM brands like Dell, HP, Lenovo uses those shit and their performance is not affected because they use a backplane?
Yes!
I had one lying so decided to use it in this build
How would i understand this mate?
Just like the Chelsio NICs? They have CR and SO-CR.
So, Switch Adapters are more better than Retimers?
So, this is the best option for using U.2 drives, yeah?
BTW, @etorix for my another build, i plan to add a couple of drives, and i need an HBA Card, I’m getting a great deal on LSI 9400-16i. Would it be fine to use it for the HDD/SATA SSD? Or you have some better recommendation?
Then, if it’s x4+x4+x4+x4, you don’t need PLX switch or HBAs, just PCIe expansion cards. You still might need a retimer, but that’s another story.
@Fastline I see you on these forums and asking a lot of questions to build what sounds like various systems. I’m just curious if you are building these systems for yourself or for friends or maybe even for customers. I’m not judging, I’m just curious. If they were all for you, I’d think you have at least 3 to 4 NAS machines by now and probably 25 PB of storage.
4x4 bifurcation is pretty much a standard feature on Xeon Scalable server boards.
I suppose it is a mix of inertia about selling a 30-year-old design and Broadcom being very persuasive… As shown here, performance is barely affected… until Tri-Mode hits the wall and stops there while “direct attach” reaches further.
Obviously this build is not about saving a few dollars, or 10 W idle from the boot drive… Still you could get any small regular SSD for boot here and get back this P4801X for use as SLOG on another (HDD-based) NAS.
Simple case: The drives ask for more lanes than the CPU can supply. (Not yet the case here.)
Complex case: You do some @NickF1227-style testing and find out that your NAS is held back by the CPU handling the switching.
Noting that Chelsio’s naming is somewhat reversed: CR are the highest performers while SO-CR offloads some tasks to the server.
Retimers do not switch, but switches retime.
These are the last of the “classic” HBAs before the Tri-Mode 9500 and the rewrite of the driver stack, so should be fine.
Yes, i agree. I ask a lot of questions cause i’m curious and when i cannot find the right answers, i have to rely on you guys
I do have several hardwares. Some from old systems and some new. Some NAS have retired now and its time to upgrade them!
Actually, a couple of. One for the business use, one at home and one for my brother
Technically, i own those NASes cause two of them are mine and as brother lives with me and he needed one for his work, i decided to build one for him as well.
Maybe when i’m expert like you guys, i can build for customers as well
and probably 25 PB of storage.