I’ve heard this in the past and have also heard it from our consultants. In practice, all of our current TrueNAS servers run this BOSS card and we have not had any problems. The problem with what you suggest here is that the RAID is hard coded and can’t be separated. The RAID controller in this card is doing its own verification so I don’t see much advantage in giving up two of my front side drives just so that the boot drives can be mirrored via TrueNAS.
It’s not a ‘heard’ thing. It’s part of the BOSS spec. See here:
Now I can’t find where it’s specifically called out on the BOSS S-1, but if the S-2 and even the N-1 don’t support patrol read, then the S-1 certainly doesn’t either.
@firesyde424, since I know you have the R7525 you’re very likely using the S-2 version. Dell’s documentation says clearly that patrol-read is not supported. TRIM is only supported in non-raid mode, and that consistency checks are also not supported.
You can run the BOSS card in non-raid mode so it shows up as 2 separate disks, then use TrueNAS ZFS mirror for your boot drive. At least then you are getting CoW and checksumming and all the advantages that ZFS offers over a hardware RAID1 that doesn’t do any data verification.
EDIT: Looking closer at the BOSS N-1 it appears Dell removed the ability to run the NVMe version as separate disks. It’s only RAID0 or RAID1. But the OP with his 730xd, or other 14th or 15th gen servers using the S-1 or S-2, the above stands true.
When I meant “heard”, I was specifically referring to the suggestion that we separate the RAID1 on the BOSS card and use ZFS to manage the independent drives. I was not referring to the capabilities of the BOSS card itself. I should have been more precise in my wording there.
I’m not opposed to a change in how our TrueNAS servers are constructed, but given the complete lack of long term issues with our BOSS-S1, S-2, and N-1 backed TrueNAS servers, never mind the myriad of other non-TrueNAS servers we’ve deployed that use these cards without issue, I’m less inclined to do so. Additionally, even when it’s possible to split the RAID1, I’m not aware of a “passthrough” mode available for these cards. While Dell’s usage of the terms can be inconsistent, “Non-RAID disk” is usually not the same thing as “passthrough” or “HBA mode.” In my experience, “Non-RAID disk” still usually has some kind of obfuscation in front of it and is not a direct pass through in the way the TrueNAS hardware guide asks for.
I do recognize that my design for our TrueNAS servers around their boot drives can add risk. However, the decision was made to use those cards for a few reasons. For our use case, overall capacity matters. In the example of the R7525 you mention, giving up two chassis slots for boot drives would have reduced the usable capacity of that server by nearly 60TB. For the rest of our NVME based TrueNAS servers, that cost would be anywhere from 16 to 30TB, just so I could have boot drives that supported TRIM and consistency checks. For us, the decrease in capacity is not worth the reduction in risk. We perform automated weekly configuration backups of all our TrueNAS Core and Enterprise servers, manual backups after any configuration changes, and we also do not use a swap partition on our TrueNAS boot drives.
I have an R7525 and BOSS S-2 arriving in about a week. I’ll test with the split drives configuration.
Thank you so much. When I went back to my vendor, they said they also have the R7425 that supports both Gen 1 and 2 Epyc. Is the R7425 with a Rome Gen 2 Epyc “better” than the 14th Gen Dell with the Gen 2 Xeon? 24 NVME configuration. Am I getting more PCIe lanes, maximum performance? or does that not matter since they both have the PCIe switches like you and this document below mention. Seems like the off lease price for the R7425 and R740xd aren’t too far off… I can afford either of them.
I read this: NVMe and I/O Topologies for Dell EMC Intel and AMD PowerEdge Servers | Dell USA
Also, if I understand your conversation… the S-1 might not be optimal? should I ask for the S-2? or does the S-1 have a non raid mode?
We have had some minor issues in the past with our AMD Epyc TrueNAS Core servers, with things like incorrect CPU temperature probe scaling and an issue where a hardware queue setting had to be changed before the system would detect all the drives properly.
Having said that, these days, we don’t see those issues anymore and I would strongly suggest you get the AMD Epyc based system. The Rome Epyc CPUs especially are in a situation where a single Epyc has more PCI Express lanes than two Gen 1 or Gen 2 scalable Xeon(128 vs 2x48). Most workloads will struggle to saturate the 32 PCIE lanes that a 740xd gives you for NVME drives. Where you will notice more is in the higher single threaded performance and the cheaper pricing.
1 Like
You’re asking some great questions.
The BOSS S-1 is for 14th generation servers that don’t have dedicated BOSS slots on the chassis. The S-1 is just a normal PCIe card that has 2 M.2 SATA ports on it.
The BOSS S-2 is for 15th generation servers. The form factor is proprietary and makes accessing the SATA SSD’s easier because they attach to little sleds in the back of the server.
The BOSS N-1 is for Dell’s newest 16th generation servers. They are M.2 NVMe rather than the previous S-1 and S-2 being M.2 SATA.
Regarding the questions about the r740xd vs the r7425, the IO topology diagrams spells things out nicely.
For the R740XD 24x NVMe backplane, it requires 2 extender cards. 12 drives cabled to each card via 2x8 cables. I’m not sure why the diagram says it’s an x16 cable because its 100% 2 separate cables to each of the extender cards. Either way, you’re taking 12 drives at 4 lanes each (48 lanes) and using a PCIe switch on the back of the backplane to switch that down to 16 lanes. Which means you’re getting ~0.33 lanes (EDIT: 1.33 lanes) per drive.
For the r7425 you’re still using PCIe switches, except now you don’t need PCIe extender cards, you’re just mapping those lanes directly to the CPU. However you’re still switching, but according to Dell’s diagram its a 31:16 ratio (or ~0.5 lanes per drive). Something I don’t understand about the 7425 though is why they depict 12 drives only being 31 lanes when we all know it’s 48 lanes. Regardless, I would echo @firesyde424 's advice and go with the AMD based system.
“I’m not sure why the diagram says it’s an x16 cable because its 100% 2 separate cables to each of the extender cards.”
Yeah, I wondered that too. We have several NVME backplane R740xd servers for various uses, one of which has served as a TrueNAS Core for more than 4 years now. I dug through that server’s iDRAC and found that it’s using a PEX 9733 as the chip for the PCIE expander. It’s interesting because the specs in the iDRAC and Broadcom’s website both list it as a 33 lane part. Given that the card has two ports, it’s reasonable to assume that each might be x16, but the entire card itself is only an x16 slot width so I’m not entirely sure. I dug up some old pictures of the internals on that server and, sure enough, two cables per card to the backplane.
I did some spelunking through Dell’s documentation on the R740xd and I wasn’t able to find anything either way. I have a couple pictures of the cable’s part number, but they aren’t legible enough to get anything usable so I couldn’t even look it up that way.
1.33 lanes per drive I think.
(16 lanes, 12 drives)
You’re right, it would be 0.33 x 4. So ~1.33 lanes per drive.
1 Like
Which is about 1.25GB/s per drive. Which is not that bad.
Or about SAS 12gbps.
Assuming gen3
But then why not just use SAS. Latency I guess.
And I guess that explains the design.
I appreciate you saying that, I can’t word things as well as I want to but after digging in the rabbit hole, I feel like I have finally found the right way forward…
S-1 sounds like a good solution, it’s a shame the Dell only has 1 SataDOM… Dell marketing at its finest.
Trying to reconcile the conversation between you and firesyde424 about lanes makes me concerned about which CPU I get… figure the wrong CPU won’t have enough lane support to take advantage of the storage layout?
The Intel 6246 (12 core, 24 thread) supports 48 lanes, so thats 96 lanes between the two. PCIe 3
The Rome 7642 (48 cores, 96 threads) supports X lanes? and PCIe 4 Can’t seem to find the details on max lanes… I see 128 lanes, so 2x 7642 would be 256?
Waiting now to see which Rome processors they have available for my build. I requested these 3, 7642 7542 7532. The Rome processor seems like a next gen processors compared to the Xeon?
I am also trying to understand the lanes per drive. Seems like the AMD has a more direct data path without the expander? Does the AMD will have the benefit? R740xd = 1.33 lanes? and R7425 = .5 lanes per drive? Am I reading something wrong but it seems like the 740 has more lanes per drive? Having a hard time understanding this…sorry. Is the R7425 still the way to go?
The Dell BOSS S-1 (adapter card) has 2 sata doms.
https://www.dell.com/support/manuals/en-us/boss-s-1/boss_s1_ug_publication
For the r740xd you don’t need to worry about lanes. Because of the extender cards you’re only getting 32 lanes total from all the NVMe drives.
For the r7425 you don’t need to worry about lanes either. Because again, with the PCIe switch you’re only getting 32 lanes total from all the drives. The only real difference between how the r740xd is doing it from the r7425 is that with the r7425 you don’t need the extender cards. The PCIe cables go from the backplane directly to the motherboard. With the r740xd the PCIe cables go from the backplane to the extender cards which plug into PCIe slots on the risers.
If you don’t need the extra PCIe slots, go with whatever option gives you better bang for the buck on the refurb market regarding CPU cores, clockspeeds, memory, etc.
Regarding the lanes per drive, my math was wrong on both systems. It’s ~1.33 lanes per drive on the r740xd, and ~2.0 lanes per drive on the r7425.
2 Likes
[quote=“cyr0nk0r, post:34, topic:856, full:true”]
The Dell BOSS S-1 (adapter car…
Advice that is worth its weight in gold…thank you, my business thanks you too!
I’m going to go with the AMD solution based on the CPU cores, clock speed. Here is my final build, if you anyone has any comments… I’m still a bit nervous about this. Have never had an AMD based server, and as I have read… getting good performance is not a guarantee. I was a little nervous about the XXV710-DA2 but have not read anything derogatory and can always replace it if need be. Did read the guide and see that X540 has a different driver than the X550, and that the X540 might be better choice… using an X550 in another system with scale and no issues. I also realize now that the NVME drives are no guarantee of improved performance… however my per drive cost is going to be less than SSDs, allowing me to get more raw storage for the same price.
Dell PowerEdge R7425 24-Bay NVMe 2.5" 2U Rackmount Server
1024GB [16x 64GB] DDR4 PC4-3200AA ECC RDIMM
2x AMD EPYC 7532 2.4GHz 32 Core 256MB 200W 2nd Gen Processor
20x Dell 3.2TB NVMe SSD 2.5" Gen3 MU Solid State Drive
Dell Dual Port 10GBASE-T + Dual Port 1GBASE-T rNDC | Intel X550 I350
PCIe Slot: Dell Dual Port 25GB SFP28 PCI-E CNA | Intel XXV710-DA2
PCIe Slot: Dell Dual M.2 6G PCI-E BOSS-S1 Controller + 2x Dell PE 120GB SATA SSD M.2 6Gbps RI Solid State Drives
2x Dell PE 1600W 100-240V 80+ Platinum AC Power Supplies
Dell iDRAC9 Remote Access Enterprise License
Dell PE 2U RapidRails II Sliding Rail Kit
Dell PE 2U Front Bezel
2x Power Cords
1 Like