Let me start by mentioning that @dan pointed me to @jgreco’s guide 10 Gb connectivity, here: 10 Gig Networking Primer | TrueNAS Community . I’m reviewing it now, but I’ve already run into a bit of a conundrum that I need some advice on.
tl;dr I’d appreciate any advice on whether I should go with the Intel x520-da2 or the Mellanox ConnectX-4 Lx (latest NVIDIA stock firmware) to use with DAC cables (and maybe RJ-45 transceivers later) on a machine with a single PCIe 4.0x4 open slot.
I have a PCIe 4.0x4 slot available. It’s my only slot.
I’ve got an Intel X520-da2 and a Mellanox Connect-X4 Lx available. The Mellanox was a Dell OEM card, but I cross-flashed it to the latest NVIDIA/Mellanox stock firmware.
I know that the Intel card is considered the more robust option for driver support over the older X2 and X3 models, but what about the X4?
The Intel X520-da2 is out of support and very picky about SFP+ RJ-45 modules, and the Mellanox is/was getting firmware updates as late as September 2024.
I’ve also never tried to put the Intel X520-da2 in an x4 slot. It’s a PCIe 2.0x8 card, so I’m not sure how it will like that.
I’m also not sure why a 2x10GbE card like the x520-da2 needed a PCIe 2.0x8 connection (4 GB/s full duplex), but I suspect that’s just me blundering around not quite knowing exactly what I’m doing, as usual.
I have 3 Mellanox ConnectX 4 Lx cards and they all work great at 25 gbps. Not a single issue with a driver under Linux / Opnsense.
I do have E810 Intel card that I bought for 25gbps nonsense router, mainly because I read Intel driver has better support for multi-queues (ice_ddp). But I ended up not using it as Mellanox gives me close to 25gbs throughput and cost $50
I should have specified in my OP and not just the tags: I’m on the latest version of SCALE/CE. So, Linux. The PCIe 4.0x4 slot is a physical x4 open slot.
I agree that that a PCIe 3.0 NIC is the better option; I was just a bit concerned after picking up that Mellanox could be disfavored in a TrueNAS environment.
The Mellanox card I have is the same one @bugacha uses. The documentation actually specifies that it’s designed to run in 3.0x4 mode, as well as 3.0x8, so I wouldn’t be doing anything unexpected with it.
I’ve tested it in a PCIe 3.0x4 slot on an Arch Linux system and got two separate, full-speed 10 GbE interfaces.
Still rebuilding my system so I haven’t actually got 10 GbE set up yet, but the card (with 2 interfaces, according to lspci) came up just fine using the mlx5 driver.
Will report in once I have everything working just to close the loop for anyone who finds this thread later.
The PCIe 2.0 aspect of the X520 would be a dealbreaker for me. As for the Cx-4 in an x4 slot, it works fine as I have one here. Real-world throughput will max around 3.2GB/s thanks to the confluence of PCIe 3.0 and the x4 slot width. I’ve twisted every knob under the sun and spent far too much time trying to exceed this number with zero success across multiple OS.
No issue with 2x 10Gb but 2x 25Gb is bottlenecked.
Thanks for your reply. So far, it’s stable and working well at 2x10 GbE. I don’t have any 25 GbE equipment, so I can’t be tempted into trying to make it work.
Though, I will say: having 10 Gbps+ networking sure turned out to be a great way to find all the bottlenecks in my network that have nothing to do with the NICs in my servers or my switches.
Intel X520 uses PCIe 2.0 x8 = 4GB/s of bandwidth (500MB/s per lane), a 10Gb port = 1GB roughly so PCIe 2.0 x8 has plenty of overhead to run dual 10Gb ports at full speed.
Personally I am a Chelsio person (aside from the current one in my TrueNAS), always get full bandwidth with no iffy stuff, nor worrying about fake intel cards if you get them used off Ebay.
I reckon it’d be fine on x8. With the above math it’d be OK on x4 as well but I’ve never been able to run any slot at its full “PCIe math” speed ever.
At PCIe 3.0 x4 the very best real-world throughput I’ve seen is 3.3GB/s. I’ve tweaked interrupt coalescing (both count and delay), switched to poll mode, enabled aggressive CQE compression, jumbo frames, loose PCIe re-ordering – pause frames on/off, hardware flow control on/off. Checked for overruns in the IP stack, retransmits, buffer exhaustion, codel vs cubic… Everything I can get my hands on. 3.3GB/s was it. Extrapolating that down to PCIe 2.0 suggests it’d bottleneck a pair of 10G ports unless you could move to an x8 slot width.
Not that 1.6GB/s is a bad thing. The disk subsystem in my server can’t sustain that and I’ll bet there’s a bunch of us in the same boat.
I have a Dual 10Gbps SFP+ Port, Intel 82599(X520-DA2).
Configured in LACP.
I want to change over to scale as core is not supported anymore but having issues with the dual sfp+ nic in lacp.
I have even tried fresh install and had to give up and wait for an update.
Still the latest scale will not initialise the sfp+ ports. but then I install core and it works fine.
I had to give up on this upgrade as I had other projects to attend to. Now I have some time I want to know why the intel 520 dual sfp+ port will not initialise.
to clarify TrueNAS Scale sees them but fails to bring them up.
I am stumped and don’t know where to go from here.
I run PCIe x8 with a x520 and can do about 9.6Gb/s, which is about the expected max with overhead, using iperf testing on an HP Z6 G4 and also on a Dell T5820.
I had to reinstall Core 13.3 to get the SFP working.
But as to what was tried…
None of the SFP+ ports would go online.
I removed lacp settings and tried to get one port up which it would not.
I looked in log and the Itel x520 spf ports was detected but failed to initilise.
Log showed a driver issue which I was unable to correct as this is locked.
After which I had to revert back to core but was forced to use Core 13.3 because what scale did to the zfs. so Core 13.3 was the only comatible os available which supported the zfs features.
