Intel X710-DA2: Required Tuning on SCALE?

TrueNAS General
1

Hello,

as far as I know the NIC of type Intel X710-DA2 is some kind of special.
Is there anything else that should be tuned beside:

  • Disable LLDP
  • Remove Vendor Lock

Are there any tuning options in SCALE that have to be considered for the X710?

Thanks a lot in advance,

Thomas

2

Really noone with an Intel X710-DA2 and some experience on this card?

The 10 Gig Networking Primer mentions

The X710 is an advanced card with some rather fiddly features, such as an in-firmware LLDP driver, and the ability to provide full VF support tp a TrueNAS VM when properly configured in ESXi. As with the X520, this high performance driver is authored by Intel, but unlike the X520, the SFP vendor lock is enforced in the card firmware, so Intel-compatible SFP’s are required.

So what I found so far to disable LLDP in the firmware:

To remove the vendor lock:

Questions:

  • Is there anything else to consider with the Intel X710?
  • Does someone know which version of the X710 driver is included in the latest TrueNAS SCALE?
  • Even that this thread is about the Intel X710, is there any other 10 Gig SFP+ NIC that supports higher C-States, supports ASPM and works with TrueNAS SCALE? At least the X710 supports higher C-states and ASPM regarding to https://z8.re/blog/aspm.html. The Mellanox cards seem to have no proper support of the power saving measures.

Thanks a lot in advance.

3

I’ve got the XXV710-DA2, currently running in 10Gb. Haven’t tuned anything in particular other than some TCP buffers etc (not specific to this particular card). I bought the card used and flashed the firmware to latest version from Intel’s web site.

Having said that, I use LACP but haven’t looked into LLDP as you mention above, should maybe look at that


4

Oh, and I use DACs from fs.com so have never had to worry about transceiver compatibility, the DACs work just fine even if they are coded for Cisco.

2 Likes
5

Final note about Mellanox. I know they are not so popular here, probably due to poor support in FreeBSD? But Linux support is rock solid, with Mellanox themselves a fairly active contributor, there’s even a Mellanox firmware update tool as part of standard Debian. They are also one of the most commonly seen brands in data centres.

I’ve read like yourself that they prevent power saving in higher C-states, at least the older cards. Haven’t assessed that, but I do have a couple of their cards in my home lab (on Linux/Proxmox) and only had good experiences. Given Scale is based on a modern Debian kernel I absolutely expect Mellanox would be well supported there too, but no personal experience.

1 Like
6

What Mellanox models are you using and could you check which C-states your system reaches and if ASPM is supported (a lot of infos can be found at https://z8.re/blog/aspm.html)?

7

ConnectX-4 Lx and they seem to have ASPM enabled. According to powertop I’m typically in C2 most of the time but there are also no higher states listed. I’ve never looked at this before, maybe there are also Bios settings related to this.

All NICs are on latest available firmware. CPU is AMD.

8

Zur Info: Unter https://www.reddit.com/r/homelab/comments/15eqm9s/mellanox_connectx4_questions_before_buying/ wird darĂŒber diskutiert, dass die Mellanox trotz ASPM verhindert, dass das System höhere C-States erreichen kann.

9

Maybe Intel is the way to go then if you care about power efficiency and the server idles a lot.

This seems like a bit of a jungle though
 Hardware, firmware, bios settings, OS and CPU governors, software
 all of which combine in different permutations leading to different behaviour and results.

I’ve never looked into it much. Probably should.