10gb NIC Issues

TrueNAS General
,
1

Hey fellow TrueNAS netizens! I really would appreciate some help here.

I’ve been trying to diagnose and fix an issue I’ve been having with multiple different ‘Intel’ 10gb NICs in my TrueNAS system. I’ll detail the NICs, DACs, Optics, and other fixes I tried.

All these Optics and DACs where plugged into Mikrotik switch, namely the MikroTik CSS610-8G-2S+in.

NICs:
2 different 10Gtek Intel X520-DA1 clones
1 Genuine Intel X520-DA2

DACs:
1 3m 10Gtek CAB-10GSFP-P

Optics:
2 H!Fiber AXS85-192-M3
2 10Gtek AXS85-192-M3
2 Intel FTLX8571D3BCV-IT

When the cable (DAC, Optic) is plugged into both the NIC and the Switch, the Mikrotik WebUI is able to see the SFP device, and shows that there is a 10G Link on that port.
However, in TrueNAS the NIC comes up, then immediately down again.

Another fix I tried was taping over PCI pins B5&B6 on the NIC, in reference to this post I saw recently, no change.
https://www.reddit.com/r/homelab/comments/1j1s6ec/info_used_enterprise_cards_hbanic_and_smbus_issues/

As an aside, I tried to plop in a NICGIGA AQC113C 10gb Ethernet card, along with a ipolex 10GBase-T transceiver on the Mikrotik. That came up immediately, so I know 10gb is possible on this server, and this PCIE slot.

The specs of the TrueNAS server are as follows:

System Information:
Version:ElectricEel-24.10.0.2
CPU: Intel I7 8700
MOBO: GIGABYTE B360 HD3
RAM: 64GB 3200MHz DDR4 (4x16GB)
HBA: LSI 9207-8i IT Mode
PSU: EVGA 550W PSU

2

Why do you have multiple NICs? Two would already be puzzling, but three when the switch only has two 10G ports…
What are you trying to achieve?

3

I think that is a listing of the NICs the OP has tried. I hope all three aren’t installed at the same time.

1 Like
4

If the Mikrotik connection remains active, but it drops off FreeNAS, then it’s not a “hardware” issue per se, but the Intel NIC shutting down comms to the port & faulting out — likely because of an optics compatibility issue.

Not sure if you’re aware - but the firmware on Intel X520-DA NICs have a vendor tag lock on optics - IE, they reject any optics other than genuine Intel ones (with the little silly blue hologram label) or signed 3rd party ones.

This might even apply to your Intel transceiver - it may A) not be a supported Intel model, or B) be a clone — neither of which will authenticate the vendor tag correctly with the NIC.

Run lspci from Shell & check the status of the X520 - it’ll likely report “Faulted - SFP Unsupported non-genuine Intel”, or something along those lines.

The kicker is, that the NIC faults out that port on an unsupported transceiver to such an extent… that any subsequent attempts to connect a DAC or even a correct Intel receiver, will still fail to establish a link.

IE - any DAC should work just fine, BUT, if the NIC has already shut down the port comms on account of the optics, you won’t get it reactivated without power cycling the server. (There may be a command to manually reset the port / entire NIC - I’m unsure)

Shut down, unplug the PSU, then plug in the DAC, fire the server back up again and see if it now manages keeps the link up. If it does, you’re off to the races…

If you prefer optical though, there are two methods to get that working:

  1. A workaround command that can be inserted into your sysctl config… aka the Unsupported Bypass.

Set a sysctl value of hw.ix.unsupported_sfp to “1”

Edit to add: I spotted now that you’re on TrueNAS Scale - I’m unsure if a sysctl entry is the right procedure… (Linux noob and haven’t played with Scale much at all)

SSH into your TrueNAS

enter the following command:

sudo midclt call system.advanced.update ‘{“kernel_extra_options”: “ixgbe.allow_unsupported_sfp=1”}’

Then, having the SFP+ module installed and connected to a switch or another 10G port, reboot.

Now the interface should show up and be configurable in TrueNAS UI.

From:

or, 2) Somewhat more difficult - patch the NIC’s EEPROM to remove the transceiver check… Then you can use any transceiver on any machine, without having to employ an unsupported bypass.

Hope this helps.

2 Likes
5

Sorry it’s a long thread I didnt read it in full

Did you try to fix speed on Mikrotik side?

6

Oh, and just for the sake of being thorough…

When attempting to use Intel Optics Transceivers on both ends of the connection between my own X520-DA2 NIC and Unifi US-48-500W switch (and my USW-Aggregation switch for that matter), that ALSO didn’t work for me.

Apparently Unifi kit doesn’t play nicely with Intel optics… which just added to the complexity of my own wild goose chase, getting X520’s to work in my setup… :roll_eyes:

While your Mikrotik should accept Intel optics just fine, your mileage may vary - Alibaba has plenty of listings offering non-genuine clones of this particular transceiver model - I don’t even have to point out how much further a sketchy, half-baked clone transceiver could further confuse matters.

If in doubt, don’t be afraid to mix and match different brands of similarly specced (850nm MM) transceivers… Intel on the NIC side, and 10Gtek / H!Fiber on the switch side…

And if your link speeds stubbornly stick to 1Gb on Auto-Negotiation, simply disable it and manually specify a 10Gb connection rate on the switch… Intel NICs and Optics can be fidgety about correctly auto-negotiating 10Gb - or even correctly engaging in negotiation to begin with.

In fact, as alluded to previously - restart the server from scratch, and first verify that all the NICs are working correctly via DAC… only then move to optics and tweaking unsupported bypass and/or combinations and/or settings till you figure out a working recipe.

1 Like
7

Hey, thanks for all your ideas!

I decided to pull the server out of the rack and place it right next to the switch for this testing.

My current setup is Using only the Intel X520-DA2, and the 3m DAC.

In my Mikrotik Router, it shows the connection as up, and set to 10g.

image

However in Truenas, both the SFP interfaces are down.

image
image938×255 5.98 KB

When I try to change the settings of the interface the DAC is plugged into, like so:

image
image452×290 4.22 KB

I get the following in dmesg, showing the link coming up for a second, then back down.

image

Any ideas at this point? If even the DAC doesn’t work, I’m not sure where to go.

8

To any thread watchers, or future Googlers that find this thread, the problem ended up being the Mikrotik Switch. (SS610-8G-2S+in)
It needed a Firmware update, go figure!

Going from SwOS v2.13 → v.2.18 allowed all the Optics and DACs I had on hand to work!

If any of you experience connection weirdness, check out your switch too!

1 Like
9

No ways… wow…

Honestly wouldn’t have expected that from a Mikrotik switch.

We live and we learn.

Glad you got sorted - hopefully the SFP transceiver info helps you further down the line.

10

OS updates exist for a reason…