Problem Connecting Servers using SFP+

General Discussion
, ,
1

Hello everyone,

Disclaimer: as this is my first Post and English is not my mother tongue feel free to educate me on misplaced words or Posts.

I recently bought an Atom based miniPC for use as the NAS in my network
and run light 24/7 tasks to reduce power consumption. The existing AMD box should be used for Temporary heavy workloads and use the storage of the new Atom box.
After setup there was no link on the sfp+ ports.

I tried to forcing negotiation speed on both servers, but couldn’t figure out how to do it on scale.
Then i started to try out different combinations of hardware i have. These tests i summarized in the following Scearios.

I also tried Scenaroi1 with another scale box in the basement used for backups and connected using fiber. Same symptoms on Server1.

Setup:
Server1: Qotom Q20332G9-S1
builtin intel x553 4Port SFP+
Truenas scale Dragonfish-24.04.0

Server2: Custom build AMD box
pcie Chelsio Communications Inc T320 2Port SFP+
Proxmox 8.2.2

Scenario1:
When i connect Server1 and Server2 with DAC from 10GTek on Server 2 the Port light up. On Server1 nothing happens.
Server1 dmesg shows

ixgbe 0000:0b:00.1 eno2: detected SFP+: 4

Server2 dmesg shows

[75275.709949] cxgb3 0000:2b:00.0 enp43s0: unknown PHY module inserted

Scenario2:
When connectiong 2 Ports on Server1 the connection gets established.
Server1 dmesg shwos

[ 3102.989686] ixgbe 0000:0c:00.0 eno3: detected SFP+: 3
[ 3104.067891] ixgbe 0000:0b:00.1 eno2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 3104.087861] br5: port 4(eno2) entered blocking state
[ 3104.088692] br5: port 4(eno2) entered listening state
[ 3104.960065] ixgbe 0000:0c:00.0 eno3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 3104.978987] br5: port 3(eno3) entered blocking state
[ 3104.979780] br5: port 3(eno3) entered listening state

Scenario3:
2 Ports on Server2 with DAC
dmesg shows

[75582.813737] cxgb3 0000:2b:00.0 enp43s0: unknown PHY module inserted
[75596.517885] cxgb3 0000:2b:00.0 enp43s0d1: unknown PHY module inserted
[75597.589738] cxgb3 0000:2b:00.0 enp43s0: link up, 10Gbps, full-duplex
[75597.590255] vmbr3: port 1(enp43s0) entered blocking state
[75597.590643] vmbr3: port 1(enp43s0) entered forwarding state
[75597.601738] cxgb3 0000:2b:00.0 enp43s0d1: link up, 10Gbps, full-duplex
2

Hello,
It seems a linux drivers problem (there are some discussion about this).
I have also a qotom running Proxmox and 2 VM:
One VM is OPNSense (based on FreeBSD) and the ports are working (PCIe passthrough mode).
The other VM il TrueNas Scale and the ports are not working (also in Proxmox the ports era KO and both are based on Linux).
The error from Proxmox console is:

root@pve-00:~# ethtool eno3
Settings for eno3:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Auto-negotiation: off
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no

Best Regards,
Antonio

3

Thanks for the reply. I read something like that, but thought it had been solved with the newest Kernel.

I’ve tried it with Core and it works.
Looks like I will be using Core for the time being.

Florian

1 Like
4

Not sure that’s really a solution to getting it to work in Scale :wink: