ASRock Rack E3C246D4U2-2T 10GBE ports only running at 1GBE

TrueNAS General
1

New system setup. Ubiquity managed switches CAT7 cables. When system first boots my switch identified port as 10GBE then after a brief time it drops to 1GBE. I cannot seem to find where I would specify link speed in TRUENAS settings. I specified on the switch port however does not seem to make a difference. I suspect a Ethernet driver issue, however I am new to TRUENAS so I thought I would ask here before chasing wabbits.

2

What’s supplying the 10gb on the server? Can you bypass the switch and go direct then run iperf to run some tests? If it’s a realtek chip it’s no good.

3

I don’t think running iperf will help when the actual link speed drops.
I also don’t think specifying a link speed will help - I suspect it drops to 1Gb because it can’t sustain 10.
You may check with ip -stat s link whether there are errors on the interface - those could explain this.

4

I’d try hard coding everything network related and see if that changes the behavior (i.e. turn off auto-negotiation and code 10GbE, full-duplex, default 1500 MTU, and so on).

Sometimes, the auto-negotiation stuff works for a while - until it doesn’t. Had some issues between Mikrotik and my NAS negotiating to run at 1GbE while using a SFP+ port in the NAS.

1 Like
5

I would like to but I am new to TRUENAS and I haven’t played with Linux in a serious way since Redhat 6. How would I specify link speed in TRUENAS? what is the .conf file or modern day equivalent.

6

The motherboard has dual Intel X550-AT2 10GBE connecting to Ubiquiti switch 10GBE port via CAT7 cable approximately 12’ long. Tested multiple cables and ports. I have another workstation using the same cable at 10 GBE on the same ports. Which tells me it should be the onboard NICs or drivers for them. I’m going to toss a windows live USB on the machine tomorrow and see if it connects at 10GBE under windows, that should tell me if it’s the NIC or the driver.

7

While it’s working in windows see if you can use their firmware tool to make any changes to the card. I’d force 10gb, there is a history of that chipset acting up especially when it’s trying to auto negotiate. Google has posts going back years about that chipset being weird unless it’s in windows.

I went 10g also but all my stuff is fiber. Heaps of good sfp hardware from Intel and mellanox.

8

So I found a post on a Ubuntu site that addressed the x550 not going past 1GBE and tried it out on my TRUENAS and it worked. I am posting the site so if someone else runs in to the same issue I did they have the source. https://askubuntu.com/questions/1406445/ubuntu-22-04-server-intel-x550-advertised-speed-not-correct

TLDR:

Bandaid= ethtool -s advertise 0x1800000001028
fixes the issue but if you reboot you have to do it again.

Permanant Solution= sudo nano /etc/systemd/system/ethtool-advertise.service

Then paste this in:

[Unit]

Description=Set ethtool to advertise all supported link modes

After=network-online.target

Wants=network-online.target

[Service]

Type=oneshot

ExecStart=/usr/sbin/ethtool -s eno1 advertise 0x1800000001028

ExecStartPost=/bin/bash -c ‘/sbin/ip link set eno1 down && /sbin/ip link set eno1 up’

RemainAfterExit=yes

[Install]

WantedBy=multi-user.target

***** CHANGE eno1 to whatever your ethernet port is *****

then ctrl-o followed by ctrl-x

next type:

sudo systemctl daemon-reload

sudo systemctl enable ethtool-advertise.service

then after a reboot it should persist

2 Likes
9

NIce, exactly the same issue here and thanks to your post I found the solution.

I have an AsRock Rack E3C246D4U2-2T and thus 2x 10G RJ45 connections. It is hooked up with cat6a cable to an unmanaged 2.5G rated switch in the room, which again is connected to the main 2.5G switch near my modem/router (on its 2.5G rated port). So I would expect everything should be able to run at 2.5G speeds on paper. Fair enough, I have no other devices in the network which actually can take it beyond 1G but as that is about to change over time I like things to be ready.

In Truenas shell ethtool gives me the following (and same for eno2):

Settings for eno1:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        MDI-X: Unknown
netlink error: Operation not permitted
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

The linked AskUbuntu post uses the following to force advertisement of all speeds:

ethtool -s <ethX> advertise 0x1800000001028

I did this for both eno1 and eno2, both advertise all speeds now and immediately the result shows Speed: 2500Mb/s.
I’ll follow your additional steps to make it stick.

2 Likes
10

It should probably be done by an Init/Shutdown script or setting like in post number 4 of the linked old, forum post. I don’t know if other methods will survive upgrades of TrueNAS

1 Like