Truenas Scale how to set network interface full duplex and flow control on?

TrueNAS General
1

Hello,

How do i set up my 100gbe network card inside a Scale to handle full duplex and flow control? I cannot find any command in the cli or the UI, only how to change the MTU.

Thank you!

2

bump. I can’t find any relevant information that isn’t obsolete on flow control for truenas and how to set it, the only references I’ve found are sysctl tunables that don’t exist in scale.

3

First, let us see what your NIC and switch support. Please supply the output of ethtool against your NIC. My test VM’s example output;

root@truenas[~]# ethtool enp0s3 
Settings for enp0s3:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Supported pause frame use: No
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	MDI-X: off (auto)
	Supports Wake-on: umbg
	Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
	Link detected: yes

A non-VM would likely also include a section on the Link Partner, (aka Ethernet switch);

	Link partner advertised link modes:  10baseT/Half 10baseT/Full
	                                     100baseT/Half 100baseT/Full
	                                     1000baseT/Half 1000baseT/Full
	Link partner advertised pause frame use: Symmetric Receive-only
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
1 Like
4

Settings for eno1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: g
Wake-on: g
Current message level: 0x00000000 (0)

5

Up stream of is a Ubiquiti Aggregation switch (it’s an all SFP+ switch), and one of their Pro Max 16 PoE switches upstream of that, flow control is enabled on all the network switches.

6

It appears your NIC does not support auto-negotiation. It could be your SFP+ does the auto-negotiation of the speed it’s self, and does not allow further negotiation.

Now is the time to ask for the model of the NIC card, chip manufacturer, and chip model. Plus, the same for the SFP+ plugged into it.

7

NIC is a Dell XGRFF daughter card, quad SFP+, the underlying chipset I believe is a Broadcom 57840s and the cabling is Cisco Direct Attach

8

Sorry, I have no further information.

Perhaps someone else will be able to assist you now.

10

did anyone figure this out? I am wanting to hardcode speed and duplex.

11

@Soleous Hard coding speed and duplex, while not recommended at 1Gbits/ps or higher, should be straight forward using;
ethtool -s NIC_DEVICE --change speed X duplex half

This of course will not survive a reboot… so a post boot script will likely be needed.

You many have to experiment around this, as I can’t give you exact syntax since I have not had to hard code any NIC, at home or the Enterprise in at least 10 years.

12

@Arwen thanks for the response.

why do you say its not recommended for 1Gbe and above?

13

It is simple, at 1Gbit/ps Ethernet speed, auto-negotiation is a required feature and must work. But, hard-coding could be used.

At 10Gbit/ps or higher, auto-negotiation is mandatory. Meaning it must work and be used by default.

Not sure about the fill in speeds of 2.5Gbit/ps and 5Gbit/ps speeds, but they should be similar to 10Gbit/ps in that auto-negotiation must be implemented and used by default.

In essence, if you have a 1Gbit/ps, (or higher), Ethernet link that is not working well with auto-negotiation, then you have bigger problems than hard coding speed or duplex. For example, if you have a high speed sending device but the receiver can’t keep up, normal TCP or SCTP would throttle the speed. But, if not, then Ethernet can use receive or send pausing, which are features of an auto-negotiated link.

2 Likes
14

Thanks for providing some colour. I’ll look into this in more detail.

Regarding anyone else looking into this or for posterity. Not over the past decade mind, I have needed to hardcode duplex and speeds for ISP’s or device to device connectivity. Device to device used to be a big problem with negotiation tripping over itself, today at 10Gb its responding in a similar way hence me needing to hardcode. I used Arwen’s suggestion resulting in ethtool -s enp4s0f1 speed 10000 duplex full.

Additionally, my X550’s do not autonegotiating at 2.5Gb and 5Gb, to resolve this I used ethtool -s enp4s0f0 advertise 0x1800000001028. This enables advertising at 100BASE-T, 1000BASE-T, 2500BASE-T, 5000BASE-T and 10000BASE-T.