Help with TrueNAS Scale - Link Aggregation Not Working

tushar · September 1, 2024, 11:38pm

Hello everyone,

Setting up TrueNAS Scale for the first time. I have a network card with 2 10G SFP+ ports which I have connected to my TP-Link switch which also has 10G SFP+ ports.

Currently TrueNAS Scale shows two links (2 ip addresses) active, one for each 10G port. I am now trying to setup Link Aggregation.

On TP Link, I have added the two TrueNAS ports to LAG1.

But on TrueNAS, I am not able to get the bond created.

Here are the steps I am following.

After saving this, I see bond0. But when I submit “test changes”, the GUI goes in loading for a minute or two and the bond disappears.

I don’t know what I am doing wrong. I also tried with CLI - network interface create... but even after committing the changes, the bond isn’t created.

What am I missing?

I also checked kernel logs and found following logs

Sep  1 16:07:53 truenas kernel: ixgbe 0000:41:00.0: registered PHC device on enp65s0f0
Sep  1 16:07:53 truenas kernel: bond10: (slave enp65s0f0): Enslaving as a backup interface with a down link
Sep  1 16:07:53 truenas kernel: ixgbe 0000:41:00.0 enp65s0f0: detected SFP+: 3
Sep  1 16:07:53 truenas kernel: ixgbe 0000:41:00.1: registered PHC device on enp65s0f1
Sep  1 16:07:53 truenas kernel: ixgbe 0000:41:00.0 enp65s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Sep  1 16:07:53 truenas kernel: bond10: (slave enp65s0f1): Enslaving as a backup interface with a down link
Sep  1 16:07:53 truenas kernel: bond10: Warning: No 802.3ad response from the link partner for any adapters in the bond
Sep  1 16:07:53 truenas kernel: bond10: (slave enp65s0f0): link status definitely up, 10000 Mbps full duplex
Sep  1 16:07:53 truenas kernel: bond10: active interface up!
Sep  1 16:07:53 truenas kernel: ixgbe 0000:41:00.1 enp65s0f1: detected SFP+: 4
Sep  1 16:07:53 truenas kernel: ixgbe 0000:41:00.1 enp65s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Sep  1 16:07:53 truenas kernel: bond10: (slave enp65s0f1): link status definitely up, 10000 Mbps full duplex
Sep  1 16:08:54 truenas kernel: ixgbe 0000:41:00.0: removed PHC on enp65s0f0
Sep  1 16:08:55 truenas kernel: bond10: (slave enp65s0f0): link status definitely down, disabling slave
Sep  1 16:08:55 truenas kernel: bond10: active interface up!
Sep  1 16:08:55 truenas kernel: ixgbe 0000:41:00.0: registered PHC device on enp65s0f0
Sep  1 16:08:55 truenas kernel: bond10: (slave enp65s0f0): link status definitely down, disabling slave
Sep  1 16:08:55 truenas kernel: ixgbe 0000:41:00.1: removed PHC on enp65s0f1
Sep  1 16:08:55 truenas kernel: ixgbe 0000:41:00.0 enp65s0f0: detected SFP+: 3
Sep  1 16:08:55 truenas kernel: bond10: (slave enp65s0f1): link status definitely down, disabling slave
Sep  1 16:08:55 truenas kernel: bond10: now running without any active interface!
Sep  1 16:08:55 truenas kernel: ixgbe 0000:41:00.0 enp65s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Sep  1 16:08:55 truenas kernel: ixgbe 0000:41:00.1: registered PHC device on enp65s0f1
Sep  1 16:08:55 truenas kernel: bond10: (slave enp65s0f0): link status definitely up, 10000 Mbps full duplex
Sep  1 16:08:55 truenas kernel: bond10: (slave enp65s0f1): link status definitely down, disabling slave
Sep  1 16:08:55 truenas kernel: bond10: active interface up!
Sep  1 16:08:56 truenas kernel: ixgbe 0000:41:00.1 enp65s0f1: detected SFP+: 4
Sep  1 16:08:56 truenas kernel: bond10 (unregistering): (slave enp65s0f0): Removing an active aggregator
Sep  1 16:08:56 truenas kernel: bond10 (unregistering): (slave enp65s0f0): Releasing backup interface
Sep  1 16:08:56 truenas kernel: ixgbe 0000:41:00.0: removed PHC on enp65s0f0
Sep  1 16:08:56 truenas kernel: ixgbe 0000:41:00.1 enp65s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Sep  1 16:08:56 truenas kernel: bond10 (unregistering): (slave enp65s0f1): Releasing backup interface
Sep  1 16:08:56 truenas kernel: ixgbe 0000:41:00.1: removed PHC on enp65s0f1
Sep  1 16:08:56 truenas kernel: bond10 (unregistering): Released all slaves
Sep  1 16:08:58 truenas kernel: tg3 0000:46:00.1 eno2: Link is up at 1000 Mbps, full duplex
Sep  1 16:08:58 truenas kernel: tg3 0000:46:00.1 eno2: Flow control is off for TX and off for RX
Sep  1 16:08:58 truenas kernel: tg3 0000:46:00.1 eno2: EEE is disabled

Captain_Morgan · September 2, 2024, 4:41am

Its best that no Ip addresses on interfaces initially. follow these instructions.

Its much easier if the webUI is going through a separate port… not the bonded interface.

tushar · September 2, 2024, 5:29am

I tried with and without connecting the interfaces to switch. My GUI is on eno2 and the bonded interfaces are unused for the entire setup. The link aggregation still doesn’t work. I followed the post you shared but I am not sure what I am missing. Can you point me to specific section I am doing wrong?

Captain_Morgan · September 2, 2024, 9:00am

isn’t eno2 one of the interfaces for the bond?

tushar · September 2, 2024, 11:21am

Nope, bond is on enp65s0f0 and enp65s0f1.

I only enabled eno2 for logging into GUI outside of bond interfaces.

Captain_Morgan · September 2, 2024, 3:30pm

The GUI should not be effected… by the bond. Unless its getting an IP address on the same subnet as the bond.

Suggest not using DHCP and documenting your IP addressing.

tushar · September 2, 2024, 7:04pm

Ip addresses are in one of the screenshots and they don’t change. Here they are
eno2: 10.0.0.53
enp65s0f0: 10.0.0.48
enp65s0f1: 10.0.0.49

I tried static ip but didn’t work.

I’m confused about what you said about ip being in same subnet. Can you please explain?

Captain_Morgan · September 2, 2024, 7:35pm

You can’t have Multiple interfaces on the same IP subnet… TrueNAS is confused about which interface to use outbound.

Whiskydrinker · September 3, 2024, 5:18pm

Are these DHCP assigned IPs on your interfaces or static ones?

Also is your switch really capable of layer 2 and 3 hashing? Because my rather cheap HP switches at home only support layer 2. And when I had 2 and 3 enabled in Truenas my bond was behaving weirdly when setting it up until I limited Truenas to Layer 2 only.

tushar · September 3, 2024, 5:40pm

You can’t have Multiple interfaces on the same IP subnet… TrueNAS is confused about which interface to use outbound.

I am not sure how to fix this without creating VLANs. But I haven’t read about this in the documentation you posted in the first message either. My question is - creating LACP bond will create one IP for both interfaces combined right? How would that work when interfaces are on separate subnets? Why is creating LAG1 group on my switch not sufficient? Also why does this restriction exist when Synology is able to create LACP bonds in same subnet (just an example)?

Are these DHCP assigned IPs on your interfaces or static ones?

Right now they are assigned via DHCP. But tried using static IP as well. Didn’t work

Also is your switch really capable of layer 2 and 3 hashing?

Pretty sure it is. I am using TL-SX3008F | JetStream 8-Port 10GE SFP+ L2+ Managed Switch | TP-Link It has L2 and L3 features.
I haven’t tried TrueNAS LACP setup restricted to L2. I’ll give it a try.

Captain_Morgan · September 3, 2024, 9:11pm

Your TCP UI session will be diverted as you create the bond.

So you need to separate out your management and data vlans.

Whiskydrinker · September 4, 2024, 6:23pm

Now we’re getting closer to the problem. There shouldn’t be any settings for an ethernet port at all that you want to include in in bond group.
That your wanted members get an IP shows that there’s something completely wrong wrong with you config.

So in order to get a working bond:

Go into the settings of enp65s0f0 and clean them completely out. They should look like for one of my bond members.

Bildschirmfoto 2024-09-04 um 19.44.47944×1540 39.8 KB

When they have been cleaned up hit the save button.
Do the same for enp65s0f1.
Now that you have clean interfaces you can join them for the bond. Hit the add button next to interfaces.

Type: Link Aggregation
Name: Something like bond0
Check DHCP and Autoconfigure IPv6 if you want automatically configured IPs.
Link Aggregation Protocol: LACP
Transmit Hash Policy: Layer2 only since the data sheet of your switch mentions this only under layer 2.
LACPDU Rate: SLOW
Link Aggregation Interfaces: Now select enp65s0f0 and enp65s0f1 as members there.
Aliases: If you haven’t checked an automatic configuration for Ipv4 or IPv6 earlier you can set fixed IP addresses there.

Hit the save button.
Now also empty the settings for eno2 completely and hit save.
If you have DHCP configured it’s now time to open your DHPC server’s client table.
Click the “test settings” in Truenas and you should see another IPv4 getting assigned in your DHCP’s client table.
Try to open this IP (or the fixed one if you have done this) in your browser and after logging in you should see a working bond.

It’s been quite a while since I’ve set up mine so it’s possible that there’s also a final apply settings button. If yes click it.