BCM57414 25G NIC limited to 10G — bnxt_en driver not advertising 25G link modes

TrueNAS General
1

Hey all, running into a strange issue with a direct 25G link between my TrueNAS SCALE box (Dell R740XD2) and my workstation (Dell Precision 7920).

Setup:

  • Both machines have Broadcom BCM57414 dual-port 25G SFP28 mezzanine cards
  • Connected via 10Gtek CAB-ZSP/ZSP-P3M 25G SFP28 passive DAC (3m, 28AWG)
  • Same NIC firmware on both: 23.31.18.10
  • TrueNAS SCALE 25.10.0.1 (Goldeye), kernel 6.12.33-production+truenas

Problem:
The link negotiates at 10G instead of 25G. ethtool on TrueNAS shows only 1000baseX/Full and 10000baseCR/Full in Supported link modes — no 25G modes at all. On the Windows side (same card, same firmware), the adapter correctly shows 25G and Speed & Duplex: 25 Gbps Full Duplex.

What I’ve tried:

  • Rolled back NIC firmware from 36.11.73.00 to 23.31.18.10 — no change
  • ethtool --set-fec ens1f1np1 encoding rs — FEC configured but Active: Off
  • ethtool -s ens1f1np1 speed 25000 duplex full autoneg off — netlink error: Invalid argument
  • Checked both ports — same result on both

Key observation:
The TrueNAS bnxt_en driver version is 6.12.33-production+truenas. On Windows, the stock Broadcom driver correctly exposes 25G modes for the identical card. This strongly suggests the TrueNAS-patched bnxt_en driver is missing 25G support for this card.

Has anyone successfully run BCM57414 at 25G on TrueNAS SCALE? Is there a way to load a stock Broadcom bnxt_en driver without breaking the system?

2

Hello.

After upgrading TrueNAS from 25.04.2.3 to 25.04.2.6, both Broadcom BCM57414 10/25GbE interfaces (driver bnxt_en) stopped negotiating correctly at 25Gb.

Symptoms observed:

  • Interfaces failed to establish link on 25Gb DAC connections to Juniper QFX switches.
  • ethtool initially showed:
    • Speed: 10000Mb/s
    • Auto-negotiation: off
    • Advertised link modes: Not reported
    • Link detected: no
  • Manually re-enabling autonegotiation and FEC advertisement improved the interface state, but links still failed to come up:
    • 25G modes became advertised correctly
    • FEC modes (RS / BASER) became visible
    • however ‘Link detected: no’ persisted.
  • Both interfaces were affected simultaneously after the update.
  • No hardware, PCIe, firmware, DAC, or switch errors were reported.
  • Removing interfaces from the LACP bond and restarting switch ports did not resolve the issue.
  • Booting back into the previous Boot Environment (25.04.2.3) immediately restored normal 25Gb link negotiation and connectivity.

This strongly indicates a regression introduced in TrueNAS SCALE 25.04.2.6 affecting Broadcom BCM57414 (bnxt_en) 25Gb link negotiation/FEC handling with DAC connections and/or LACP environments.

We would be interested to know whether anyone else has experienced similar issues with BCM57414 adapters, bnxt_en, 25Gb DAC links, or Juniper QFX environments on recent TrueNAS SCALE releases, especially within the 25.10.x branch.

3

@Nixoid_Tab @jose.domenech You can try using Report a Bug in the TrueNAS GUI (smile icon on top, upper right for Feedback/ Report a Bug). Be sure to attached Debug Dumps and be detailed in anything you have tried.

You can include a link to this forum thread in the report as I think it has some good info on a TrueNAS version that was working. It may give clues to driver changes.

4

Quick update regarding this issue:

I opened a bug report with iXsystems and provided the collected information, debug data, reproducible behavior, and references to this thread. The report was closed with the conclusion that this is “not a TrueNAS bug”, although no technical explanation or additional context was provided.

JIRA reference: NAS-140993
URL: ixsystems.atlassian[.]net/browse/NAS-140993

I would appreciate some clarification on the reasoning behind that conclusion, because the behavior still appears strongly correlated with the upgrade itself and seems fully reproducible:

  • Same hardware

  • Same Broadcom BCM57414 NICs

  • Same DACs

  • Same Juniper QFX switches

  • Same LACP configuration

  • No network changes

  • No hardware changes

  • Only difference: Boot Environment / TrueNAS version

Behavior observed:

  • Boot 25.04.2.3 → interfaces negotiate and operate normally at 25Gb

  • Boot 25.04.2.6 → both interfaces fail simultaneously

  • Roll back to 25.04.2.3 → connectivity immediately restored

We also tried several troubleshooting actions without success:

  • Re-enabled autonegotiation manually

  • Adjusted FEC settings

  • Removed interfaces from the LACP bond

  • Restarted switch ports

  • Reviewed dmesg and journalctl

  • Verified no hardware, PCIe, firmware, or link-related errors

Since both interfaces fail simultaneously and immediately recover after reverting the Boot Environment, it still looks like some change between releases could be involved (kernel, bnxt_en, firmware package, PHY handling, networking stack behavior, etc.), even if the root cause is ultimately upstream.

If anyone has seen similar behavior with BCM57414 adapters, bnxt_en, Juniper QFX, DAC links, or newer TrueNAS versions (including 25.10.x), I would be very interested in hearing about your experience.

At this point, without additional clarification or understanding of the root cause, this issue effectively becomes a blocker for continuing TrueNAS upgrades on systems using BCM57414 adapters, since upgrading currently introduces a reproducible loss of 25Gb connectivity.

I’m also willing to schedule a maintenance window and temporarily boot back into 25.04.2.6 to collect additional diagnostics if someone can suggest specific data that may help narrow this down.

5

I downloaded the Linux driver package from Broadcom and viewed the Readme. Wow. there is a lot going on with those chipsets.
I am wondering if you have tried turning off the other speeds using ethtool and trying to get it to use only 25Gbps. There was a mention of both ports needing to use 25 and you couldn’t have a 10G and 25G at the same time. I don’t know how old the notes are or what software versions they are referring to for the driver or ethtool.

Port Speeds
===========

On some dual-port devices, the port speed of each port must be compatible
with the port speed of the other port.  10Gbps and 25Gbps are not compatible
speeds.  For example, if one port is set to 10Gbps and link is up, the other
port cannot be set to 25Gbps.  However, the driver will allow incompatible
speeds to be set on the two ports if link is not up yet.  Subsequent link up
on one port will render the incompatible speed on the other port to become
unsupported.  A console message like this may appear when this scenario
happens:

   bnxt_en 0000:04:00.0 eth0: Link speed 25000 no longer supported

If the link is up on one port, the driver will not allow the other port to
be set to an incompatible speed.  An attempt to do that will result in an
error.  For example, eth0 and eth1 are the 2 ports of the dual-port device,
eth0 is set to 10Gbps and link is up.  There is a lot more info in the readme but below is an example.

   ethtool -s eth1 speed 25000
   Cannot set new settings: Invalid argument
     not setting speed

This operation will only be allowed when the link goes down on eth0 or if
eth0 is brought down using ifconfig/ip.
6

So what is fix for this for those facing similar issue with these broadcom “crappy” 57414 cards?

@Nixoid_Tab @jose.domenech

7
  • Go to the Ask ROVO on your JIRA issue and ask the agent why it closed it. This will give you more information.
  • Was it “Closed without Changes”?
  • Maybe post the link to the JIRA issue here for others to look at.
8

Ticket number was listed in a previous post

9

…and the reason for closing is pretty clear: The issue lies upstream in the Broadcom driver itself, don’t expect TrueNAS to fix it. (And maybe avoid Broadcom NICs.)

10

weirdly, I cannot find the JIRA link, just the issue number you have posted.

Broadcom BCM57414 10/25GbE negotiation problem after upgrade.

11

This does not appear to be an error in TrueNAS code or regression in functionality.

This is not clear. Hence if you ask the AI it will give you a more in-depth reason why it closed the issue, which would throw more light on it’s reasoning.

From my side of things, I am not technical enough to understand the technicals on this discussion but I have been bitten by the TrueNAS AI, hence my suggestion. :smiley:

12

Thanks for the comments.

I understand the point that the underlying problem may ultimately be in the upstream Broadcom bnxt_en driver. However, I don’t think that fully explains the behavior we are seeing.

The key point in my case is that this is reproducible by simply changing the TrueNAS Boot Environment:

  • 25.04.2.3 → BCM57414 links negotiate correctly at 25Gb
  • 25.04.2.6 → both BCM57414 links fail
  • Back to 25.04.2.3 → both links immediately work again at 25Gb

Nothing else changes: same server, NICs, NIC firmware, DACs, Juniper ports, LACP configuration, etc.

So I think there is still an important question to answer: what changed between 25.04.2.3 and 25.04.2.6 that triggers the Broadcom issue?

If exactly the same bnxt_en module is present in both releases, then attributing the regression simply to the Broadcom driver would be particularly difficult to reconcile with these results. In that case we would need to look at other differences such as firmware packages, kernel configuration, module parameters, PHY/FEC initialization, or how TrueNAS configures the interfaces.

I have not yet compared the actual bnxt_en.ko binaries between both Boot Environments. When I have a maintenance window, I can boot both versions and compare at least:

uname -r
modinfo bnxt_en
sha256sum $(modinfo -n bnxt_en)
ethtool -i <interface>
ethtool <interface>
ethtool --show-fec <interface>

If anyone knows whether bnxt_en or the relevant Broadcom firmware/kernel components actually changed between 25.04.2.3 and 25.04.2.6, that information would be very useful.

I am not expecting TrueNAS to fix an upstream Broadcom driver bug if that is ultimately where the defect lies. What I would like to understand is why a working BCM57414 configuration becomes non-working as a direct consequence of moving from one TrueNAS Boot Environment to another, and whether the same issue is present in 25.10.x.

Without understanding that, upgrading remains effectively blocked for our TrueNAS systems using BCM57414 adapters.

13

I think this comes down to TrueNAS Company can’t put resources toward an issue like this for hardware that isn’t considered Enterprise. TruenNAS pretty much follows Debian for releases so you might want to search to see if Debian or any other Linux users are having the issue also.

Have you tried 25.10 series of TrueNAS or the 26 Beta 3 to see if the issue is still there? If this issue happens with another Linux and the driver, it may get better action from reporting it there, instead.

14

Could you please upload a debug in the failed state on .6? The debug you have from .3 is helpful, but the other would be instructive. Also, have you tried this with a cold reboot vs. warm reboot? A warm reboot would hold the adapter’s state, so the outcome may be different. Among other things, it would force a clean link renegotiation.

15

I would just like to add that I have a Supermicro MBD-H12SSL-NT with duel onboard BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01) as LACP lagg. Although both negotiates 10G, for the last few major release upgrades (currently on 25.10.7 - Goldeye) the server throws the following, that appears to cause a brief network outage (ssh terminal pauses during the event).

[1570439.919737] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:02a0:98ff:fe12:e06d error=-5
[1570439.930602] bnxt_en 0000:46:00.1: QPLIB: cmdq[0x1609]=0x11 status 0x1
[1570439.930941] bnxt_en 0000:46:00.1 bnxt_re1: Failed to add GID: 0xfffffffb
[1570439.931276] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=2
[1570439.931613] __ib_cache_gid_add: unable to add gid fe80:0000:0000:0000:02a0:98ff:fe12:e06d error=-5
[1570439.932050] bnxt_en 0000:46:00.1: QPLIB: cmdq[0x160a]=0x11 status 0x1
[1570439.932388] bnxt_en 0000:46:00.1 bnxt_re1: Failed to add GID: 0xfffffffb
[1570439.932725] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=2
[1570439.933179] bnxt_en 0000:46:00.1: QPLIB: cmdq[0x160b]=0x11 status 0x1
[1570439.933525] bnxt_en 0000:46:00.1 bnxt_re1: Failed to add GID: 0xfffffffb
[1570439.933870] infiniband bnxt_re1: add_roce_gid GID add failed port=1 index=2

I’ve upgraded the M/B firmware to the latest available on the vendor site with no resolution.

I’m providing additional info as it may be related.

Regards