BCM57414 25G NIC limited to 10G — bnxt_en driver not advertising 25G link modes

TrueNAS General
1

Hey all, running into a strange issue with a direct 25G link between my TrueNAS SCALE box (Dell R740XD2) and my workstation (Dell Precision 7920).

Setup:

  • Both machines have Broadcom BCM57414 dual-port 25G SFP28 mezzanine cards
  • Connected via 10Gtek CAB-ZSP/ZSP-P3M 25G SFP28 passive DAC (3m, 28AWG)
  • Same NIC firmware on both: 23.31.18.10
  • TrueNAS SCALE 25.10.0.1 (Goldeye), kernel 6.12.33-production+truenas

Problem:
The link negotiates at 10G instead of 25G. ethtool on TrueNAS shows only 1000baseX/Full and 10000baseCR/Full in Supported link modes — no 25G modes at all. On the Windows side (same card, same firmware), the adapter correctly shows 25G and Speed & Duplex: 25 Gbps Full Duplex.

What I’ve tried:

  • Rolled back NIC firmware from 36.11.73.00 to 23.31.18.10 — no change
  • ethtool --set-fec ens1f1np1 encoding rs — FEC configured but Active: Off
  • ethtool -s ens1f1np1 speed 25000 duplex full autoneg off — netlink error: Invalid argument
  • Checked both ports — same result on both

Key observation:
The TrueNAS bnxt_en driver version is 6.12.33-production+truenas. On Windows, the stock Broadcom driver correctly exposes 25G modes for the identical card. This strongly suggests the TrueNAS-patched bnxt_en driver is missing 25G support for this card.

Has anyone successfully run BCM57414 at 25G on TrueNAS SCALE? Is there a way to load a stock Broadcom bnxt_en driver without breaking the system?

2

Hello.

After upgrading TrueNAS from 25.04.2.3 to 25.04.2.6, both Broadcom BCM57414 10/25GbE interfaces (driver bnxt_en) stopped negotiating correctly at 25Gb.

Symptoms observed:

  • Interfaces failed to establish link on 25Gb DAC connections to Juniper QFX switches.
  • ethtool initially showed:
    • Speed: 10000Mb/s
    • Auto-negotiation: off
    • Advertised link modes: Not reported
    • Link detected: no
  • Manually re-enabling autonegotiation and FEC advertisement improved the interface state, but links still failed to come up:
    • 25G modes became advertised correctly
    • FEC modes (RS / BASER) became visible
    • however ‘Link detected: no’ persisted.
  • Both interfaces were affected simultaneously after the update.
  • No hardware, PCIe, firmware, DAC, or switch errors were reported.
  • Removing interfaces from the LACP bond and restarting switch ports did not resolve the issue.
  • Booting back into the previous Boot Environment (25.04.2.3) immediately restored normal 25Gb link negotiation and connectivity.

This strongly indicates a regression introduced in TrueNAS SCALE 25.04.2.6 affecting Broadcom BCM57414 (bnxt_en) 25Gb link negotiation/FEC handling with DAC connections and/or LACP environments.

We would be interested to know whether anyone else has experienced similar issues with BCM57414 adapters, bnxt_en, 25Gb DAC links, or Juniper QFX environments on recent TrueNAS SCALE releases, especially within the 25.10.x branch.

3

@Nixoid_Tab @jose.domenech You can try using Report a Bug in the TrueNAS GUI (smile icon on top, upper right for Feedback/ Report a Bug). Be sure to attached Debug Dumps and be detailed in anything you have tried.

You can include a link to this forum thread in the report as I think it has some good info on a TrueNAS version that was working. It may give clues to driver changes.

4 Likes
4

Quick update regarding this issue:

I opened a bug report with iXsystems and provided the collected information, debug data, reproducible behavior, and references to this thread. The report was closed with the conclusion that this is “not a TrueNAS bug”, although no technical explanation or additional context was provided.

JIRA reference: NAS-140993
URL: ixsystems.atlassian[.]net/browse/NAS-140993

I would appreciate some clarification on the reasoning behind that conclusion, because the behavior still appears strongly correlated with the upgrade itself and seems fully reproducible:

  • Same hardware

  • Same Broadcom BCM57414 NICs

  • Same DACs

  • Same Juniper QFX switches

  • Same LACP configuration

  • No network changes

  • No hardware changes

  • Only difference: Boot Environment / TrueNAS version

Behavior observed:

  • Boot 25.04.2.3 → interfaces negotiate and operate normally at 25Gb

  • Boot 25.04.2.6 → both interfaces fail simultaneously

  • Roll back to 25.04.2.3 → connectivity immediately restored

We also tried several troubleshooting actions without success:

  • Re-enabled autonegotiation manually

  • Adjusted FEC settings

  • Removed interfaces from the LACP bond

  • Restarted switch ports

  • Reviewed dmesg and journalctl

  • Verified no hardware, PCIe, firmware, or link-related errors

Since both interfaces fail simultaneously and immediately recover after reverting the Boot Environment, it still looks like some change between releases could be involved (kernel, bnxt_en, firmware package, PHY handling, networking stack behavior, etc.), even if the root cause is ultimately upstream.

If anyone has seen similar behavior with BCM57414 adapters, bnxt_en, Juniper QFX, DAC links, or newer TrueNAS versions (including 25.10.x), I would be very interested in hearing about your experience.

At this point, without additional clarification or understanding of the root cause, this issue effectively becomes a blocker for continuing TrueNAS upgrades on systems using BCM57414 adapters, since upgrading currently introduces a reproducible loss of 25Gb connectivity.

I’m also willing to schedule a maintenance window and temporarily boot back into 25.04.2.6 to collect additional diagnostics if someone can suggest specific data that may help narrow this down.