Mellanox ConnectX-4 EN 100GbE network card

TrueNAS General
1

Bug Report Summary for TrueNAS SCALE

Title: Total Ingress Packet Loss on Mellanox ConnectX-4 (40GbE) due to rx_steer_missed_packets Regression

System Information:

  • Product: TrueNAS SCALE 25.04.2.4
  • Kernel Version: 6.12.15-production+truenas
  • Hardware: Mellanox ConnectX-4 EN (OEM, PSID: MT_2150110033)
  • Firmware: 12.28.4704
  • Link Speed: 40Gb/s
  • **Dual 100Gbe Card

Problem Description: A Mellanox ConnectX-4 EN card experiences total ingress packet loss when operating at 40GbE. The ethtool -S counters show that packets are received at the physical layer (rx_packets_phy) but are immediately dropped by the hardware, as indicated by the rx_steer_missed_packets counter incrementing in lockstep. The OS-level rx_packets counter remains at 0. The kernel log (dmesg) shows a clean driver initialization with no errors.

Key Diagnostic Finding: This is definitively a software regression bug in the mlx5_core driver. The exact same hardware configuration (card, transceiver, cable, switch) functions perfectly in a Proxmox environment running a newer 6.14.8 kernel, ruling out any hardware or firmware fault.

Failed Mitigation and Diagnostic Attempts: All standard Linux workarounds and diagnostic methods for this type of issue have failed, indicating the kernel is compiled without the necessary features to address this at runtime.

  1. devlink Steering Mode Change: Attempting to switch from the default dmfs to smfs fails.
  • devlink dev param set pci/... name flow_steering_mode value smfs cmode runtime
  • Result: Error: mlx5_core: Software managed steering is not supported by current device.
  1. tc Hardware Offload: Attempts to manually program a hardware steering rule fail.
  • tc filter add... matchall skip_sw
  • Result: RTNETLINK answers: Operation not supported
  • tc filter add... u32 skip_sw
  • Result: The command is accepted, but tc -s filter show reveals the rule is not_in_hw, proving the offload silently failed.
  1. Driver Debug Logging: Attempting to enable verbose driver logging via a loader tunable fails.
  • Setting loader tunable mlx5_core.debug_mask=1
  • Result: Error on boot indicating Sysctl 'mlx5_core.debug_mask' does not exist in kernel.

Conclusion: The mlx5_core driver in this version of TrueNAS SCALE has a bug that prevents it from correctly programming the hardware receive steering table. The kernel is compiled in a way that disables all known workarounds (alternate steering modes, TC hardware offload) and prevents the collection of advanced diagnostic logs. Resolution requires a patched driver from the TrueNAS development team.

(more detailed)

Technical Report for Advanced Kernel/Driver Analysis

Title: Analysis of rx_steer_missed_packets on Mellanox ConnectX-4 EN (OEM) at 40GbE Link Speed on TrueNAS SCALE (Kernel 6.12.15)

1. Executive Summary:
An OEM Mellanox ConnectX-4 EN 100GbE network card (PSID: MT_2150110033) is experiencing total ingress packet loss when installed in a server running TrueNAS SCALE (25.04.2.4, Kernel 6.12.15-production+truenas). The card is physically connected to a 40GbE switch port using a 40GbE transceiver, and the link successfully negotiates and establishes at 40Gb/s. Despite a clean kernel log and a link-up state, all incoming packets are dropped by the hardware before reaching the OS networking stack. The issue is identified by the rx_steer_missed_packets counter in ethtool, which increments in lockstep with the rx_packets_phy counter, while the OS-level rx_packets counter remains at zero. The exact same hardware configuration (card, transceiver, switch) functions perfectly on a Proxmox server running a newer Linux Kernel (6.14.8-2-pve), proving the hardware is not faulty. The root cause is hypothesized to be a specific, silent regression bug within the mlx5_core driver in the 6.12 kernel that is triggered by the combination of this OEM card’s firmware and the 40GbE link speed. Standard mitigation attempts, including firmware updates (to the latest OEM version) and manually forcing link speed, have failed.

2. System Configuration & Comparative Analysis:

Component Failing System (TrueNAS SCALE) Working System (Proxmox)
Operating System TrueNAS SCALE 25.04.2.4 Proxmox VE 8.x
Linux Kernel 6.12.15-production+truenas 6.14.8-2-pve
Network Card Mellanox ConnectX-4 EN (MCX416A-CCA) Mellanox ConnectX-4 EN (MCX416A-CCA)
Card PSID MT_2150110033 MT_2150110033
Card Firmware 12.28.4704 (Latest OEM) 12.25.1020 (Older OEM)
Interface Name ens6f0np0 ens2f0np0
Driver mlx5_core mlx5_core
Driver Version 6.12.15-production+truenas 6.14.8-2-pve
Transceiver 40GbE QSFP+ 40GbE QSFP+
Switch 40GbE Capable Switch 40GbE Capable Switch
Negotiated Speed 40000Mb/s 40000Mb/s

3. Key Diagnostic Data & Observations:

  • Primary Symptom: ping fails (no response). No ingress traffic of any kind is processed. Egress traffic (tx_packets) increments normally.

  • Critical Counter Evidence (TrueNAS):

    • rx_packets: 0

    • rx_steer_missed_packets: >0 (and increments with traffic)

    • rx_packets_phy: >0 (and increments with traffic)

    • rx_steer_missed_packets value is nearly identical to rx_packets_phy.

  • Kernel Log (dmesg on TrueNAS): The mlx5_core driver initializes the card without any errors or warnings. It correctly identifies the firmware, PCIe bandwidth, and reports “Link up”.

  • Link State (ethtool on TrueNAS): The link is correctly detected at Speed: 40000Mb/s with Link detected: yes.

  • Firmware Status: The card is an OEM model, restricted to the 12.x firmware branch. The latest available firmware (12.28.4704) has been successfully applied, but this did not resolve the issue.

  • Comparative Success: The Proxmox system, with an identical hardware setup (but a newer kernel), functions perfectly. This is the definitive control case, isolating the problem to the TrueNAS kernel/driver environment.

4. Failed Mitigation Attempts:

  1. Firmware Update: Updated card from 12.25.1020 to 12.28.4704. No change in behavior.

  2. Manual Link Speed Configuration: Using ethtool -s ens6f0np0 autoneg off speed 40000 successfully sets the link but does not resolve the packet drop issue. This suggests the bug occurs during the driver’s initial resource allocation at probe time, not during the negotiation phase.

  3. MFT Tools on Host: mlxconfig confirms the card is an Ethernet-native “EN” model and does not have a LINK_TYPE parameter.

2

Use the GUI to Report a Bug, smile icon on upper right or the upper right of the Forum, Report a Bug

1 Like
4

I did just re-submit with the above details since this technically is a new issue.

Most issues I post that way are technical and specific enough that the devs don’t want to deal with it. I’m hoping to find some people who share connectx woes to see if we can resolve this in kernel. I’m to the point I’m considering tearing out the network stack. We need bonded 100G links.

5

Please post a link to the JIRA bug ticket here. The RC1 of 25.10 is out now and we are expecting the full release later this month. I don’t know if the driver version is the same. The next version, 26.04 was just updated to the next version of Debian (Trixie). 26.04 is nightly development and due around April, 2026

6

I appreciate it, I considered moving to an RC/nightly but we use active directory and the latest version of debian has a bug relating to kerberos, so debian itself is kind of bricked too - but that’s not an us problem. I’m working on formulating that bug report with a solution once the code is done.

I cant post links on here, so here is the ticket number.
NAS-137966

7

Adding a link
https://ixsystems.atlassian.net/browse/NAS-137966

8

The ticket was just closed. Apparently 100 gig networking is outside of what they support… As expected, they dont want to deal with it. “Sorry but this is outside the scope of what we support. Suggest reaching out to our forums to see if someone else has had similar hardware and issues and if they came to a solution.”

They would rather the community fix complex issues for them. Not trying to gripe, but mellanox card support is important to a lot of people both on community and paid.

1 Like
9

100G should be more relevant for Enterprise than CE, but if TrueNAS does not use Mellanox in its hardware there’s zero incentive to investigate. So it seems you’re out of luck. :roll_eyes:

10

They are the most prevalent high bandwidth cards on the market and are extremely afforable and easy to work with. I see no reason for home users not to use them. You can set up a dual link for a total of less than $200.

11

I know. The question is: Why would a “home user” bother with a 100G link?
Setting up a pool which can sustain the throughput will NOT be a $200 exercise…

I do not mean to say that you should be left on your own troubleshooting your issue and the possible regression; I’m saying that, unfortunately, you may well be on your own here.

1 Like
12

For reference TrueNAS-25.04.2.1 and TrueNAS-25.04.2.4 both do work with 40gbe and ConnectX-3 (MLX4_Core). So it seems to be limited to MLX5_Core

=== ens7 ===
driver=mlx4_en
ens7             UP             <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP>
ethtool: driver: mlx4_en
ethtool: version: 4.0-0
ethtool: firmware-version: 2.42.5000
ethtool: expansion-rom-version:
ethtool: bus-info: 0000:c3:00.0
ethtool: supports-statistics: yes
ethtool: supports-test: yes
ethtool: supports-eeprom-access: no
ethtool: supports-register-dump: no
ethtool: supports-priv-flags: yes
modinfo mlx4_en:
    filename:       /lib/modules/6.12.15-production+truenas/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
    version:        4.0-0
    license:        Dual BSD/GPL
    description:    Mellanox ConnectX HCA Ethernet driver
    author:         Liran Liss, Yevgeny Petrilin
    srcversion:     9387ACE7DA21292EABF96A2
    alias:          auxiliary:mlx4_core.eth
    depends:        mlx4_core
    intree:         Y
    name:           mlx4_en
    retpoline:      Y
    vermagic:       6.12.15-production+truenas SMP preempt mod_unload modversions
    parm:           udp_rss:Enable RSS for incoming UDP traffic or disabled (0) (uint)
    parm:           pfctx:Priority based Flow Control policy on TX[7:0]. Per priority bit mask (uint)
    parm:           pfcrx:Priority based Flow Control policy on RX[7:0]. Per priority bit mask (uint)
    parm:           inline_thold:Threshold for using inline data (range: 17-104, default: 104) (uint)

On the side… this is really worrisome that a slightly too nice of NIC would mean they won’t even look into it. ConnectX 3 is obsolete. CX4 should be almost a minimum. And 100G is 4 generations behind top of the line. You can get 100G switches for under $1,000.

Ubuntu is starting to look pretty attractive. You can login using your AD credentials for free, no SSO tax. You can use RDMA for free.

1 Like
13

It would be pretty easy to at least fill arc cache, and a lot of commodity servers have a lot of ram. 256gb at 25 gigabytes per second is pretty epic until disks throttle things.

I get your point, it is a pretty low level issue… Im not opposed to fixing it myself since I have other systems this works on so I should be able to compare and identify the actual problem.

14

Debian supports this natively, so its just a truenas network stack mod I think causing this. At this point ill start looking into /why/ this issue exists and will report back.

Im kind of sad Im not seeing Me Too posts, I was hoping to have some test subjects for fixes. Lol

15

Did you mention Debian 12 was working? That may have made the difference in the ticket getting glossed over and closed or the issue getting worked on. Maybe TrueNAS sees Mellanox as a Realtek and it’s just very low priority.

16

Greetings guys,

i am also experiencing serious issues with CX455A-ECAT NIC on my truenas box.
It started to happen after i upgraded to 25.04.2.4.

[ 549.075361] [4432]: scst: TM fn 0 (mcmd 0000000052825e22) finished, status -1
[ 549.075365] [4432]: iscsi-scst: iSCSI TM fn 1 finished, status 0, dropped 0
[ 549.075405] [411]: iscsi-scst: iSCSI TM fn 1

[ 549.075548] [411]: scst: Aborted cmd 00000000832e648f finished (tag 2, ref 3)
[ 549.075560] [4432]: scst: TM fn 0 (mcmd 00000000d95594bd) finished, status 0
[ 549.075564] [4432]: iscsi-scst: iSCSI TM fn 1 finished, status 0, dropped 0
[ 549.075619] [411]: iscsi-scst: iSCSI TM fn 1

My ESXi hosts are haning when there is a write operations to the datastores.

Would be great if this issue is fixed in the next truenas releases.

17

Yes, most modern versions of debian work with these cards in pfsense, proxmox and other debian founded systems

18

My particular issue is 100% drop, so if you have a link you can ping, its a different issue. Ill try to help you out on monday though.

1 Like
19

Considering that Goldeye supports 400G and RoCE for enterprise customers, meaning newer Mellanox cards are in the mix, perhaps it’s worthwhile trying out Goldeye RC temporarily to see if the issue goes away?

20

Have you tried replacing the CX4 cards in both devices and testing them?
Also, try running mstconfig -d [pciid] reset to restore the default settings and reboot.

21

At the end, my issue was not with truenas. It was something with crs504 switch. After upgrade from 7.13 to 7.20 due to introduction of QoS, something messed up very badly. I’ve reverted to 7.13 and issue is gone. So you can ignore the errors i pasted.

1 Like