Slow SMB Slow Read Performance - ConnectX-5 Mellanox Card Q's

TrueNAS General
, ,
1

(update - I think I nailed it down to 25gb card - see below)

Hey All, for the first time that I can remember…I’m a little stumped about getting some slow SMB speeds off a fresh bare metal install. Is anything jumping out to you all? I’m remotely troubleshooting this one - right now this Mac is the only 10gbe connected device but I should have a 10gbe Windows machine online in a little bit to test. I know Macs can be a little funky - but never really had any issues until now! As always - appreciate any insight on this one!

System & Network Specs

  • TrueNAS SCALE

    • Hardware:
      • AMD EPYC 8324P (32 cores)
      • ~250 GiB ECC RAM
    • Version: ElectricEel‐24.10.2
    • Storage Pool:
      • Layout: 3× RAIDZ1 VDEVs (15 spinning disks total)
      • ~196 TiB usable; ~36 TiB used
    • Network:
      • NIC: 25 GbE - connected via DAC to Switch
      • MTU: 9000 (Jumbo Frames)
      • Verified link speed: 25 Gb/s full duplex (cat /sys/class/net/.../speed)
      • 01:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
        Subsystem: Mellanox Technologies ConnectX-5 EN network interface card, 10/25GbE dual-port SFP28, PCIe3.0 x8, tall bracket ; MCX512A-ACAT [15b3:0020]
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core

  • Switch:

    • Ubiquiti UniFi USW Enterprise XG 24
      • 25 Gb SFP28 port → TrueNAS
      • 10 Gbe → Mac
      • Jumbo Frames, Flow Control toggled on/off in different tests
      • Firmware version not explicitly stated, but reboots performed

  • Mac Client - Mac Studio M1 Ultra w/ Sonoma MacOS 15.1

    • NIC: 10 GbE, manually set to 10Gbase‐T, full duplex, MTU 9000
    • Flow Control: auto‐negotiated on the Mac, shown in ifconfig
    • SMB signing: disabled via /etc/nsmb.conf
    • macOS: Using AJA System Test, iperf3, etc. for throughput tests

Pool and Datasets Overview

  • Pool Name: tank
    • Layout: 3 × RAIDZ1 VDEVs (5 disks each, total 15 HDD)
    • Usable Capacity: ~196 TiB
    • Used: ~36.19 TiB (~18.5%)
    • Encryption: Unencrypted
    • ZFS Dedupe: OFF

Performance Tests

iPerf3 Single‐Stream

  • Forward (Mac → TrueNAS): ~6–7 Gbits/s
  • Reverse (TrueNAS → Mac): ~2–3 Gbits/s initially, with occasional stalls or heavy retransmissions

iPerf3 Multi‐Stream (-P 4)

  • Forward: Often ~9.7–9.8 Gbits/s (close to saturating the Mac’s 10 Gb link)
  • Reverse: Ranged from stalling at 2–3 Gbits/s to ~6–7 Gbits/s, but with thousands of TCP retransmissions. Reboots and toggling settings sometimes improved or worsened it.

Jumbo Frames & Flow Control Toggling

  • Jumbo On, Flow Control On: Forward ~9.7 Gb/s, reverse stalling or ~2–3 Gb/s
  • Jumbo Off (MTU=1500), Flow Control Off: Reverse ~3.9 Gb/s
  • Re‐enabled Jumbo: Reverse improved to ~6–6.5 Gb/s but still with high retransmissions
  • In all scenarios, forward direction is near 9–10 Gb/s, reverse direction is lower

SMB / AJA Tests (macOS)

  • AJA System Test saw ~500 MB/s writes, ~300 MB/s reads to SMB share.
    Write speed (to me) seems expected - but I def should be getting full 10gbe speeds with the setup - I’ve set this up a quite a few times like this before (same exact systems as well) with no issues.
2

UPDATE - I was able to bypass the 25gbe connection and just to 10gbe from the Mobo to the switch - then to the mac in question - and everything works as expected. Is there anything funky with the following Mellanox card? Pretty sure I have a few of these in production and working just fine as is (never had to update firmware or drivers)

01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
        Subsystem: Mellanox Technologies ConnectX-5 EN network interface card, 10/25GbE dual-port SFP28, PCIe3.0 x8, tall bracket ; MCX512A-ACAT [15b3:0020]
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core
01:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
        Subsystem: Mellanox Technologies ConnectX-5 EN network interface card, 10/25GbE dual-port SFP28, PCIe3.0 x8, tall bracket ; MCX512A-ACAT [15b3:0020]
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core
3

Ok - well…looks like I had a rogue VM wreaking havoc on the NIC for some reason - removed the VM and everything is back to normal. Sheesh!