Poor 100Gb performance, SMB-multichannel and LACP LAGG

TrueNAS General
, ,
1

System:

TrueNAS SCALE 2025.10.3
Supermicro H13SSL-NT (3.8, latest fw) with AMD EPYC 9124 (16-core 3GHz)
256GB DDR5 ECC JEDEC RAM
Intel E810-CQDA2 2x100Gb NIC (latest fw)
LR-Link LRNV9F24 PCIe4.0x16 Retimer adapter
Boot: Seagate 512GB M.2 FireCuda 530
2 x Broadcom LSI 9500-16i HBAs (fw ver 37.00.00.00)
AIC BBPHHD40013B backplane (LSI 35X28)
60 x Seagate Exos X20, 20TB HDDs (6×10raidz2)
2 x Micron 7450 U.3 3TB SSDs for metadata (special vdev, mirror) (latest fw)
1 x 16TB Solidigm D5-P5336 PCIe SSD (it’s a scratch disk for users)
4 x AcBel R1CA2801A PSU

This system is connected to several 1x25Gbps Win11 clients, via a stack of 2 Dell enterprise switches. (Clients use Intel, Mellanox, Chelsio NICs.)

Writes to the NAS seem to work well, approaching the line rate, even with a simple Windows Explorer 1-file copy operation. But in the read direction, I’m stuck at 700 MB/s, and CrystalDiskMark [SEQ1M Q8T1] benchmarks 950 MB/s.

The only thing that helps the reads — and quite drastically — is SMB Multichannel. As soon as it’s on, Windows Explorer manages to run the same copy at ~1200MB/s , a 50% increase. The same CDM test returns 2050-2600 MB/s (depending on the client).

SMB-multichannel is mostly talked as an alternative to LACP — multiple physical connections — and little is talked about how effective it can be with a single link. (Microsoft does mention it.)

Thing is, at least in TrueNAS, SMB-multichannel is incompatible with LACP. iX says it’s “less appropriate” and “not a best practice”, and here flat out “SMB multichannel does not function if you combine NICs into a LAGG.” Indeed, enabling multichannel + LACP in the GUI doesn’t work and I’m back to 700MB/s.

This conflicts with my network topology, where every server is dual-linked to each ‘leg’ of the switch stack with LACP. I could forego that, but I lose the link redundancy, switch redundancy, load balancing across clients, and extra headroom that 2x100Gb links allow. We also know the Intel E810 ain’t the fastest, so it could use the help.

  1. Can someone shed more light on LACP + multichannel? I just saw this user was able to make both work happily with SCALE 24.10. So it does function with CLI commands, just buyer-beware?
  2. Not even FAILOVER LAGG is supported with SMB-multichannel?
  3. I considered forcing both ports to have separate IPs on the same subnet, but after some further reading see it’d be an even worse idea. But I don’t see how setting the 2nd port on a new subnet would make any sense for my network.
  4. If the right answer is “you should be able to get these SMB I/O speeds without needing multichannel”, I’m happy to get into the weeds of the testing and tuning I’ve done.
2

LACP isn’t bandwidth aggregation. A physical packet can’t be multiplexed over multiple interfaces. LACP is statistical round-robin queuing, essentially. When the host wants to output a packet, is hashes fields in the packet to select which NIC will emit the packet. Typically, the hash takes some n-tuple including the source & destination IPs and source & destination ports. This means a flow between a given source & destination will remain pinned to a single NIC, so LACP doesn’t benefit you in most cases. It does help if the host is serving multiple dozens of clients, because their load is spread over both NICs in the LACP LAGG.

SMB multichannel allows a single client to use all its NICs over a single session to a multi-NIC server and will effectively allow for multiples of the client or server NIC bandwidth, whichever is less. It works best if both client and server have multiple NICs, with each set of client/server NICs exist their own respective subnets.

SMB multichannel won’t work over a LAGG, because it sees the LAGG as a single NIC, which is effectively capped at a single LAGG member’s bandwidth.

3

Thanks, Samuel. I do know the basics of LACP and SMB multichannel.

  • We have dozens of clients. Even 3 x 25Gb clients and 4 x 10Gb clients (which is a typical possible load for this system) could bottleneck one 100Gb link. In reality our pool can deliver 5-7GB/s, but also in reality the E810 can’t push 100Gbps.
    Furthermore, LACP will benefit you if one of the links, or one of the switches, goes down.
    These are the reasons why LACP is relevant and beneficial in our case.
  • As I said, SMB multichannel still makes a dramatic difference with one NIC. Of course it will be capped at 25Gbps, and that’s fine by us. I don’t need to spread a single client’s load across >25Gbps or across the 2x100 on the other side of the switch.

The performance I described we got with CrystalDiskMark and Windows Explorer tracks what I see with iperf. iperf3 -R -P1 is stuck at ~8Gbps , ever with different wiring, different server NIC (I tried a Chelsio T62100 in the NAS instead of the E810), 3 different clients & NIC brands, LACP on/off, I even checked the transceiver laser power, server+client CPU utilization… With -R -P4 I get much closer to saturating the link: ~23Gbps, which is “good enough”. SMB multichannel appear to leverage the same benefit that -P4 does.

4

I was able to enable SMB-multichannel over LACP LAGG with the smb_options command, and so far in testing it’s working great. Tested 3 simultaneous clients and was able to get good read speeds without instabilities or hiccups.

I can see how this isn’t a good default for everyone, but if you know your environment… @NickF1227 said about it, “you are doing load balancing at layer 2 and layer 3 at the same time. A side effect of this can be wasted traffic (extra overhead) and out of order packets. I have no doubt it works fine for your use case. But scalability here is a problem.”

To break it down:

  • Whether the LAGG is set to L2+3 or L3+4 can really make a difference. In my case, I need RSS more as an “unlock parallel processing flag” in Windows; I don’t need to spread SMB channels across interfaces. If the LACP hash is calculated based on L2+3, it should “pin” each host’s channels (SMB sessions) to the same path to avoid concerns about out-of-order packets.
    L3+4 should take into consideration the port, which could cause the “double load-balancing” concerns that Nick is warning about.
  • I assume that “scalability here is a problem” refers to how this may not work well if you introduce routing and more complex networks. I do know that a separate, simple, “flat” L2 network dedicated to storage is not unusual in my industry.

So yeah, not a button that any TrueNAS user can just turn on, except it all to work, and not complain in the forums “hey why is my SMB-multichannel + LACP acting weird?”… But perhaps it should be a warning in the GUI instead of flat-out disabled.

5

Yeah, agreed.

In the case of Microsoft Windows, SMB Multichannel is on by default and used automatically when it detects a configuration that is compatible with using it.

With regards to setting up a LAGG- Microsoft specifically states that SMB Multichannel is recommended in combination with NIC teaming (NIC teaming is Microsoft’s terminology for bonding multiple NICs together for either bandwidth aggregation and/or failover). Note that Microsoft allows a few different configurations when setting up NIC teaming, with or without using LACP.

“When SMB is deployed with SMB Multichannel, SMB creates multiple TCP/IP connections for a single session to achieve a better balance across CPU cores and better use of the available bandwidth. NIC Teaming continues to offer the failover capability, which works faster than using SMB Multichannel by itself. We also recommend NIC Teaming because it offers failover capabilities to other workloads that do not rely on SMB, because those workloads cannot benefit from the failover capabilities of SMB Multichannel.” (Deploy SMB Multichannel | Microsoft Learn).

The Microsoft article also has a table that does a good job of summarizing the various options/configurations supported for SMB multichannel. They clearly show that SMB multichannel can be used in multiple configurations (for example, single NIC as long as it supports RSS, as well as multiple NICs in a team with RSS, etc…).

image
image1070×811 64.4 KB