I’ve a question regarding multichannel. As far as i’m aware, SMB uses single process and is a single thread operation. I recently tried Multichannel on SCALE and it kinda worked fine. So, my question is does multichannel combines the port speed or it simply uses 2 threads simultaneously or is it multi thread operation?
Secondly, when doing multichannel setup, will it act as rendundant connection to the server and client? If so, then what is LACP Bond and LAGG? Are those different?
Does the switch also need to have multichannel support? if so, how can i check if my switch supports it? I tested the multichannel using direct cable connection, without involving the switch.
Also, is Multichannel available on TrueNAS Core as well? I couldn’t find the settings for multichannel on TrueNAS Core interface.
LAGG setup in LACP provides redundancy from each port and if setup correctly independent switches but also provides another path for clients to channel through a bit like a load balancer. It’s a common misconception that having two ports on a LAGG using LACP improves a single clients throughput as it doesn’t. It does however provide improved throughput of the TrueNAS as it allows more bandwidth for your clients.
SMB multichannel aims to improve client throughput as it can use multiple network connections in parallel however I have no practical experience with this yet.
So, if i understand correctly, it can be summed up as, use SMB Multichannel if you want to have redundancy+better throughput for a single client and an example could be:
TrueNAS: 1GbEx2
Client: 1GbEx2
Total Throughput= 2GbE (only if your TrueNAS and Client can support that much speeds). As SMB is a single threaded connection, it utilizes only single thread and maxes out the connection, no matter how much the server or client can support speeds or how fast or capable the NIC is. It will just use a single thread and that’s the max it can do. However, the SMB Multichannel can utilize two threads or more (depends on the network configuration) and combines the speed to improve the total throughput. It is faster due to the connection not being single anymore.
Whereas, for the LAGG, you create a bond on the TrueNAS (the server side) and not the client. This ensures, you get a proper throughput on both the client sides and it kinda load balances the connection. An example could be:
TrueNAS: 1GbEx2
Client A: 1GbE
Client B: 1GbE
Total Throughput= 2GbE (only if your TrueNAS and Client can support that much speeds). Here, with LAGG, each client is guaranteed to have 1GbE throughput and 2GbE combined together, and the LAGG is only configured on the server side (TrueNAS)
In addition, you cannot use LAGG and SMB Multichannel together due to both utilizing different protocols.
@Johnny_Fartpants Now my question is what if i don’t create the LAGG on server side and connect the client to the server and do read/write operation from the client simultaneously? Provided that the server has 1GbE and so the clients, what’s the expected speed i’ll possibly get on the client side?
OP, if you have the ability to do SMB multichannel I would just do that and ignore LACP. Bonding connections at the Ethernet level involves multiple touchpoints to manage and maintain and in most cases it’s only advantageous when there’s a fan-out from the server to multiple clients (multiple clients would find themselves pseudo-randomly distributed across the connections).
Apparently there’s a mechanism for connections to be distributed across LACP uplinks based on TCP/UDP port number therefore a single client could leverage the full combined bandwidth. But this opens a new can of worms when we ask how to force SMB into using specific port numbers in both directions. More touchpoints. Just messy…
SMB is a single checkbox that usually “just works” and accelerates any number of clients that can participate; even just one. There’s no technical debt created and no administrative overhead. The switch doesn’t get involved – any $20 unmanaged Aliexpress crapbox works with this. When I experimented with SMB multichannel it kept on trucking when I shut down one of the ports. Bring the port back up and SMB would start using it again.
What I don’t know is how SMB multichannel handles a client with multiple uplinks to the switch (e.g. 2x 1Gb) but the server only has one larger uplink (e.g. 1x 2.5Gb). Or vice-versa. I think SMB multichannel will still accelerate this after it figures out which interfaces can see which interfaces.
When I tried SMB multichannel for the first time I overthought it and furiously Googled how to configure DNS (e.g. multihomed A record or just one?) and if the two links had to be in the same subnet or different subnets. Turns out none of that matters after the initial connection has been made. Just make sure the interfaces that need to see each other indeed have IP connectivity. SMB multichannel, um, finds a way.
The LAG hash when using LACP is negotiated between the host and the switch. You can pick what gets configured by changing the configuration on the host or switch.
cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.6.44-production+truenas
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
This is what my system (TrueNAS MiniXL+) is currently running connected to a Ubiquity Enterprise-48 switch. When you move traffic that single flow will be hashed to a link. So in my 2x10G setup the max bandwidth a single flow could see would be 10G (well, minus the overhead). Ideally host will hash to different links given enough flows. This is my home setup and just supports my desktop and a few systems at 2.5G along with all the media players in the house. All mounts are SMB. I have found this to work very well and be quite stable. I played with multichannel and did not really see much benefit from it. I my case when I am working and the family are streaming 4K videos to different TVs the flows seems to hash in a pretty balanced way. If I pull an interface the flow will move to the other link. Honestly having built large scale production data centers for years as well as having worked at a switch vendor, my answer is if you need more speed get bigger links. The resiliency matters more. If you are smart enough to even have this discussion then setting up a LACP LAG on the switch is within your skill set.
The SMB is a single threaded connection and generally, its recommended to use the highest core clock for a CPU. But does this also applies to SMB Multichannel as well?
Secondly, how many cores does the SMB Multichannel can use for a given setup? Does the number of cores being used depends on the number of ports on the NIC?
