It got me thinking about the way I’ve got my storage set up in my small home lab/home office setup. I don’t think there’s anything horribly wrong with how I’ve been doing it, but I’m starting to think my early lack of understanding might have led me down the less efficient path.
So, I find myself here asking for a sanity check.
Basic Setup
I’ve currently got an isolated (by firewall) storage network with a single VLAN (Storage VLAN 1).
My TrueNAS server connects to the storage network via a single two-port NIC in LACP mode.
My Proxmox cluster talks to the NAS via the Storage Network. One of my PVE nodes has a single one-port 10 Gbps NIC, and the other uses a single dual-port NIC in LACP mode.
That is, the TrueNAS server and both PVE nodes each have only one IP on the Storage Network.
I have a single set of core and edge switches arranged in a hub-and-spoke configuration. So, I don’t have two parallel networks running in a redundamnt physical setup.
From the Livestream:
My big takeway was that if I create second storage VLAN (Storage VLAN 2), and break all the LACP bonds so that the TrueNAS server and the dual-NIC PVE node have separate NICs accessing the separate storage networks, I can theoretically get much better performance over NVME over TCP and iSCSI in particular on machines with two NICs.
Without doing this, if I remain with the LACP-based setup, I’m effectively going to be capped at 10 Gbps per connection because none of SMB multiichannel, pNFS, or multipath iSCSI/NVMEoF really like LACP at all.
Trying to use any kind of multipathing with LACP will make me very sad.
There’s a question here, I swear:
Everything I have works.
At what point does it become worthwhile to take on the burden of maintaining the more complex dual-storage network setup to get maximum speed between TrueNAS, Proxmox, and whatever other client devices I want to hit TrueNAS with?
I’m not actually saturating any 10 Gbps links yet, and I feel like a year or two ago the meta advice was to stick with 10 Gbps LACP in small home/home office deployments because it was fast enough. (Before the Coming Affordable of 25 GbE in 2025).
Put another way: in y’all’s anecdotal experience, when did it become truly advantageous to move away from a 2x10Gbps LACP-based storage backbone to a multipathed 2x10Gbps pair of storage networks?
I’m just trying to understand, from a real-world perspective outside of large enterprises, when and why it’s worth making the transition. I really want to understand the benefits before I willingly break my working storage network config.
Unless I’m missing something, I real like the closet thing to a right answer for a question with no right answer, in this case, is when I somehow start to really feel like 2x10Gbps LACP connections are a real bottleneck to an actual workload.
Bingo. Unless you’re feeling that a single 10Gbps connection between your host and TrueNAS is a limiting factor, your 2x10Gbps in LACP is likely to remain functional and just fine.
Depending on your switch’s load-balancing behavior you might also get lucky enough to have your two PVE nodes going on different interfaces of the LACP bond to the TrueNAS machine - such that they can each leverage the full capacity of one of those 10Gbps connections between TrueNAS and the switch. You can do a quick fio or iperf test from both nodes at the same time to see if this is the case.
IMO, LACP and multipath are different philosophies of organising the network infrastructure.
LACP is like: we have this many nics, and we use all these nics together. If our network somehow would go misconfigured/hacked/overwhelmed we’ll sink (or swim?) together. Part of the crew, part of the ship! AIUI, you have to connect all the nics to the one switch, making it one big point of failure. Also, AFAIK, expensive branded switches have their proprietary protocols to bring LACP to the multi-switch scenario.
Multipath is more for the scenarios with segregating the nics to different switches, thus providing some kind of HA for the network infrastructure.
My personal observations for the topic:
All small/medium companies I’ve been working at used LACP. With having a separate network only for CEPH. I think the main reason is the ease of management. We finally got a new nic for the server? Good, connect all ports to our switch!
Multipath can work even in fancy set ups.
Multipath sometimes doesn’t work well with some software. For one, MS recommends (or at least used to recommend) using my-smb.my-domain.tld A-record which resolves to 2+ IPs from different subnets. I had such a record for my backup share. On the machine with only 1 NIC windows explorer could connect to this share. However, windows backup&restore couldn’t find it. Because nslookup showed only one IP from the different subnet. I probably somehow messed up DNS. After retaining only the IP from the “main” subnet, all returned back to normal.
It’s fine to use both, but don’t put a multipath-aware protocol on top of an LACP link, because it’ll be inefficient at best and dysfunctional at worst.
iSCSI (MPIO) and NVMe-oF (ANA) know how to manage multiple network paths all on their own - let them do their thing.
(And remember - even when you’re sure it’s not DNS, it’s DNS.)
Does that mean I shouldn’t try to use NVME-over-TCP with a TrueNAS box that uses a 2x10Gbps LACP link? Or can I configure TrueNAS not to attempt multipathing when doing that? (Hopefully the default behavior is to not attempt to multipath.)
It’ll work, but won’t work as quickly as it should, because you’ll only establish a single link to the host over the LACP trunk. Don’t set up two VLAN/virtual IPs on both sides of that LACP link and have them both run at 10Gbps effective throughput; that’s a recipe for out-of-order packet delivery and delayed/retransmitted packets. Storage protocols don’t like that.
NVMe-oF has Asymmetric Namespace Access - think iSCSI’s MPIO+ALUA combo - and it will in fact identify and leverage multiple routes to the same namespace, compacting them into a single NVMe device client-side.
chris@nvme-test:~$ sudo nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1 /dev/ng0n1 a570d4118ed369375abd TRUENAS-[REMOVED] 0x1 5.37 GB / 5.37 GB 512 B + 0 B 25.10.0-
chris@nvme-test:~$ sudo nvme list-subsys
nvme-subsys0 - NQN=nqn.2011-06.com.truenas:uuid:1ae780be-677c-4d3e-ba50-eebc7f57b25d:chrisnvmetcp
hostnqn=nqn.2014-08.org.nvmexpress:uuid:0c718233-269b-4774-9473-2077609013f9
\
+- nvme0 tcp traddr=10.10.10.41,trsvcid=4420,src_addr=10.10.10.129 live
+- nvme1 tcp traddr=10.10.10.42,trsvcid=4420,src_addr=10.10.10.129 live
+- nvme2 tcp traddr=10.20.20.41,trsvcid=4420,src_addr=10.20.20.129 live
+- nvme3 tcp traddr=10.20.20.42,trsvcid=4420,src_addr=10.20.20.129 live
chris@nvme-test:~$ sudo nvme list-subsys /dev/nvme0n1
nvme-subsys0 - NQN=nqn.2011-06.com.truenas:uuid:1ae780be-677c-4d3e-ba50-eebc7f57b25d:chrisnvmetcp
hostnqn=nqn.2014-08.org.nvmexpress:uuid:0c718233-269b-4774-9473-2077609013f9
\
+- nvme0 tcp traddr=10.10.10.41,trsvcid=4420,src_addr=10.10.10.129 live optimized
+- nvme2 tcp traddr=10.20.20.41,trsvcid=4420,src_addr=10.20.20.129 live optimized
chris@nvme-test:~$
In this case, it sees all four paths (two to each controller) and it’s actively using/optimized for the active controller on the .41 IP’s. If I trigger a failover, it jumps to the .42’s on the other.
