Looking for 10gbe/jumbo frames best practices

I have a large home network with several vlans. I have a PC, and several NAS units on 10gbe connected to a US 16 XG switch, I am also running other Unifi switches all linked by a UDM Pro.

I want to enable jumbo frames but the only devices that would use 10gbe. That would be the PC, the NAS (two synology and one truenas), and eventually a Mac. Everything else will stick with an MTU size of 1500. How is this best accomplished?

I am led to believe that you can’t mix 9000/1500 devices. The PC and Mac would still need access to other devices on the network such as printers and such. I would just skip jumbo frames altogether but my speeds (via iperf3) are much slower with it disabled.

My current assumption (please correct me if I’m off base here) is that I should keep everything as MTU 1500 and create a separate VLAN for the NAS units as well as the PC and mac that access them, using a second nic on each of those devices for that purpose.

pc and mac, in this scenario, only use the NAS vlan for access to storage (or each other) and everything else would be handled on the normal (non jumbo frame) vlan.

Is this the most efficient way of doing this or am I missing something here?

Thanks!

Honestly, the most efficient way to do it may just not to do it at all… The hard drives in the NAS / PC / Mac will probably not sustain such a transfer rate. There is also not that many things that can benefit of such a high bandwidth… As a last point, how much your time worth ? Consider the cost of the setup plus the time to configure it, will you ever recover your investment in time and money by going to 10G on your local network ?

As for dual homing your system, it is possible but is also a frequent cause for many problems.

At the end, if you insist doing it, indeed you should keep your 10G completely separated from the rest with VLANs and dual homing but again, I doubt you will ever get anything good in return for such an effort even if you achieve a technical 10G transfer from RAM to RAM.

2 Likes

Yeah, this really is one of those “big pain, limited gain” things. You need to go through everything and pray that stuff actually works as advertised. At the end of the day, the potential gain is fairly limited, anyway.

2 Likes

You can read thorugh the Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks | TrueNAS Community.
If I remember correclty, there was also a thread about a user’s experience in the painful path of jumbo frames… it should be at least a year old, possibly on CORE.

3 Likes

Can you create a 10Gb storage-only network? Everything on this would use Jumbo Frames. It may not make sense to have two NICs on a laptop?

2 Likes

To the first question: yes you can, but you do not create or manage a network from TN.
To the second question: unless you use adapters, it’s generally not possible or incredibly challenging to have more than one NIC on a laptop.

2 Likes

I am not using a laptop. The PC currently has two 10gbe nic cards so that isn’t an issue

I would run a couple of tests with limited network components and see if it makes a significant difference to have jumbo frames enabled or not. For a NAS with spinning rust and less than 4 Z2 VDEVs, I doubt you’ll saturate the network. If your pool consists of flash drives that can sustain a long workload, then I could see the benefits, potentially, as SMB is single thread and so on.

But in general, it’s like folk who dabble too much with VLANs and then suddenly find themselves locked out of their router (ie that would be me). I’d keep things as simple as possible to avoid potential trouble with devices that are not compatible.

If you still want to go down that path, I’d consider setting up VLANs that are dedicated to high speed use with jumbo frames while leaving a default MTU for the “normal” networks and/or anything related to connecting to the router, the internet, and so on.