TrueNAS issues using jumbo frames :)

TrueNAS General
,
1

My backbone network is 10GB using 9000b jumbo frames. Today I tried to change my TrueNas scale (ElectricEel-24.10.2.1) settings to optimal support that. … I did not manage.

The NAS is connected to the network using a ConnectX4-card (one port used with a 10G SFP+). That 10Gbe port is used as a trunk towards the network and is carrying all the VLAN’s.

My idea is to leave the mtu of all low traffic vlans at 1500 and the vlans carrying heavy load to 9000.

MTU’s are a bit vague term. It can refer to the content and e.g to the L2-frame.
The L2-mtu should be bigger than the content MTU since it has overhead like vlan tags and more.

So for the L2-MTU you see values like 9014 (default connectx4) and higher values e.g. 9200.

And I do not know if the value as to be entered at the interface level should have the content value (9000) or the L2-value (higher 9014??).

Whatever I decide to define 9014 for the trunk. No problem. System still running and accessible.

Than I did go to the TrueNas base VLAN, I mean the VLAN having the core NAS stuff: default route, the gui, the SMB and iSCSI etc.

That is the vlan carrying the heavy traffic, so I changed the MTU to 9000 since I regard the vlan as the endpoint (I am not using a bridge here, perhaps I will try that as well later)

The result … the GUI was not accessible any more. So I had a big big issue.

After some time I did discover that the interface settings are changeable via a bootmenu option. With that option I could make the system accessible :frowning:
and I could try other values for the vlan MTU.

The result of those tests where … simple … all MTU values <> 1500 do make the GUI-inaccessable …

So I really wonder how to use jumbo frames withour loosing the GUI !! :rage:

Note here a related but other issue. There is only one routing table with one default route. Where I really, really would love a default route per vlan (vlans as related to the NAS-core. I am also using vlans’s for my VM’s they are routing per vlan like it IMHO should)

2

My friend kept bugging me about MTU’s and jumbo frames when I started moving to 10g fiber and sfp’s. I found they were more trouble than they help, only maybe a 3-5% boost at best. My opinion, for what it’s worth.

3

I think it will bring something. There are multiple places where a bigger frame helps:

  • transmitting the data at the nas
  • more efficient data transport
  • far less frames to analyze in the firewall (significant)
  • less frames to receive at the destination

I do not know the total advantage. I will measure when I have it working. But of course an important platform like TrueNas should and I hope does support it

1 Like
4

Nowadays 10 Gbit shouldn’t really need Jumbo frames. Both the Linux Kernel and CPU performance have evolved a lot in the two decades since it was introduced.

In terms of raw throughput you’re saving under 5% of overhead.

In terms of packets per second:
Are those packets even transitioning through a firewall? Why would you put your Storage and high performance client devices on different VLANS? Because low performance clients wont need or support Jumboframes and high performance clients should be in the same L2 storage network.

If only the WebUi breaks: Any environment that would benefit from jumboframes should be equipped to have a separate storage and management network. So leave the WebUi on an interface with 1500 MTU and bind SMB/NFS/iSCSI to the 9000 MTU one.

1 Like
5

I completely agree with your saying that the webui should be on a different interface, but what is a different interface. Is that necessarily a different physical interface or … just another vlan? (I prefer the latter).

So what I prefer and also have are multiple vlans. One of them is the management lan having gateway-1

And related all vlans related to different traffic streams have their own gateway in the firewall…

Back to the other remark that the traffic should be in the same interface. No!

The NAS is in my ‘GreenZone’ the systems using its data are not. So there is a FW in between.

Next to the problem about setting up jumbo frames, on NAS-functionallity level there is only one gateway. Where I would like at least two gateways !!

  • one for the management GUI
  • one for the data SMB/NFS/iSCSI

However as far as I can see that is not possible :grimacing:

Writing this (routing problem) I am not sure any more. I did define it this way years ago. For my VM’s I do have a gateway per vlan. I will try tomorrow if I can split the data streams!

6

I spend a couple of hours this morning trying to find a way to define two working vlan’s:

  1. one for management
  2. one for data

And despite trying a lot, I did not manage. I really hope I am wrong but as far as I can see it is still not possible.

The problem is that both use the same gateway being the default gateway and that does not work :grimacing:

7

I’ve been using MTU 9000 on TrueNAS for last 12 months. 9014 on Windows.

Not a single issue whatsoever.