Jumbo frames and default route


We have TrueNAS Core up and running (dual controller) with the following details:

Primary ISCSI vlan 10 (10gb)

Secondary ISCSI vlan 20 (10gb)

Management on vlan 30 (1gb)

Connected to a Cisco UCS FI

Nexus 9k as the layer 3 router/switch that owns the vlans

Jumbo frames enabled all the way through and on all vlans, besides management vlan 30


ESXi hosts can use vmping to ping TrueNas with jumbo frames correctly

TrueNAS can ping the other controller IP with jumbo frames correctly

TrueNAS can NOT ping the primary or secondary ISCSI vlan gateway (vlan 10/20) with jumbo frames

These gateways are pingable via jumbo frames by ESXi and our other SANs

My thought is because the default route of the TrueNas is on the management vlan (30) it might be sending this traffic via this port instead of the Primary/Secondary connections. If this is case, should I change the default route to not be the gateway of the management network and instead be the gateway of the Primary ISCSI vlan?

Any other thoughts?


Gateways are only used to send packets to addresses that are not local subnets. There should be no need to specify a gateway on the iSCSI networks, as you don’t want to send traffic to a non-local subnet on that connection.

Secondly, when you try to connect to an IP address, the host will determine the interface based on local subnets. If the IP address is on a locally connected subnet, then the NIC that holds an address on that subnet will be used to send the packet (assuming there is a host that answers the ARP for that IP address). Otherwise, the packet will be sent to the default gateway.

So, if the address you are trying to ping (the “primary or secondary iSCSI VLAN gateway”) is not local to a subnet connected to the system, it will try to send via the default gateway, which will fail. If it is local to a subnet connected to the system, you need to check the routing table and see if there is a route for that IP address that isn’t correct.

route -n or ip route will tell you where your system thinks it should send packets based on the destination IP address.

1 Like

That should/would be the expected behavior but I also had to add gateways to be able to reach my ESXi boxes…

eg TNC has an (extra) adapter, an ESXI host in the same subnet ( should be reachable just fine.
It is not.
A soon as I add a gw ( for things started working…

1 Like

That’s weird; having an interface configured with an address of should automatically add a route for via the interface the address is configured on.

That’s not an OS or software thing, it’s just how TCP/IP works ¯\_(ツ)_/¯

You should never have to add a gateway for a network your interface is directly connected to.

I totally agree.

Nevertheless that’s the way it is (with several installations). Guess how long it took me to try that…

Yeah, I can guess your reaction when you figured it out :smiley:

1 Like

You definitely have some sort of broken network configuration, then. I would tear everything down and start over (delete all network config and add each interface one by one).

Other than that, you need to set the metric on all the gateway entries in your routing table so that the actual default route has the lowest metric. This should hopefully prevent any attempt to send packets to non-local subnets out the other interfaces.

Essentially, unless you really do have multiple interfaces that can send packets to the Internet, if route -n returns more than one line with a “Destination” of, you have a broken network config.

Cant test on my primary box at this point, but I’ll try to spin up up the secondary and provide some output.

I also would have put it down to user error if it did not happen on multiple new installations. O/c all have a similar layout, all connect to ESXi, so maybe the same error for all of them.
But there is not much to do wrong with dedicated vlan interfaces (defined on the switch, not on the box) that are just supposed to carry nfs traffic.