We have TrueNAS Core up and running (dual controller) with the following details:
Primary ISCSI vlan 10 (10gb)
Secondary ISCSI vlan 20 (10gb)
Management on vlan 30 (1gb)
Connected to a Cisco UCS FI
Nexus 9k as the layer 3 router/switch that owns the vlans
Jumbo frames enabled all the way through and on all vlans, besides management vlan 30
Behavior:
ESXi hosts can use vmping to ping TrueNas with jumbo frames correctly
TrueNAS can ping the other controller IP with jumbo frames correctly
TrueNAS can NOT ping the primary or secondary ISCSI vlan gateway (vlan 10/20) with jumbo frames
These gateways are pingable via jumbo frames by ESXi and our other SANs
My thought is because the default route of the TrueNas is on the management vlan (30) it might be sending this traffic via this port instead of the Primary/Secondary connections. If this is case, should I change the default route to not be the gateway of the management network and instead be the gateway of the Primary ISCSI vlan?
Gateways are only used to send packets to addresses that are not local subnets. There should be no need to specify a gateway on the iSCSI networks, as you donât want to send traffic to a non-local subnet on that connection.
Secondly, when you try to connect to an IP address, the host will determine the interface based on local subnets. If the IP address is on a locally connected subnet, then the NIC that holds an address on that subnet will be used to send the packet (assuming there is a host that answers the ARP for that IP address). Otherwise, the packet will be sent to the default gateway.
So, if the address you are trying to ping (the âprimary or secondary iSCSI VLAN gatewayâ) is not local to a subnet connected to the system, it will try to send via the default gateway, which will fail. If it is local to a subnet connected to the system, you need to check the routing table and see if there is a route for that IP address that isnât correct.
route -n or ip route will tell you where your system thinks it should send packets based on the destination IP address.
That should/would be the expected behavior but I also had to add gateways to be able to reach my ESXi boxesâŚ
eg TNC has an (extra) adapter 10.10.10.100/24, an ESXI host in the same subnet (10.10.10.20) should be reachable just fine.
It is not.
A soon as I add a gw (10.10.10.2 for 10.10.10.0/24) things started workingâŚ
Thatâs weird; having an interface configured with an address of 10.10.10.100/24 should automatically add a route for 10.10.10.0/24 via the interface the address is configured on.
Thatâs not an OS or software thing, itâs just how TCP/IP works ÂŻ\_(ă)_/ÂŻ
You should never have to add a gateway for a network your interface is directly connected to.
You definitely have some sort of broken network configuration, then. I would tear everything down and start over (delete all network config and add each interface one by one).
Other than that, you need to set the metric on all the gateway entries in your routing table so that the actual default route has the lowest metric. This should hopefully prevent any attempt to send packets to non-local subnets out the other interfaces.
Essentially, unless you really do have multiple interfaces that can send packets to the Internet, if route -n returns more than one line with a âDestinationâ of 0.0.0.0, you have a broken network config.
Cant test on my primary box at this point, but Iâll try to spin up up the secondary and provide some output.
I also would have put it down to user error if it did not happen on multiple new installations. O/c all have a similar layout, all connect to ESXi, so maybe the same error for all of them.
But there is not much to do wrong with dedicated vlan interfaces (defined on the switch, not on the box) that are just supposed to carry nfs traffic.
