My TrueNas system is connected to the network using a trunk containing multiple vlan’s. One of those vlans is for Kubernetes applications.
Data send towards Kubernetes is send via that vlan and answers are expected back via the same vlan.
However that is regretfully not the way TrueNAS behaves. Answers are send back via the default gateway and vlan and that is what I absolutely(!) do not want.
So I need policy based routing.
I tried to define policy based routing, but … I did not yet manage.
TrueNas Scaler is based on Debian Linux. So I looked at sites like
and tried a lot. In general there are a couple of problems:
add an extra routing table for this vlan (add to /etc/iproute2/rt_tables)
(working rules) to that table
like ip route add 192.168.100.0/24 dev vlan100 table applications and
ip route add default via 192.168.100.1 dev vlan100 table applications
(where 192.168.100.0/24 is my application vlan and 100.1 the GW)
make the table and the rules survive network restarts and reboots
(I tried configs in /etc/network/interfaces.d/)
That is the direction which probably (!) should be used, but as said I did not manage.
Hopefully someone has the solution
I managed to implement ^Policy base routing^, so actually traffic arriving from vlan-x for intended for vlan-x is routed via vlan-x using the gateway as related to that vlan.
I did that by adding extra routing tables for the vlan’s as related to:
the management vlan
the kubernetics vlan
and an secondary back-up management vlan
For VM’s this is not necessary, since it works by default as it (IMHO) should
In table ^/etc/iproute2/rt_tables^ you have to add a routing table entry for each routing table you would like to add. In the example below I added a table applications
nano /etc/iproute2/rt_tables
In directory ^/etc/network/if-up.d/^ you have to add a file containing the entry’s as needed in the new defined routing tables.
add eg. a file ^my_routes^
make the file executable ^chmod 751 my_routes^
The content of the file should be like this
#!/bin/sh
#if [ “$IFACE” = “vlan100” ]; then
ip route del default via 192.168.100.1 dev vlan100 table applications
ip route del 192.168.100.0/24 dev vlan100 table applications
ip rule del from 192.168.100.0/24 table applications
ip rule del to 192.168.100.1/32 table applications
ip route add default via 192.168.100.1 dev vlan100 table applications
ip route add 192.168.100.0/24 dev vlan100 table applications
ip rule add from 192.168.100.0/24 table applications
ip rule add to 192.168.100.1/32 table applications
#echo “vlan100 done(ipv4)” >> /tmp/my.log
#fi
Note that:
$IFACE does NOT contain the (by me) expected interface name. Logging shows ^lo^ or ^–all^. So for that reason I did not use the if statements
Also note that boot and also during ^ /etc/init.d/networking restart^ the file is executed twice. For that reason I did add the delete rules to prevent double entry’s
I also added rules for IPV6 like this
ip -6 route del default via aaaa:bbbb:ccccc::dddd::1 dev vlan88 table
ip -6 route del aaaa:bbbb:ccccc::dddd::/64 dev vlan88 table
ip -6 rule del from aaaa:bbbb:ccccc::dddd::/64 table
ip -6 rule del to aaaa:bbbb:ccccc::dddd::1 table
ip -6 route add default via aaaa:bbbb:ccccc::dddd::1 dev vlan88 table
ip -6 route add aaaa:bbbb:ccccc::dddd::/64 dev vlan88 table
ip -6 rule add from aaaa:bbbb:ccccc::dddd::/64 table
ip -6 rule add to aaaa:bbbb:ccccc::dddd::1 table
Notes:
ipv6 support in truenas is … ^limmited^, never the less
the delete trick is not working properly here, I do not know why. There are double routing entry’s. Not OK, but it works
Some useful commands:
ip rule show
ip -6 rule show
ip route show table_name / ip -6 route show table_name
/etc/init.d/networking restart
In the TrueNAS GUI network screen should show all added default routes (IPV4 anhd IPV6)
