I have a system with three interfaces all with static IPs.
I have not defined a default gateway as I would like each interface to have it own gateway defined. However, I can’t find how to set the gateways manually on all the interfaces.
I’m running on TrueNAS Scale Dragonfish-24.04.0.
We will need a bit more information on what your LAN infrastructure is and which NIC(s) are supposed to have connectivity to the internet.
How is that supposed to work? The default gateway is the default for anything not covered by a specific static route - which is what you should probably look at. It’s not clear why you’d want a gateway for SMB or NFS, though.
Might it also be considered a gateway between seperate (LAN) networks as well?
That’s possible, of course, but something that probably deserves some thought as to how appropriate it is.
I’m assuming the complexity is likely not needed and can lead to difficulty in general, so may be best to avoid. There is likely a way to use filtering or something in the router/firewall for some separation.
Thanks for all the questions. The reason I ask is that I’m running pfsense as firewall that are using Interface Bounding State as Firewall state policy. And it seems that not being able to set a gateway for the SMB share interface (vlan 20) makes pfsense to see it as an asymmetric connection, which drops the packages.
So the only way for me to get it to work is to set the interface to get IP and gateway from the DHCP-server.
How would pfSense even know if another host on the network has a gateway set? I don’t think I follow what you’re describing.
I’m not near an expert neither in networking, pfsense or Truenas. However, what I experience is that traffic is dropped (at firewall level) unless my storage vlan (25) has set a gateway IP. For example if I set storage interface’s gateway as the default gateway under general network settings things works fine. Previously I have used the GUI interface as the default gateway.
So my question is if it possible to manually assign gateway IP on the specific storage interface?
If there is anything else that I should consider please share. 
Edited, added console output:
root@nas[~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.15.1 0.0.0.0 UG 0 0 0 eno1
192.168.15.0 0.0.0.0 255.255.255.0 U 0 0 0 eno1
192.168.25.0 0.0.0.0 255.255.255.252 U 0 0 0 enp2s0f0
192.168.26.0 0.0.0.0 255.255.255.248 U 0 0 0 enp2s0f1
192.168.98.0 0.0.0.0 255.255.255.252 U 0 0 0 eno2
root@nas[~]#
TrueNAS does not support per-interface default gateways. Place all clients using the storage and the storage interface in the same VLAN - makes sense, anyway.
Or use NAT so all connections come from the pfSense IP address in that VLAN.
Anything else leads to asymmetric routing and dropped connections.
This definitely sounds like a case of a few too many subnets separated for no good reason.
Okay, thanks for the reply and clarifying that Trueness doesn’t support multiple gateways and the suggestions for a fix.
TrueNAS does not support proper routing IMHO. Multiple gateways should be (!!). Incoming Traffic should be answered via the same vlan / path as it arrived.
Terrible and not secure!!
It routes exactly like every Unix host has ever done since IP was integrated into BSD Unix in the 70s and 80s. Everything about that is “proper” according to all relevant RFCs.
It was perhaps OK in the '70 but despite the RFC’s is is absolutely not OK in my opinion.
My firewall is blocking that kind of traffic, since the outgoing traffic does not match the incoming traffic.
Next to that I explicitly check if the traffic arriving at the firewall has an address matching the (vl)lans subnet.
Note that threats does not necessary come from internet, they can also be initiated by unintended software on one of your own machines
TrueNAS is a storage appliance. It’s networking stack behaves exactly like it is supposed to and as is documented.
Place it in an isolated storage network.
That is exactly one of the problems! With the VM’s and the Kubernetes applications, it is an all in one solution! With multiple tasks and multiple security levels !!!
That is also the reason I have chosen for TrueNAS.
But the storage part is green zone!! Should be highly shielded! The rest is less secure. And given the fact that there is a application or VM containing a threat or a vulnerability, I have my network segmented.
And of course one of my own computers or the computer of a guest / my kids can be infected as well. So not unlikely that there can be threats inside my network.
So my firewall is not only filtering what is entering a network segment, but to a certain extent also outgoing traffic. Towards internet, but surely also towards other vlan’s
Yep, you can. The default gateway = (v)lan in the GUI can be used in favor of:
Since I would like to be sure that I never loose the management, the most logical choose is perhaps to use the default (v)lan for the management and define policy routing for the other (v)lans see my (actual) thread related to “policy routing”
(I think multiple gateways / separating traffic is essential, I know not every body agrees)
Which other product offers this on a single system? VMware ESXi can run multiple IP stacks, but doesn’t do storage at all. I am not aware of anything that behaves as you would prefer.
You can separate apps from storage and management via the K3S network settings. Also on CORE and on SCALE you can isolate VMs and jails by using a bridge interface on which the NAS host does not need to have an IP address at all. And since VMs bring their own OS and network stack, no problem here - complete isolation. I run this at home and at work. It’s no worse that ESXi in that regard, apart from the fact that you need a separate bridge/vSwitch per VLAN.
But management, storage, reporting, … all “NAS services” run on a single host on a single IP stack and cannot be meaningfully separated. That’s just the state of the product and not a bug. You could file a feature request for an isolated management plane.