I was a bad boy. When support for Helm Charts was dropped, I stopped updating my TrueNAS Scale deployment. I finally bit the bullet, moved all of my apps off to some other VM’s running somewhere else, and upgraded to Fangtooth v25.04.2.4.
I migrated my Storj APP back off the temporary VM it was running on, and now I see some strange activity. Inbound traffic is using the correct / assigned NIC & IP address, but outbound traffic is going out through a different NIC / IP address.
I’ve got a couple of different NIC’s in my system. Mostly setup to spread the load across 1G NICs, segregate traffic and allow my firewall to do it’s job. All of my APPs are supposed to use VLAN 125 / 10.10.125.120 / eno2.
The other night when I was not exactly clear eyed, I noticed a whole bunch of traffic on my MGMT interface, VLAN 100 / 10.10.100.120 / eno1. The only systems on that VLAN are other servers and possibly my laptop. What in the world is sending 20+ Mb/s out on VLAN 100?
A few tcpdumps later, I realize all of that traffic is going out to a StorJ IP. So I killed the APP, and the traffic stopped! Turn it back on, the traffic starts. positive identification!
I’ve got my app configured to use VLAN 125. The app is listening on and receiving inbound traffic on VLAN 125… but all of the responses are going out VLAN 100.
Am I doing something wrong? Or is this expected behavior?
This is just a wild guess, since nobody else is answering.
Take it with a huge grain of salt.
AFAIK there is no way to tell TrueNAS which interface to use for “internet stuff”.
To decide which interface to use, it looks at your default gateway. If your default gateway is reachable by multiple interfaces, it selects one randomly.
So my guess is that your default gateway is reachable from VLAN 100, so TrueNAS uses that and your NAT points to the interface on VLAN125, so traffic comes in there.
Thanks Sara! That’s kinda what I was thinking, but since it’s new behavior and since I did just jump a couple of major releases I figured it was worth asking!
Once I migrate some of my other apps back, I’ll have to pay attention… and see if it’s affecting all apps or just StorJ!
I noticed this when I set a remote firewall rule to allow incoming traffic only from the IPv6 GUA of VLAN 50. I used that for remote replication. A few weeks later, all of a sudden the replication did not longer work. And I did also not get a mail notification about that error. Why? Well TrueNAS decided to use my management VLAN 51 for “internet stuff”. The problem is, that VLAN is not allowed to any traffic inside or outside.
Make sure you use not the link local IPv6 of your firewall as gateway (since that is the same for mutliple interfaces) but the GUA instead.
If you have multiple VLANs with a default gateway then yes the server will “randomly” select one for internet access. If you want the server to use only one interface for internet, then only one vlan should have internet access. Controlling traffic with the router and/or firewall rules is a reliable method for enforcing network policies. This way the network controls traffic at the source, preventing devices from sending internet bound traffic out on the wrong VLAN. You can do this with things like pfSense, OPNsense, or using a managed smart switch with VLANs.
Networking is not my ball of wax and I actually hate having to do much of anything with networking or why it does not work or work as expected.
No be more precise, not should only one VLAN have internet access, only one VLAN should be able to reach your gateway.
Sorry, but that is not gonna work. See my example above. Yes you can block TrueNAS from reaching the internet over a certain VLAN, but TrueNAS will not fallback because of that. So you need to set it right at the TrueNAS leve
Truenas is an NAS appliance and does not have it’s own firewall like windows or Linux would. Anything happening would need to be at the router level or from a separate piece of software like an app or separate firewall with configurable policies.
For Fallback if desired, Truenas only offers a LAGG which could be configured for fallback only. If configured to combine multipe interfaces into one then the algorithm picks which interface to use for in or out and that may change at any time.
If using IPMI then in the BIOS it should not be attached to one of the normal interfaces and the IPMI interface should be on a separate local only vlan.
I also don’t think there are is any way to set network policies on Truenas.
You should only have one vlan from the router that can access the internet and only those computers needing access to the internet need to be on the vlan for the interface. It can be configured in such a way as to allow the various vlans to route locally only so the second interface of the nas can have local only access.
