TrueNAS, VLANs and shares

Apps and Virtualization
1

Hi,

For educational purpoese, I am implementing some VLAN segregation in my home network. I have a Unifi network, and my Truenas Scale (ElectricEel-24.10.1) server is, along with everything else, on the default network (VLAN 1). The IP range here is 192.168.32.0/24. On there I also have a VM, hosted in the Truenas, that runs docker containers, and accesses NFS shares on the TrueNAS.

No, to complicate things, I have created a VLAN with ID 100, (IP range 192.168.100.0/24) and setup an Active Directory there.
The VLAN is created in my Unifi UI, and I have a VLAN type NIC setup with a static IP for it on my TrueNAS. At first it was set at DHCP, and it recieved an IP address in the correct subnet.
Now, since this is a separate VLAN I can not access it from my default network, so I setup a jump host, that has a NIC in both networks. That way I can RDP to that from my default network, and access everything in VLAN 100 from there. Everything works as planned, so happy thoughts.
But now I want to move my docker VM into VLAN 100, and access the NFS shares on my TrueNAS from there. But no host in VLAN 100 can ping the TrueNAS servers NIC in VLAN 100. It shows up in my Unifi UI with correct IP, so it seems to have some sort of connection.

Trying to ping a host inside VLAN 100 from the shell of my TrueNAS gives this:

PING 192.168.100.5 (192.168.100.5) 56(84) bytes of data.
From 192.168.100.2 icmp_seq=1 Destination Host Unreachable
From 192.168.100.2 icmp_seq=2 Destination Host Unreachable
From 192.168.100.2 icmp_seq=3 Destination Host Unreachable
From 192.168.100.2 icmp_seq=4 Destination Host Unreachable

192.168.100.2 is the IP of my TrueNAS in VLAN100.

Pinging 192.168.100.2 with 32 bytes of data:
Reply from 192.168.100.5: Destination host unreachable.
Reply from 192.168.100.5: Destination host unreachable.
Reply from 192.168.100.5: Destination host unreachable.
Reply from 192.168.100.5: Destination host unreachable.

All other hosts inside VLAN 100 can ping eachother.

netstat -rn on the TrueNAS gives this:

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.32.1    0.0.0.0         UG        0 0          0 br0
192.168.32.0    0.0.0.0         255.255.255.0   U         0 0          0 br0
192.168.100.0   0.0.0.0         255.255.255.0   U         0 0          0 vlan100

What do I have to do in order to access the NFS shares on my TrueNAS from within VLAN 100? Is it even possible?
It seems there is no gateway for VLAN 100, is there supposed to be?
I have tried disabling all firewall rules that has anything to do with VLAN 100, even though I think that shouldn´t be an issue, since I am within the same subnet. Is there something that needs to be done on the Truenas to make it communicate over the VLAN NIC? What have I missed. Please let me know if any more info is needed.

2

You can only have one default gateway as there is only one router “stack”

Have you bound NFS to the new card?
System/Services/NFS/Edit

Ahh - you can’t ping the host - my bad

3

Exactly. I am stumped, I don´t know what to look for. If anyone knows how to solve something like this, help is very welcome :slight_smile:

4

Aha wait a minute - you docker VM. Is it on the TrueNAS Host?

If it is - it needs a bridge rather than an interface

5

True, but that´s not what I mean. My Docker VM is part of my “old setup”. I have that bridge interface setup, so that the TrueNAS can communicate with the VMs and vice versa. But this is when the VM is outside of my new VLAN (100). Any machine inside that VLAN cannot ping or access the TrueNAS. And that is what I am after.

My Network setup looks like this:

image
image865×321 6.33 KB

eno1 - physical NIC
br0 - Bridge interface for VMs. (old setup).
vlan100 - Interface for VLAN 100.

The vlan100 interface has a static IP set, but even if I set it to DHCP, it gets an IP address in the correct subnet, so it has some connection. But then, from a VLAN in VLAN 100, it seems impossible to ping 192.168.100.2, which is TrueNAS.
If I ssh into my router I can ping the TrueNAS on that IP just fine. And all other devices can ping each other just fine.
All VMs inside of VLAN 100 have internet access.

So the goal is to put all servers inside of VLAN 100, and make the jump host (with a NIC in both networks) the only bridge between my default network and VLAN 100.

6

Or wait, do you mean I need another bridge interface, for the vlan100? If so, can I have several? I would like to test it before I break my old setup… Should it replace the VLAN interface I created?

7

I made a small discovery. If I try to access the internet from the VLAN100 interface on my TrueNAS (by pinging google), that doesn´t work either:

ping -I vlan100 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.100.2 vlan100: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
25 packets transmitted, 0 received, 100% packet loss, time 24552ms

Is this a routing problem? Or where should I be looking?
If I do the same thing, but use my br0 interface instead, it works as it should:

ping -I br0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.32.181 br0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=2.36 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=2.31 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=2.32 ms
8

Diagram please.
It helps

9

Wow. Ok, i´m not really great at drawing. But I´ll give it a shot. Not sure exactly what it should contain, and I simplified it a bit. I left out switches and stuff, since there is no problem there. Let me know if it´s incomplete in any way:

image
image906×907 22.4 KB

10

Well your diagram has the TrueNAS have 3 interfaces where your explanations say 2.

I think you need to include switches, VLAN’s and IP Addresses (of the important devices)

11

Ok, how about this?

The TrueNAS is physically connacted to a switch, which in turn is connected to the router. Networking works fine at this level. The physical NIC is eno1.

Network interfaces:
eno1 is my physical NIC.
br0 is a bridge interface created in TrueNAS. It has eno1 as the only bridge member. Docker VM has br0 as NIC, and resides on an IP in the main network, VLAN ID 1(192.168.32.0/24).
VLAN100 is a VLAN type interface created in TrueNAS. dc01 and jump01 has that as a NIC, and both reside on IP addresses in the 192.168.100.0/24 subnet.
interconnection between them works fine, they can access the internet and everything.

The thing about VLAN100 interface is that the TrueNAS has an IP address here, but cannot communicate with the VMs inside the VLAN100 subnet, and can also not access the internet using that interface.

Just for the sake of trying everything, I added the VLAN100 interface to the br0 bridge. That didn´t work. Quite the opposite. The VLAN100 interface became unusable for all VMs inside VLAN 100 subnet.
Hope that clears something up :slight_smile:

12

As a temporary workaround I setup an additional NIC for the docker VM in the same subnet as TrueNAS ls connected to. So now it can talk to TrueNAS, and mount NFS shares shared on the TrueNAS. Not ideal, and I´d really like this to work within the VLAN 100 subnet. Mostly for understanding of how these things work.

Since separating clients from servers is the goal I would like for all servers to only reside in the VLAN, but as a security measure for now, I set a SSH Listenaddress on the VM, so it only accepts SSH connections on the interface in VLAN100. Is that sufficient? Is this the solution?

13

I have now been reading up on TrueNAS VLANs and networking for a couple of days. And i think this might be the answer.

So after reading this.

https://www.reddit.com/r/truenas/comments/rjee50/scale_i_am_lost_with_a_bridge_setup_help/?rdt=54851

Here is my new plan of attack:

  1. Remove IP from VLAN100 interface under networking

  2. Stop all VMs, and set start at boot to off.

  3. Remove VLAN100 NICs from all VMs

  4. Stop services SMB and NFS

  5. Reboot TrueNAS

  6. create bridge interface (br100) set a static IP outside of DHCP scope in VLAN100. Set vlan100 as bridge member

  7. Start services again

  8. Create new NICs for VMs, connected to br100.

  9. Start VMs

Verify that everything works.
Set VM start at boot = on

Have I missed or over complicated anything?

14

Seems sensible

15

I can confirm that I followed these exact steps, and it worked as a charm. Now my TN is fully reachable within the VLAN100 network.

Thanks for all the help.