Network Configuration problem

TrueNAS General
,
1

Hello everyone, good day!

So I am having a very weird problem and at this point I can’t figure out what I am doing wrong… although I believe it might be obvious.

My TrueNAS has two NICs, the one from the motherboard and an extra one that has 4 interfaces in it. For simplicity I will call them enp3 and enp4f0, enp4f1, enp4f2 and enp4f3

I have created a bond between enp4f0 and enp4f1, that seems to be working fine. They are working in a fixed IP of 192.168.30.108. They are connected to a Switch, which will be Switch A, these interfaces allow multiple VMs to communicate with the users in the network.

I have enp4f2 in a separate subnet as 192.168.3.25. This one is also working fine (apparently,) and has been my savior so far. Is meant to communicate specifically with a separate NAS server. So is a direct connection to the server, id doesn’t pass through a switch. Basically a P2P, or that is what is meant to be. Due to the communication problem, this is the interface I have been able to use to communicate and access the TrueNAS UI.

Then… enp4f3 is unplugged…

And, I am trying to use enp3 as my main interface for management. But this one is basically unreachable… This interface is connected to a “Switch B”. I have tried setting DHCP in this interface, but while I can reach the UI (sometimes) through other subnets (192.168.70.0 and 192.168.80.0), I cant reach it through 192.168.30.0, which is the subnet configured on that port.

Switch A and Switch B are parallel connected, and they belong to a Unifi System.

I have set up 8.8.8.8 to my TrueNAS dns, and allowed the gateway to be detected by DHCP.

I have also tried to use static addresses, but no success.

I know this explanation might be messy… do we need a diagram?
Thank you for the help!

2

Diagram? Yes :slight_smile:
Also, login via ssh and give us the output from running the command ‘routel’ so we can see what addresses and masks you’ve got assigned, and what routes and gateways are configured.

I’d avoid using DHCP to set up interfaces on your NAS if possible, manually give the interfaces IP addresses, masks and gateways (if necessary) that are correct for the network they’ll connect to. Manually setting the DNS server is fine, so long as you’re able to reach it via one of your specified gateways.

In System Settings->General Settings->GUI make sure the Web Interface IP4 Address is either 0.0.0.0 (if you want it accessible on every network the NAS is connected to) or the specific IP address (bound to a NIC) that you want it to be accessible on.

What do you mean by “Switch A and Switch B are parallel connected, and they belong to a Unifi System.”

3

This seems to be the same subnet as enp4f0:1. Use a different subnet.
TrueNAS does not like having two different interfaces on the same subnet.

4

@WiteWulf Thank you for your answer!
@etorix Thanks also, I have tried what you mention, but I get the same result, more info below.

Here I am attaching some configuration snapshots of where I am currently, there have been a lot of different attempts though.

image
image855×420 14.8 KB

And the diagram:

image
image764×511 22.8 KB

Notes:

  • As you can see, at the moment I have enp3s0 in the same subnet as enp6s0f1.
  • enp6s0f1 is bonded with enp6s0f0
  • enp6s0f2 is working without problems
  • I have tried moving enp3s0 to a different subnet, like shown in the diagram, but this doesn’t work, I still cant access the GUI, and it also makes the bonded interface to not allow me either. But enp6s0f2 always works.
  • Disconnecting enp3s0 doesn’t make the bond to work either, which i find this point particularly weird.
  • The only way I can access the GUI through enp3s0 is when it is in DHCP, as in the image, and the access is only happening through my laptop, when connected in the .70 subnet, but none of the PC ins the 30 subnet can access. Which is also strange.

The dotted line is how I am debugging this at the moment, I am cable connecting to the 3 subnet, but that link cannot stay there.

5

Oh, mate, that’s all sorts of broken :joy:

You appear to have four different subnets in use (192.168.30.0/24, 192.168.70.0/24, 192.168.80.0/24 and 192.168.3.0/24). You haven’t specified what IP addresses are assigned to NICs on your router, but I’m guessing you’re not using VLANs and subinterfaces. A router needs to have a leg (a configured NIC) in every network it wants to route between. Yours only appears to have two legs, so there’s no way that hosts on all your different subnets can communicate with each other.

Why do you have so many subnets? What’s the purpose? If you can make this network much simpler, with fewer subnets, you have a chance of getting it working.

Hosts on different networks cannot talk to each other without going through a router, even if they’re connected to the same switch.

Next, looking at your TrueNAS interface configuration, you have a bond0 interface, but it has no IP address associated with it. You talked about enp4f0 and enp4f1 in your original post, but I can’t see those on that screen grab.

You’ve also (from your diagram) got two NICs with the same subnet (192.168.180.0/24) connected connected to the same physical network segment. This isn’t allowed. You should only have one NIC (or bond) per network, the NAS should be connected to either Switch B or Switch C (unless you’re doing LACP with those two NICs and switch B and C are actually a stack?). Those two NICs are connected to two switches, but the two switches are linked together by the SFP link, effectively making it one network segment. I suspect you have a configuration issue, in that one of your NICs, as I previously mentioned, is configured for 192.168.108.0/24, when your diagram shows 192.168.180.0/24

Lastly, I think you understand this from your previous post, but the NAS on the 192.168.3.0/24 network segment will only be able to talk to your TrueNAS. TrueNAS is not a router, it won’t pass packets from one interface to another and allow the NAS to talk to other hosts on your network. And the laptop you have, how is that connected without a switch? It won’t be able to talk to anything there anyway as it has a 192.168.70.0/24 address and that is a 192.168.3.0/24 network.

I don’t mean to be rude, but you seem to not understand the basics of TCP/IP subnetting and routing, and have created yourself an overly complex network.

If you draw up another diagram of how you want your network to look, and explain what you want it to achieve, I’d be happy to help design it with you.

1 Like
6

@WiteWulf

I am definitely not an expert here, but I think there are some confusion points. It is a messy structure, I know, this is not my home.

There are 4 subnets, and is not my design:
80 - IoT
70 - Wireless
30 - Ethernet
10 - Management of Unifi equipment

Yes, I get the point of different networks, the router is a DreamMachine from Unifi. It does have the functionality to allow multiple subnets to communicate between them. Also, every single switch allows management of each port and allows to manage which subnets can be accessed or not. At the moment, its free for all basically, so that there are no hidden blocks.

My original post without a diagram I used similar names, not the original ones: “For simplicity I will call them”. Maybe I shouldn’t have done that. Now with the print screen you got the real names.

The print screen doesn’t show all the specs of each interface:

enp6s0f0 & enp6s0f1 are the ones linked tot he bond0. Following a tutorial the IP was static on one interface, the enp6s0f1, should the static IP be in the bond? This is a new feature for me, so I am not familiar with it.

  • We basically need the TrueNAS to connect to the other NAS Directly (check)
  • We need to have one interface for management with the GUI (not check)
  • We need a way for the rest of the network (switch A, B, C and any other device here) to communicate with TrueNAS, this is a bond to increase the bandwidth.
7

Are switch B and switch C stacked (ie. operating as a single logical switch) or just two separate switches with a trunk between them?

If they’re stacked, then your bond interface on the TrueNAS should work, so long as both interfaces on the switch are configured as a port group, and with a link aggregation protocol that is compatible with TrueNAS.

If they’re not stacked you will need to move both of the bonded interface’s connections to the same physical switch. You can’t use a bonded link across multiple logical switches.

And yes, that static IP should be assigned to the bond0 interface, not one of the physical member NICs.

Why do you need separate interfaces on the TrueNAS for management and connecting to the network?

The way I see it, you only need three physical interfaces on your NAS:

  • NIC1 - connects directly to the NAS (using a crossover cable) with an unrouted subnet. Both hosts must be statically configured, and the NAS will not be able to connect to anything else on your network except the TrueNAS
  • NIC2 and NIC3, form a bond interface, bond0, that’s connected to any single switch (unless switch B and switch C are stacked to form a single logical switch). Put a static IP address in whatever subnet you like, so long as it knows that your router is the default gateway. The router should handle routing between all your different subnets, so long as it has legs in each network

I’m guessing those subnets are all assigned to different vlans, and that the unifi router has subinterfaces on each vlan and routes between them. If everything’s open between the different vlans/subnets, as you suggest, there’s no point having them as it complicates your network setup and slows things down, as all inter-vlan traffic must pass through your router, and routing is intrinsically slower than switching. Hosts on the same vlan/subnet will communicate much quicker than those that have to go via the router.

8

Finally got it solved.

It was a mixture of problems:

Problem 1: This one was spotted fast, I was using the same subnet in two different interfaces, and for some reason it worked for a while. Once I wanted to fix this, it wouldn’t fully reset.

I had to reset all interfaces and configure them again.
enp3s0 is down
enp6s0f0 & enp6s0f1 are a bond → subnet 30
enp6s0f2 → subnet 3
enp6s0f3 → subnet 80

Problem 2: The bond was working, but I had the static IP on one interface, not on the bond. Since it worked, maybe not totally correct, but was providing traffic exchange, I assume it was correct. Thanks for this one @WiteWulf

Now it finally works as expected.

1 Like
9

I still don’t see why you need separate interfaces for your 192.168.30.0/24 and 192.168.80.0/24 networks. Either stick the TrueNAS on 192.168.30.0/24 and let your router handle traffic between the vlans, or stick multiple IP address aliases on your bond0 interface and vlan tag them (ensuring that the interfaces on the switch are set up as trunk, not access, ports)

But it’s working, that’s what’s important👍