TrueNAS can't access internet/gateway, but that's it

TrueNAS General
1

Odd issue that I can’t figure out, and seems to be somewhat intermittant as I recently posted this to Reddit, and it fixed itself shortly after posting, but it’s back again now.

I have a pretty simple 192.168.1.0/24 home network with internet gateway at 192.168.1.1 . All of my devices are able to ping/connect to and from my gateway and to my truenas box, but the truenas box and gateway cannot ping each other and I have no internet . This seems to be sporatic, but when it goes, it goes for a while. I can’t think of any event that would trigger it, and I’m kinda at a loss at this point. Next step is to blow away my install and start fresh again.

List of interfaces, i shut down all the running apps prior to this and did a reboot to test in case somehow something there was causing a problem.

fog@truenas:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 88:c9:b3:bb:09:81 brd ff:ff:ff:ff:ff:ff
3: enp7s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 88:c9:b3:bb:09:82 brd ff:ff:ff:ff:ff:ff
4: enp8s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
link/ether f0:2f:74:da:2c:55 brd ff:ff:ff:ff:ff:ff
5: enp9s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 88:c9:b3:b0:50:f7 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.200/24 brd 192.168.1.255 scope global enp9s0
valid_lft forever preferred_lft forever
inet6 fe80::8ac9:b3ff:feb0:50f7/64 scope link
valid_lft forever preferred_lft forever
7: br-5583cd2b89c6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:f4:c0:83:ab brd ff:ff:ff:ff:ff:ff
inet 172.16.8.1/24 brd 172.16.8.255 scope global br-5583cd2b89c6
valid_lft forever preferred_lft forever
8: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:ef:94:7e:42 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.1/24 brd 172.16.0.255 scope global docker0
valid_lft forever preferred_lft forever

Routing table

fog@truenas:~$ ip route
default via 192.168.1.1 dev enp9s0 proto static
172.16.0.0/24 dev docker0 proto kernel scope link src 172.16.0.1 linkdown
172.16.8.0/24 dev br-5583cd2b89c6 proto kernel scope link src 172.16.8.1 linkdown
192.168.1.0/24 dev enp9s0 proto kernel scope link src 192.168.1.200

Can ping both Truenas and gateway from my desktop (192.168.1.100)

[fog@sorrento ~]$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.81 ms
^C
— 192.168.1.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.811/1.811/1.811/0.000 ms
[fog@sorrento ~]$ ping 192.168.1.200
PING 192.168.1.200 (192.168.1.200) 56(84) bytes of data.
64 bytes from 192.168.1.200: icmp_seq=1 ttl=64 time=0.840 ms
^C

Truenas to gateway just hangs until ^C

fog@truenas:~$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
^C
— 192.168.1.1 ping statistics —
146 packets transmitted, 0 received, 100% packet loss, time 148455ms

From my internet gateway to truenas via webui:

Result

Ping statistics for 192.168.1.200
Pinging 192.168.1.200 with 32 bytes of data:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss)
Round-trip times: Minimum = 0 ms, Maximum = 0 ms, Average = 0 ms

What else can I provide for troubleshooting? It’s like these two devices are mad at each other. Looking at the gateway UI, I don’t see anything out of the ordinary, the truenas box shows up like every other device on the network, there’s no block list or anything that I’m aware of. I’m at a loss here.

3

One thing I meant to add is screenshots of my truenas UI, but it doesn’t appear I can embed here or provide link on where it’s uploaded. But, there’s nothing special in that, I have 4 network devices, only 1 is in use with 192.168.1.200/24 network.

Of note, I typically have static DHCP assigned, but just set it static in the UI for troubleshooting and it made no difference.

Also forgot to mention, this TrueNAS SCALE ElectricEel-24.10.1

NICs:

fog@truenas:~$ lspci | grep Ethernet
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
08:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 26)
09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 04)

Blockquote

4

I have exactly the same problem since one month and no idea how to solve it…

5

Screenshots of your network settings would probably help. Try to give all the networking info so we can understand how you have it all set up currently. You need to paint a clear picture of how everything is laid out. Is DHCP used or have you set up it all static?

6

Output of following please:

ifconfig

Also any chance router has ip assigned to another device by chance?

7

Indeed, that could be it, or maybe there is another router on the network with the same IP.

I am curious what arp -a shows before and during one of these outages.
Ideally checked both on the TrueNAS server and another system where things continue to work.

8

Nope, at least in my config there is just one router. the are no other device with the same ip on the lan.
Unfortunately (or no…not unfortunately at all :slight_smile: ) I can’t show the result of the arp command because yesterday I managed to solve the problem. (maybe it’s not a real fix and at the next reboot I will face the problem again but for the moment I won’t take any risks.)
If you are wondering about the fix…I changed the Ip of the server with a new one.

1 Like
9

This is exactly what I neded up doing as well. Changed the IP of my servers from .200 to .250 and the problem is resolved.

Looking at my router, which was my DHCP server as well, I see no duplicate IP assignment, and I even factory reset my internet router to make sure everything was clear of it, and it still didn’t work. Such a strange issue, but since changing the IP, it’s all fine and haven’t seen the issue again since.

10

That suggests you both have a device with the same (old) IP set statically.

1 Like
11

It’s odd that no other devices had any issues connecting to either device, they just couldn’t talk to each other. After changing the IP, there is no ping response on the old IP. There have been no changes to the network for several years, 192.168.1.200 has always been my server, and it was re-installed early last year and no issues starting out , except a minor issue with defualt routes not being set up correctly.

I also thought of this as well and was troubleshooting it when the issue was existing by shutting off the server, but no response from anything and nothing showing on the DHCP server of anything. If something else on the network was trying to use the IP, it was being awfully quiet about it.

12

If 200 was in your dhcp range, it’s possible it would take a few years to get handed out… and the router’s dhcp server may not be smart enough to not hand it out.

When setting static ips on servers you really need to chose an ip that is not in the dhcp servers ip pool that it hands out.

Eg, my dhcp server is configured to hand out 100-199 addresses, thus 200 is good.

13

I have the exact same problem. I run a 192.168.2.0/24 network and for the last 3 weeks I have been racking my brains around this issue.

Truenas Scale : 192.168.2.63 fixed IP address
DHCP range for the LAN : 192.168.2.20 - 192.168.2.60
On the LAN I have other 3 other devices which are all DHCP client, so no conflict in IPs

This problem started when I changed some DNS settings to setup Pi-Hole which is now uninstalled. I reverted back to my normal DNS servers afterwards which are 8.8.8.8 and 8.8.4.4, but that is beside the point because I can’t get to the default gateway.

I can ping anything on my LAN from my Truenas box, except for the default gateway 192.168.2.1 and my Truenas box is connected directly on my isp lan modem port 2. In my ISP portal, the truenas box does not showup, even though I can connect to it with no problems from the LAN. From the ISP modem, I can’t ping the truenas server.

root@truenas[~]# ip route
default via 192.168.2.1 dev eno1 proto static
192.168.2.0/24 dev eno1 proto kernel scope link src 192.168.2.63

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.63 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fe80::4639:c4ff:fe8d:9224 prefixlen 64 scopeid 0x20
ether 44:39:c4:8d:92:24 txqueuelen 1000 (Ethernet)
RX packets 10048 bytes 2068630 (1.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7165 bytes 6624631 (6.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 20 memory 0xf7c00000-f7c20000

root@truenas[~]# route -nv
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 eno1
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eno1

I have upgraded to ElectricEel-24.10.2 hoping the problem would have resolved itself with the upgrade. The problem still remains.

I have tried changing from fixed address to DHCP or changing to another fix address, but Truenas won’t save the changes, I’m assuming because it can’t find the DHCP server on the ISP modem. AS for changing to another fixed IP, truenas, won’t let me change it.

14

This sounds unusual.
Do you have a modem and a router, or are they the same device?

If they are two separate devices, I recommend that you connect your router’s WAN-port to your modem and your TrueNAS server together with all your other client devices to LAN-ports on your router. If you run out of ports, use a switch to “split” one connection point into several.

Also, make sure your modem is in bridge mode if you use your router for DHCP. You don’t want to have two different DHCP servers.

15

Thanks for your reply.

Very unsual…indeed. I have been running this server for like forever. It’s not a new install.

Modem and ISP router is the same device with the DHCP service running on it. Only one DHCP service for the LAN.

This morning I moved over the truenas server to the switch where all the other 192.198.2.0 devices are. I get the same result. I can connect to the truenas through it’s IP address, I can ping other 192.198.2.0 devices from truenas except for the gateway.

Here are some arp -a results from two non DHCP on the same switch and both show up on the ISP modem, but not the truenas server. The truenas can’t resolve the MAC address of the default gateway.

root@truenas[~]# arp -a
? (192.168.2.11) at cc:f4:11:1a:06:01 [ether] on eno1
? (192.168.2.1) at on eno1
? (192.168.2.12) at 4c:cc:6a:c2:2e:3f [ether] on eno1

What is stranger is that truenas will not let me change to another fixed IP address, I have tried to change it to 192.168.2.70 from the web interface it comes back with this error:

We encountered an issue while applying the new network changes. Unfortunately, we were unable to reconnect to the system after the changes were implemented. As a result, we have restored the previous network configuration to ensure continued connectivity.

However, when I lookup the modem, 192.168.2.70 is registered…but the server is back to 192.168.2.63

Output from the ISP modem:
|44:39:C4:8D:92:24|Connected|LAN : 3 IP : 192.168.2.70 MAC : 44:39:C4:8D:92:24|
| — | — |

Interface config after being reverted to 192.168.2.63:

root@truenas[~]# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.63 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fe80::4639:c4ff:fe8d:9224 prefixlen 64 scopeid 0x20
ether 44:39:c4:8d:92:24 txqueuelen 1000 (Ethernet)
RX packets 8932 bytes 1792785 (1.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6990 bytes 4268710 (4.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 20 memory 0xf7c00000-f7c20000

I wonder, if I can set another fixed IP address right at the console while being logged in interactively on the server? This might work if it can be done.

17

Finally, I got to log in interactively on the server. After changing its IP address to 192.168.2.70/24, everything works.

Now I have to change my backup scripts and fstab configs to point to new address.