Networking problems

TrueNAS General
1

Hello,

A couple weeks ago my router died and I got a new isp issued one. Ever since then my truenas box has been unable to curl anything on wan, however local addresses do work. If I change the local ip, it will work for a little bit (A day or so, but sometimes as little as < 1 minute) then break again. It’s the only device on the network that can’t curl the gateway’s web ui (when it’s not working). I’ve tried rotating the mac address, reinstalling, and even getting the router replaced and I don’t know what to do next. It’s been down almost a month now and it’s getting increasingly hard to work around.

truenas_admin@truenas[~]$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 04:42:1a:f0:10:68 brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
    inet 192.168.0.44/24 brd 192.168.0.255 scope global dynamic eno1
       valid_lft 3272sec preferred_lft 3272sec
    inet6 fe80::642:1aff:fef0:1068/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
truenas_admin@truenas[~]$ ip route
default via 192.168.0.1 dev eno1 
192.168.0.0/24 dev eno1 proto kernel scope link src 192.168.0.44

It’s currently working on a completely fresh install (not even any storage) under 44, but speaking from experience that won’t last. If i switch it to it’s old address (192.168.0.35) it breaks, even on the fresh install. That’s true both via the web ui and manually with the ip command

2

Are you using a static IP and have you reserved it on the new router?

3

I’ve tried both dynamic and static, reserved and unreserved for both

4

i think the new isp box is blackholing that host after a bit. sticky arp / client filter / weird lease state. local works and a fresh ip working for a minute then dying is the giveaway.

try doin this:

  1. from the truenas shell while its broken:
ip neigh
ping -c2 GATEWAY_IP
curl -v --max-time 5 http://GATEWAY_IP/
  1. compare to a phone/laptop on the same subnet that still works. if the phone can hit the gateway ui and truenas cant, its filtered on that mac/ip
  2. on the isp router check client isolation / ap isolation, device blocklist, parental controls, and that the nas is on the main lan (not guest/iot)
  3. set a static outside the dhcp pool + reserve it, then reboot router and nas once. if it dies again after a day dump:
midclt call interface.query
cat /etc/resolv.conf
ip route
traceroute -n 1.1.1.1
  1. temp test: put a cheap dumb switch between the nas and the router (or move to a different lan port). if that stays up, the isp box port/mac filter is the problem

also paste full ip addr + ip route from the broken state.

5

For my DOCSIS cable MODEM router, it has a feature to block traffic based on Ethernet / WiFi MAC address. I basically configured it to block by default, and then any new equipment, (NAS, tablet, etc…), I have to go into it’s GUI and tell it that this device is okay to leave my home network. (It also allows me to name such MAC addresses, so I know which is what…)

Your new router might have such a feature. Or even a firewall turned on by default. Check it for a firewall and if you can whitelist your NAS.

6

If you are getting a DHCP address, from the router, it means your wiring to the router is ok.

How about we start with basics:

Ping the default route, which should be the router’s LAN address.
Ping other local devices.
Ping google.com and then ping 8.8.8.8.

Any of them fail?

I never use/pay for the ISPs router or wifi. They can be flaky when you have many connections/clients as their state table is limited due to the small memory in the device.

They also may not update the firmware as vulnerabilities get found.

7

Thanks for the help everyone, I’m away from home right now but I’ll be sure to get back to you soon ^^. I will note that all pings work no matter what

8

If all pings (from the system) work, the network looks ok.

Ping the Internet host in question and go from there.

Can’t see why the new router would block web traffic (ports 80, 443), only for that host?

9 
ip neigh       
ping -c2 192.168.0.1
curl -v --max-time 5 http://192.168.0.1/
192.168.0.22 dev eno1 lladdr ac:a7:04:e2:aa:dc REACHABLE 
192.168.0.160 dev eno1 FAILED 
172.16.5.2 dev br-e76427e053ef lladdr 86:77:2a:fe:03:ba STALE 
192.168.0.20 dev eno1 lladdr e8:b0:c5:be:72:3d STALE 
172.16.11.2 dev br-564ba7f74386 lladdr a6:7d:84:47:39:6d STALE 
192.168.0.44 dev eno1 FAILED 
192.168.0.21 dev eno1 lladdr 14:5a:fc:68:b7:ef REACHABLE 
192.168.0.100 dev eno1 lladdr d8:3a:dd:91:14:d9 REACHABLE 
172.16.9.3 dev br-73b9edd5775b lladdr 06:9c:fd:0b:4d:ae STALE 
172.16.7.5 dev br-443acd108c63 lladdr de:4d:21:db:f0:03 STALE 
192.168.0.17 dev eno1 lladdr ca:c5:b4:80:79:69 STALE 
172.16.3.3 dev br-4abfe7b0cfeb lladdr 8a:ae:74:9d:90:74 STALE 
172.16.1.2 dev br-42d569ec26bb lladdr 62:14:2e:b9:94:de STALE 
172.16.10.2 dev br-f564ee42d463 lladdr 0a:09:7f:b7:83:db STALE 
172.16.4.2 dev br-701bbf7b8113 lladdr ba:47:8a:50:dc:10 STALE 
192.168.0.6 dev eno1 lladdr d8:3a:dd:91:14:d9 STALE 
172.16.7.2 dev br-443acd108c63 lladdr f2:69:ef:ad:23:22 STALE 
192.168.0.5 dev eno1 lladdr 74:b6:b6:9d:3d:4d STALE 
192.168.0.2 dev eno1 lladdr 2c:71:ff:5f:17:b9 STALE 
172.16.6.2 dev br-4d246b714519 lladdr b6:95:6d:64:79:ea STALE 
192.168.0.3 dev eno1 lladdr 74:b6:b6:9d:e9:4d STALE 
192.168.0.118 dev eno1 lladdr 74:b6:b6:9d:8f:ad REACHABLE 
192.168.0.1 dev eno1 lladdr 00:58:28:b5:d2:be REACHABLE 
fe80::de56:7bff:fec4:d231 dev eno1 lladdr dc:56:7b:c4:d2:31 STALE 
fe80::cc47:fbff:feaa:ced8 dev br-3608dae83c94 lladdr ce:47:fb:aa:ce:d8 STALE 
fe80::4aeb:62ff:fe72:1827 dev eno1 lladdr 48:eb:62:72:18:27 STALE 
fe80::1421:5663:17ea:b6b0 dev eno1 lladdr de:0c:d7:e9:59:72 STALE 
fe80::e49e:dbff:fe6e:9064 dev br-273f71f272b9 lladdr e6:9e:db:6e:90:64 STALE 
fe80::d9a6:18d6:491c:f9b2 dev eno1 lladdr a0:80:69:7d:d0:f7 STALE 
fe80::48e:e890:1096:1325 dev eno1 lladdr 4e:eb:86:63:82:36 STALE 
fe80::4a5f:99ff:fe1b:5a99 dev eno1 lladdr 48:5f:99:1b:5a:99 STALE 
fe80::102b:5ef3:1a9e:57c0 dev eno1 lladdr fa:7c:d2:7d:34:6d STALE 
fe80::969f:3eff:fe82:88cc dev eno1 lladdr 94:9f:3e:82:88:cc STALE 
fe80::b847:8aff:fe50:dc10 dev br-701bbf7b8113 lladdr ba:47:8a:50:dc:10 REACHABLE 
fe80::3a42:bff:feb6:e52c dev eno1 lladdr 38:42:0b:b6:e5:2c STALE 
fe80::8ef:e922:e982:337d dev eno1 lladdr 3e:0b:52:d3:2a:e6 STALE 
fe80::ccc:713f:8477:f0cc dev eno1 lladdr 4e:eb:86:63:82:36 STALE 
fe80::809:7fff:feb7:83db dev br-f564ee42d463 lladdr 0a:09:7f:b7:83:db STALE 
fe80::10a3:2764:f425:5090 dev eno1 lladdr 1a:e8:23:fc:1c:92 STALE 
fe80::86f:7522:882f:37 dev eno1 lladdr 46:87:bf:5a:fe:72 STALE 
fe80::6869:f6ca:b4f1:b673 dev eno1 lladdr f4:a4:75:de:6b:d3 STALE 
fe80::8477:2aff:fefe:3ba dev br-e76427e053ef lladdr 86:77:2a:fe:03:ba STALE 
fe80::895:7632:c8a6:5e05 dev eno1 lladdr b2:b0:71:87:3d:36 STALE 
fe80::a47d:84ff:fe47:396d dev br-564ba7f74386 lladdr a6:7d:84:47:39:6d STALE 
fe80::6014:2eff:feb9:94de dev br-42d569ec26bb lladdr 62:14:2e:b9:94:de STALE 
fe80::1865:7b06:a701:ce40 dev eno1 lladdr b6:9b:43:f3:d1:0c STALE 
fdd0:0:0:5::2 dev br-4d246b714519 lladdr b6:95:6d:64:79:ea STALE 
fe80::88b:a0ea:3e99:bf07 dev eno1 lladdr 46:87:bf:5a:fe:72 STALE 
fe80::2e71:ffff:fe5f:17b9 dev eno1 lladdr 2c:71:ff:5f:17:b9 STALE 
fe80::ca3a:6bff:fee7:166 dev eno1 lladdr c8:3a:6b:e7:01:66 STALE 
fe80::c5f:e3e:e474:a48b dev eno1 lladdr ca:c5:b4:80:79:69 STALE 
fe80::4e1:c4ec:ea72:83e4 dev eno1 lladdr b6:9b:43:f3:d1:0c STALE 
fdd0:0:0:3::2 dev br-701bbf7b8113 lladdr ba:47:8a:50:dc:10 REACHABLE 
fe80::30cd:75ff:fe97:87d3 dev eno1 lladdr 32:cd:75:97:87:d3 STALE 
fdd0:0:0:8::2 dev br-564ba7f74386 lladdr a6:7d:84:47:39:6d STALE 
fe80::e0d8:c9ff:fe30:fb66 dev br-ba2fba13f31a lladdr e2:d8:c9:30:fb:66 STALE 
fe80::80f:c816:1ece:2065 dev eno1 lladdr 7a:38:39:ed:5a:20 STALE 
fe80::4e3:65d9:fd27:3ebd dev eno1 lladdr c2:fc:ee:41:76:d6 STALE 
fe80::ea4c:4aff:febc:b511 dev eno1 lladdr e8:4c:4a:bc:b5:11 STALE 
fe80::6130:cefc:3d78:572b dev eno1 FAILED 
fe80::85:7db3:ed39:fb9 dev eno1 lladdr b2:b0:71:87:3d:36 STALE 
fe80::969f:3eff:fe82:746a dev eno1 lladdr 94:9f:3e:82:74:6a STALE 
fe80::b495:6dff:fe64:79ea dev br-4d246b714519 lladdr b6:95:6d:64:79:ea STALE 
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.643 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.434 ms

--- 192.168.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1006ms
rtt min/avg/max/mdev = 0.434/0.538/0.643/0.104 ms
*   Trying 192.168.0.1:80...
* Connection timed out after 5001 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 5001 milliseconds

ip addr;ip route
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 04:42:1a:f0:10:68 brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
    inet 192.168.0.35/24 brd 192.168.0.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::642:1aff:fef0:1068/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
12: br-3608dae83c94: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether fe:1e:89:e2:e0:5d brd ff:ff:ff:ff:ff:ff
    inet 172.16.8.1/24 brd 172.16.8.255 scope global br-3608dae83c94
       valid_lft forever preferred_lft forever
    inet6 fdd0:0:0:6::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::fc1e:89ff:fee2:e05d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
13: br-42d569ec26bb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 4a:9e:22:d7:5e:4e brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.1/24 brd 172.16.1.255 scope global br-42d569ec26bb
       valid_lft forever preferred_lft forever
    inet6 fdd0:0:0:1::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::489e:22ff:fed7:5e4e/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
15: br-564ba7f74386: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether f6:6d:1a:12:47:68 brd ff:ff:ff:ff:ff:ff
    inet 172.16.11.1/24 brd 172.16.11.255 scope global br-564ba7f74386
       valid_lft forever preferred_lft forever
    inet6 fdd0:0:0:8::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::f46d:1aff:fe12:4768/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
16: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether c2:a8:c0:f7:e2:d2 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.1/24 brd 172.16.0.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fdd0::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
17: vethe31bf22@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-42d569ec26bb state UP group default 
    link/ether aa:ac:84:9b:c7:3e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::a8ac:84ff:fe9b:c73e/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
20: veth1cad93f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-564ba7f74386 state UP group default 
    link/ether f2:91:16:c7:4a:68 brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::f091:16ff:fec7:4a68/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
21: veth0c094be@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-3608dae83c94 state UP group default 
    link/ether f6:af:44:74:38:33 brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::f4af:44ff:fe74:3833/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
49: br-273f71f272b9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c6:dd:dd:74:d2:da brd ff:ff:ff:ff:ff:ff
    inet 172.16.2.1/24 brd 172.16.2.255 scope global br-273f71f272b9
       valid_lft forever preferred_lft forever
    inet6 fdd0:0:0:2::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::c4dd:ddff:fe74:d2da/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
50: veth8da1388@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-273f71f272b9 state UP group default 
    link/ether 56:a4:a0:e3:38:30 brd ff:ff:ff:ff:ff:ff link-netnsid 8
    inet6 fe80::54a4:a0ff:fee3:3830/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
53: br-4abfe7b0cfeb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether a6:97:0e:d7:c3:5b brd ff:ff:ff:ff:ff:ff
    inet 172.16.3.1/24 brd 172.16.3.255 scope global br-4abfe7b0cfeb
       valid_lft forever preferred_lft forever
    inet6 fe80::a497:eff:fed7:c35b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
54: veth23be462@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-4abfe7b0cfeb state UP group default 
    link/ether 96:1b:66:2e:82:2f brd ff:ff:ff:ff:ff:ff link-netnsid 7
    inet6 fe80::941b:66ff:fe2e:822f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
55: vethe4ce19c@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-4abfe7b0cfeb state UP group default 
    link/ether 46:62:3c:e1:8d:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 13
    inet6 fe80::4462:3cff:fee1:8de1/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
57: br-701bbf7b8113: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 42:78:8c:dc:18:c2 brd ff:ff:ff:ff:ff:ff
    inet 172.16.4.1/24 brd 172.16.4.255 scope global br-701bbf7b8113
       valid_lft forever preferred_lft forever
    inet6 fdd0:0:0:3::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::4078:8cff:fedc:18c2/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
58: vethffa34ef@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-701bbf7b8113 state UP group default 
    link/ether a6:03:1b:0e:82:db brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::a403:1bff:fe0e:82db/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
60: br-e76427e053ef: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether fe:00:87:0a:a9:9f brd ff:ff:ff:ff:ff:ff
    inet 172.16.5.1/24 brd 172.16.5.255 scope global br-e76427e053ef
       valid_lft forever preferred_lft forever
    inet6 fdd0:0:0:4::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::fc00:87ff:fe0a:a99f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
61: veth37ed6f9@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-e76427e053ef state UP group default 
    link/ether e2:9e:5d:f5:67:6d brd ff:ff:ff:ff:ff:ff link-netnsid 17
    inet6 fe80::e09e:5dff:fef5:676d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
64: br-73b9edd5775b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether ae:f1:58:5b:e9:d4 brd ff:ff:ff:ff:ff:ff
    inet 172.16.9.1/24 brd 172.16.9.255 scope global br-73b9edd5775b
       valid_lft forever preferred_lft forever
    inet6 fe80::acf1:58ff:fe5b:e9d4/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
65: vethbe27c33@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-73b9edd5775b state UP group default 
    link/ether 9a:41:40:37:4d:ee brd ff:ff:ff:ff:ff:ff link-netnsid 6
    inet6 fe80::9841:40ff:fe37:4dee/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
66: veth2b7f959@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-73b9edd5775b state UP group default 
    link/ether fe:be:d1:a3:d5:10 brd ff:ff:ff:ff:ff:ff link-netnsid 10
    inet6 fe80::fcbe:d1ff:fea3:d510/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
80: br-f564ee42d463: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 3a:a6:fe:07:4a:2b brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.1/24 brd 172.16.10.255 scope global br-f564ee42d463
       valid_lft forever preferred_lft forever
    inet6 fdd0:0:0:7::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::38a6:feff:fe07:4a2b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
81: vethd6242d6@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f564ee42d463 state UP group default 
    link/ether 2e:60:7a:83:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::2c60:7aff:fe83:eeee/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
83: br-ba2fba13f31a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 92:20:e6:c0:14:bb brd ff:ff:ff:ff:ff:ff
    inet 172.16.12.1/24 brd 172.16.12.255 scope global br-ba2fba13f31a
       valid_lft forever preferred_lft forever
    inet6 fdd0:0:0:9::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::9020:e6ff:fec0:14bb/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
84: veth55b393c@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-ba2fba13f31a state UP group default 
    link/ether be:6d:ca:87:7a:0c brd ff:ff:ff:ff:ff:ff link-netnsid 9
    inet6 fe80::bc6d:caff:fe87:7a0c/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
125: br-443acd108c63: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 2e:27:e6:3b:57:a3 brd ff:ff:ff:ff:ff:ff
    inet 172.16.7.1/24 brd 172.16.7.255 scope global br-443acd108c63
       valid_lft forever preferred_lft forever
    inet6 fe80::2c27:e6ff:fe3b:57a3/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
126: vethcd7eadd@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-443acd108c63 state UP group default 
    link/ether ba:6f:97:ff:8e:64 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::b86f:97ff:feff:8e64/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
127: veth54770c0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-443acd108c63 state UP group default 
    link/ether 22:be:62:e1:1d:3f brd ff:ff:ff:ff:ff:ff link-netnsid 12
    inet6 fe80::20be:62ff:fee1:1d3f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
128: vethfeba251@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-443acd108c63 state UP group default 
    link/ether 3e:ff:d4:7c:d6:31 brd ff:ff:ff:ff:ff:ff link-netnsid 14
    inet6 fe80::3cff:d4ff:fe7c:d631/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
129: vethca4ac83@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-443acd108c63 state UP group default 
    link/ether ba:38:55:f7:2a:60 brd ff:ff:ff:ff:ff:ff link-netnsid 15
    inet6 fe80::b838:55ff:fef7:2a60/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
131: br-4d246b714519: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 7e:53:fe:9b:4b:a7 brd ff:ff:ff:ff:ff:ff
    inet 172.16.6.1/24 brd 172.16.6.255 scope global br-4d246b714519
       valid_lft forever preferred_lft forever
    inet6 fdd0:0:0:5::1/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::7c53:feff:fe9b:4ba7/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
132: veth4b3ed06@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-4d246b714519 state UP group default 
    link/ether 6a:fc:e9:62:4d:fb brd ff:ff:ff:ff:ff:ff link-netnsid 11
    inet6 fe80::68fc:e9ff:fe62:4dfb/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
default via 192.168.0.1 dev eno1 proto static 
172.16.0.0/24 dev docker0 proto kernel scope link src 172.16.0.1 linkdown 
172.16.1.0/24 dev br-42d569ec26bb proto kernel scope link src 172.16.1.1 
172.16.2.0/24 dev br-273f71f272b9 proto kernel scope link src 172.16.2.1 
172.16.3.0/24 dev br-4abfe7b0cfeb proto kernel scope link src 172.16.3.1 
172.16.4.0/24 dev br-701bbf7b8113 proto kernel scope link src 172.16.4.1 
172.16.5.0/24 dev br-e76427e053ef proto kernel scope link src 172.16.5.1 
172.16.6.0/24 dev br-4d246b714519 proto kernel scope link src 172.16.6.1 
172.16.7.0/24 dev br-443acd108c63 proto kernel scope link src 172.16.7.1 
172.16.8.0/24 dev br-3608dae83c94 proto kernel scope link src 172.16.8.1 
172.16.9.0/24 dev br-73b9edd5775b proto kernel scope link src 172.16.9.1 
172.16.10.0/24 dev br-f564ee42d463 proto kernel scope link src 172.16.10.1 
172.16.11.0/24 dev br-564ba7f74386 proto kernel scope link src 172.16.11.1 
172.16.12.0/24 dev br-ba2fba13f31a proto kernel scope link src 172.16.12.1 
192.168.0.0/24 dev eno1 proto kernel scope link src 192.168.0.35

Router doesn’t seem to have any applicable settings/log options. I will try #4 now

10

yeah that paste shows gateway arp REACHABLE and ping works. curl to :80 times out tho.

check:
-curl -v --max-time 5 https://1.1.1.1
-curl -v --max-time 5 https://example.com
-sudo nft list ruleset | head -100
-iptables -L -n | head -40

if those internet curls hang too, something on the nas is dropping outbound tcp (host firewall or an app path). if they work and only 192.168.0.1:80 dies, the isp box is filtering http for that host while ping still gets through.

while its broken once:
-tcpdump -ni eno1 host 192.168.0.1 and port 80
then run the gateway curl again and see if syns leave with nothing coming back.

11

yeah no curling the internet either

# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                fib daddr type local counter packets 17587 bytes 1037338 jump DOCKER
        }

        chain OUTPUT {
                type nat hook output priority -100; policy accept;
                ip daddr != 127.0.0.0/8 fib daddr type local counter packets 2768 bytes 166080 jump DOCKER
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                oifname != "br-46743e4402f4" ip saddr 172.16.4.0/24 counter packets 60 bytes 3600 masquerade
                oifname != "br-c033d8b7927a" ip saddr 172.16.12.0/24 counter packets 0 bytes 0 masquerade
                oifname != "br-e011e484d9ab" ip saddr 172.16.11.0/24 counter packets 185654 bytes 18696311 masquerade
                oifname != "br-ff5d841e1936" ip saddr 172.16.9.0/24 counter packets 16 bytes 960 masquerade
                oifname != "br-8daecf5f90e4" ip saddr 172.16.7.0/24 counter packets 105 bytes 6300 masquerade
                oifname != "br-6648babffbfa" ip saddr 172.16.8.0/24 counter packets 8467 bytes 508020 masquerade
                oifname != "br-6d76dba05e84" ip saddr 172.16.3.0/24 counter packets 3145 bytes 188700 masquerade
                oifname != "br-9e60da18cd40" ip saddr 172.16.2.0/24 counter packets 6305 bytes 378300 masquerade
                oifname != "br-a906a8f9d766" ip saddr 172.16.5.0/24 counter packets 0 bytes 0 masquerade
                oifname != "br-cd09b9c6e58c" ip saddr 172.16.6.0/24 counter packets 0 bytes 0 masquerade
                oifname != "br-684e55616329" ip saddr 172.16.10.0/24 counter packets 0 bytes 0 masquerade
                oifname != "br-a154210e39a0" ip saddr 172.16.1.0/24 counter packets 8 bytes 480 masquerade
                oifname != "docker0" ip saddr 172.16.0.0/24 counter packets 0 bytes 0 masquerade
        }

        chain DOCKER {
                iifname "br-46743e4402f4" counter packets 0 bytes 0 return
                iifname "br-c033d8b7927a" counter packets 0 bytes 0 return
                iifname "br-e011e484d9ab" counter packets 0 bytes 0 return
                iifname "br-ff5d841e1936" counter packets 0 bytes 0 return
                iifname "br-8daecf5f90e4" counter packets 0 bytes 0 return
                iifname "br-6648babffbfa" counter packets 0 bytes 0 return
                iifname "br-6d76dba05e84" counter packets 0 bytes 0 return
                iifname "br-9e60da18cd40" counter packets 0 bytes 0 return
                iifname "br-a906a8f9d766" counter packets 0 bytes 0 return
                iifname "br-cd09b9c6e58c" counter packets 0 bytes 0 return
                iifname "br-684e55616329" counter packets 0 bytes 0 return
                iifname "br-a154210e39a0" counter packets 0 bytes 0 return
                iifname "docker0" counter packets 0 bytes 0 return
                iifname != "br-a154210e39a0" tcp dport 8923 counter packets 160 bytes 9600 dnat to 172.16.1.2:8923
                iifname != "br-a154210e39a0" tcp dport 9090 counter packets 13 bytes 684 dnat to 172.16.1.2:9090
                iifname != "br-684e55616329" tcp dport 30236 counter packets 0 bytes 0 dnat to 172.16.10.2:30236
                iifname != "br-cd09b9c6e58c" tcp dport 30266 counter packets 2 bytes 120 dnat to 172.16.6.2:30266
                iifname != "br-a906a8f9d766" tcp dport 30142 counter packets 36 bytes 2160 dnat to 172.16.5.3:30142
                iifname != "br-a906a8f9d766" tcp dport 30143 counter packets 52 bytes 3120 dnat to 172.16.5.3:30143
                iifname != "br-6648babffbfa" tcp dport 30103 counter packets 1 bytes 60 dnat to 172.16.8.3:30103
                iifname != "br-8daecf5f90e4" tcp dport 30013 counter packets 824 bytes 49440 dnat to 172.16.7.2:8096
                iifname != "br-8daecf5f90e4" tcp dport 30014 counter packets 0 bytes 0 dnat to 172.16.7.2:8920
                iifname != "br-ff5d841e1936" tcp dport 30068 counter packets 0 bytes 0 dnat to 172.16.9.2:30068
                iifname != "br-e011e484d9ab" tcp dport 30024 counter packets 0 bytes 0 dnat to 172.16.11.2:30024
                iifname != "br-e011e484d9ab" tcp dport 51413 counter packets 0 bytes 0 dnat to 172.16.11.2:51413
                iifname != "br-e011e484d9ab" udp dport 51413 counter packets 1236 bytes 157195 dnat to 172.16.11.2:51413
                iifname != "br-c033d8b7927a" tcp dport 30059 counter packets 0 bytes 0 dnat to 172.16.12.2:30059
# Warning: table ip filter is managed by iptables-nft, do not touch!
                iifname != "br-46743e4402f4" tcp dport 30041 counter packets 72 bytes 4532 dnat to 172.16.4.5:30041
        }
}
table ip filter {
        chain DOCKER {
                iifname != "br-46743e4402f4" oifname "br-46743e4402f4" ip daddr 172.16.4.5 tcp dport 30041 counter packets 53 bytes 3392 accept
                iifname != "br-c033d8b7927a" oifname "br-c033d8b7927a" ip daddr 172.16.12.2 tcp dport 30059 counter packets 0 bytes 0 accept
                iifname != "br-e011e484d9ab" oifname "br-e011e484d9ab" ip daddr 172.16.11.2 udp dport 51413 counter packets 1251 bytes 159712 accept
                iifname != "br-e011e484d9ab" oifname "br-e011e484d9ab" ip daddr 172.16.11.2 tcp dport 51413 counter packets 0 bytes 0 accept
                iifname != "br-e011e484d9ab" oifname "br-e011e484d9ab" ip daddr 172.16.11.2 tcp dport 30024 counter packets 0 bytes 0 accept
                iifname != "br-ff5d841e1936" oifname "br-ff5d841e1936" ip daddr 172.16.9.2 tcp dport 30068 counter packets 0 bytes 0 accept
                iifname != "br-8daecf5f90e4" oifname "br-8daecf5f90e4" ip daddr 172.16.7.2 tcp dport 8920 counter packets 0 bytes 0 accept
                iifname != "br-8daecf5f90e4" oifname "br-8daecf5f90e4" ip daddr 172.16.7.2 tcp dport 8096 counter packets 0 bytes 0 accept
                iifname != "br-6648babffbfa" oifname "br-6648babffbfa" ip daddr 172.16.8.3 tcp dport 30103 counter packets 0 bytes 0 accept
                iifname != "br-a906a8f9d766" oifname "br-a906a8f9d766" ip daddr 172.16.5.3 tcp dport 30143 counter packets 0 bytes 0 accept
                iifname != "br-a906a8f9d766" oifname "br-a906a8f9d766" ip daddr 172.16.5.3 tcp dport 30142 counter packets 0 bytes 0 accept
                iifname != "br-cd09b9c6e58c" oifname "br-cd09b9c6e58c" ip daddr 172.16.6.2 tcp dport 30266 counter packets 0 bytes 0 accept
                iifname != "br-684e55616329" oifname "br-684e55616329" ip daddr 172.16.10.2 tcp dport 30236 counter packets 0 bytes 0 accept
                iifname != "br-a154210e39a0" oifname "br-a154210e39a0" ip daddr 172.16.1.2 tcp dport 9090 counter packets 13 bytes 684 accept
                iifname != "br-a154210e39a0" oifname "br-a154210e39a0" ip daddr 172.16.1.2 tcp dport 8923 counter packets 0 bytes 0 accept
                iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
                iifname != "br-a154210e39a0" oifname "br-a154210e39a0" counter packets 0 bytes 0 drop
                iifname != "br-684e55616329" oifname "br-684e55616329" counter packets 0 bytes 0 drop
                iifname != "br-cd09b9c6e58c" oifname "br-cd09b9c6e58c" counter packets 0 bytes 0 drop
                iifname != "br-a906a8f9d766" oifname "br-a906a8f9d766" counter packets 0 bytes 0 drop
                iifname != "br-9e60da18cd40" oifname "br-9e60da18cd40" counter packets 0 bytes 0 drop
                iifname != "br-6d76dba05e84" oifname "br-6d76dba05e84" counter packets 0 bytes 0 drop
                iifname != "br-6648babffbfa" oifname "br-6648babffbfa" counter packets 0 bytes 0 drop
                iifname != "br-8daecf5f90e4" oifname "br-8daecf5f90e4" counter packets 0 bytes 0 drop
                iifname != "br-ff5d841e1936" oifname "br-ff5d841e1936" counter packets 0 bytes 0 drop
                iifname != "br-e011e484d9ab" oifname "br-e011e484d9ab" counter packets 0 bytes 0 drop
                iifname != "br-c033d8b7927a" oifname "br-c033d8b7927a" counter packets 0 bytes 0 drop
                iifname != "br-46743e4402f4" oifname "br-46743e4402f4" counter packets 0 bytes 0 drop
        }

        chain DOCKER-FORWARD {
                counter packets 3864328 bytes 2313882911 jump DOCKER-CT
                counter packets 2425206 bytes 1006477160 jump DOCKER-ISOLATION-STAGE-1
                counter packets 2425206 bytes 1006477160 jump DOCKER-BRIDGE
                iifname "docker0" counter packets 0 bytes 0 accept
                iifname "br-a154210e39a0" counter packets 161 bytes 18108 accept
                iifname "br-684e55616329" counter packets 0 bytes 0 accept
                iifname "br-cd09b9c6e58c" counter packets 0 bytes 0 accept
                iifname "br-a906a8f9d766" counter packets 0 bytes 0 accept


# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER-USER  0    --  0.0.0.0/0            0.0.0.0/0           
DOCKER-FORWARD  0    --  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (13 references)
target     prot opt source               destination         
ACCEPT     6    --  0.0.0.0/0            172.16.4.5           tcp dpt:30041
ACCEPT     6    --  0.0.0.0/0            172.16.12.2          tcp dpt:30059
ACCEPT     17   --  0.0.0.0/0            172.16.11.2          udp dpt:51413
ACCEPT     6    --  0.0.0.0/0            172.16.11.2          tcp dpt:51413
ACCEPT     6    --  0.0.0.0/0            172.16.11.2          tcp dpt:30024
ACCEPT     6    --  0.0.0.0/0            172.16.9.2           tcp dpt:30068
ACCEPT     6    --  0.0.0.0/0            172.16.7.2           tcp dpt:8920
ACCEPT     6    --  0.0.0.0/0            172.16.7.2           tcp dpt:8096
ACCEPT     6    --  0.0.0.0/0            172.16.8.3           tcp dpt:30103
ACCEPT     6    --  0.0.0.0/0            172.16.5.3           tcp dpt:30143
ACCEPT     6    --  0.0.0.0/0            172.16.5.3           tcp dpt:30142
ACCEPT     6    --  0.0.0.0/0            172.16.6.2           tcp dpt:30266
ACCEPT     6    --  0.0.0.0/0            172.16.10.2          tcp dpt:30236
ACCEPT     6    --  0.0.0.0/0            172.16.1.2           tcp dpt:9090
ACCEPT     6    --  0.0.0.0/0            172.16.1.2           tcp dpt:8923
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
DROP       0    --  0.0.0.0/0            0.0.0.0/0   

sudo tcpdump -ni eno1 host 192.168.0.1 and port 80
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
19:37:55.231022 IP 192.168.0.251.52150 > 192.168.0.1.80: Flags [S], seq 3307617049, win 64240, options [mss 1460,sackOK,TS val 3456460108 ecr 0,nop,wscale 7], length 0
19:37:56.236249 IP 192.168.0.251.52150 > 192.168.0.1.80: Flags [S], seq 3307617049, win 64240, options [mss 1460,sackOK,TS val 3456461114 ecr 0,nop,wscale 7], length 0
19:37:57.260254 IP 192.168.0.251.52150 > 192.168.0.1.80: Flags [S], seq 3307617049, win 64240, options [mss 1460,sackOK,TS val 3456462138 ecr 0,nop,wscale 7], length 0
19:37:58.284202 IP 192.168.0.251.52150 > 192.168.0.1.80: Flags [S], seq 3307617049, win 64240, options [mss 1460,sackOK,TS val 3456463162 ecr 0,nop,wscale 7], length 0
19:37:59.308243 IP 192.168.0.251.52150 > 192.168.0.1.80: Flags [S], seq 3307617049, win 64240, options [mss 1460,sackOK,TS val 3456464186 ecr 0,nop,wscale 7], length 0
19:38:00.336252 IP 192.168.0.251.52150 > 192.168.0.1.80: Flags [S], seq 3307617049, win 64240, options [mss 1460,sackOK,TS val 3456465214 ecr 0,nop,wscale 7], length 0
19:38:02.348263 IP 192.168.0.251.52150 > 192.168.0.1.80: Flags [S], seq 3307617049, win 64240, options [mss 1460,sackOK,TS val 3456467226 ecr 0,nop,wscale 7], length 0
19:38:06.412185 IP 192.168.0.251.52150 > 192.168.0.1.80: Flags [S], seq 3307617049, win 64240, options [mss 1460,sackOK,TS val 3456471290 ecr 0,nop,wscale 7], length 0

#4 from above worked for the normal amount of time (~24 hours)

12

syns leave and nothing comes back. OUTPUT is ACCEPT so the drop is probably on the isp box side (or conntrack choking on all those docker bridges).

id try next:
-stop apps for a few mins (or systemctl stop docker) then curl https://1.1.1.1 again. that many bridges can poison these isp routers after a day
-tcpdump -ni eno1 host 1.1.1.1 while curling, confirm syn leaves and zero replies
-from a working laptop: arp -a and check what mac/ip the router has for the nas
-dumb switch between nas and isp box, or a different lan port. if it sticks past 24h the isp port/mac filter is the culprit
-longer fix: put ur own router in front and set the isp box to bridge/passthrough

13

I’ll try those when i get home, but yeah i’m in the process of switching it out with a pi 3b+

14

ok I managed to track down ssh access via another lan device that hasn’t had these issues.

Disabling apps has not allowed it to recover. arp -a shows both address it’s configured with (one outside dhcp range, the one inside is reserved)

can confirm no ACK to 1.1.1.1 (from experiance, this lan device doesn’t have root access)

15

apps off and still no ack means the isp box is probably still blackholing that mac/ip pair.

two addresses on the same nic (one outside dhcp, one reserved) can keep these isp routers confused. id drop the out-of-range one so only the reserved ip stays, bounce the link, then curl 1.1.1.1 again.

if it still dies, change the nic mac once (or move the cable to another port on the isp box) and reserve the new mac. last resort is putting a real router in front of that isp gateway like u were thinking with the pi.