Connectivity Issue

TrueNAS General
1

I upgraded from Core to the latest Scale recently and have been facing a number of connectivity issues below.

I cannot connect to update server and errors include:

image
image508×600 35.1 KB

image
image1490×259 12.6 KB

Typing the following in shell produce the same errors:

openssl s_client -connect update.ixsystems.com:443 -showcert
40C722C5A27F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:…/crypto/bio/bio_addr.c:738:Temporary failure in name resolution
connect:errno=11

I have setup a static IP as per below and also assigned it in my router.

image
image1397×717 41 KB

timedatectl:

image
image649×180 19.7 KB

I also experience connectivity issues in various apps. For example, Tailscale will show my server to not be connected periodically for extended time.

I am completely stumped. Again, this only happened after upgrading from CORE to SCALE. Thank you.

2

Is that blurred nameserver ip something like a pihole or another local dns resolver?
Have you tried other nameservers like 1.1.1.1 or 8.8.8.8?

3

The nameserver IP is the same as my default route. I am letting my modem/router resolve the DNS (which is following how a Tailscale app setup is required for subnet routing). I know my router is using the ISP’s nameserver and not 1.1.1.1 or 8.8.8.8.

I can try 1.x or 8.x but if it works it’ll also screw up my Tailscale. In that case I guess I can try to change the DNS on my router to 1.x or 8.x. ?

Forgot to mention that this connectivity is intermittent. Sometimes it connects to the server and sometimes it doesn’t, but most often not.

4

Changing the DNS to 1.1.1.1 both in modem and TrueNAS and still the same error:

Error: Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/aiohttp/connector.py”, line 1317, in _create_direct_connection
hosts = await self._resolve_host(host, port, traces=traces)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/aiohttp/connector.py”, line 971, in _resolve_host
return await asyncio.shield(resolved_host_task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/aiohttp/connector.py”, line 1002, in _resolve_host_with_throttle
addrs = await self._resolver.resolve(host, port, family=self._family)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/aiohttp/resolver.py”, line 38, in resolve
infos = await self._loop.getaddrinfo(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/asyncio/base_events.py”, line 868, in getaddrinfo
return await self.run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/socket.py”, line 962, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py”, line 323, in process_method_call
result = await method.call(app, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py”, line 52, in call
result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 911, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 731, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 624, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 178, in nf
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/update.py”, line 105, in get_trains
trains_data = self.middleware.call_sync(‘update.get_trains_data’)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1030, in call_sync
return self.run_coroutine(methodobj(*prepared_call.args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1070, in run_coroutine
return fut.result()
^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/_base.py”, line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/_base.py”, line 401, in _get_result
raise self.exception
File "/usr/lib/python3/dist-packages/middlewared/plugins/update/trains.py", line 63, in get_trains_data
**(await self.fetch(f"{self.update_srv}/trains.json"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/update/trains.py", line 25, in fetch
async with client.get(url) as resp:
File “/usr/lib/python3/dist-packages/aiohttp/client.py”, line 1359, in aenter
self._resp: _RetType = await self._coro
^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/aiohttp/client.py”, line 663, in _request
conn = await self._connector.connect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/aiohttp/connector.py”, line 563, in connect
proto = await self._create_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/aiohttp/connector.py”, line 1032, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/aiohttp/connector.py”, line 1323, in _create_direct_connection
raise ClientConnectorDNSError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorDNSError: Cannot connect to host update.ixsystems.com:443 ssl:default [Temporary failure in name resolution]

Tried ifconfig -a and see extra networks. Is this normal?

br-be1c5cb80083: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.16.1.1 netmask 255.255.255.0 broadcast 172.16.1.255
inet6 fdd0:0:0:1::1 prefixlen 64 scopeid 0x0
inet6 fe80::42:30ff:fe11:ae67 prefixlen 64 scopeid 0x20
ether 02:42:30:11:ae:67 txqueuelen 0 (Ethernet)
RX packets 54 bytes 4314 (4.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 106 bytes 169712 (165.7 KiB)
TX errors 0 dropped 2 overruns 0 carrier 0 collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.16.0.1 netmask 255.255.255.0 broadcast 172.16.0.255
inet6 fdd0::1 prefixlen 64 scopeid 0x0
ether 02:42:f1:e7:c0:5f txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 6 overruns 0 carrier 0 collisions 0

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.201 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fe80::1a03:73ff:fec4:843e prefixlen 64 scopeid 0x20
ether 18:03:73:c4:84:3e txqueuelen 1000 (Ethernet)
RX packets 216652 bytes 252629603 (240.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 45430 bytes 19681093 (18.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 20 memory 0xdcb00000-dcb20000

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 165999 bytes 28036011 (26.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 165999 bytes 28036011 (26.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vethdb878dc: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::dc5a:3fff:fe97:edd3 prefixlen 64 scopeid 0x20
ether de:5a:3f:97:ed:d3 txqueuelen 0 (Ethernet)
RX packets 54 bytes 5070 (4.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 146 bytes 179956 (175.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

5

Complete shot in the dark; I know you can’t reach ipv4, but - for whatever reason, at least toggling, sometimes this fixes network issues for me:
-what happens if you check ‘autoconfigure IPv6’ for ‘eno1’?

Otherwise, I guess I’d ask, what are results of:

ping 1.1.1.1

ping 2001:4860:4860::8888

ping google.ca

ping github.com

That’ll give a good mix connectivity to ipv4, ipv6, and dns for both as well. Maybe that’ll help narrow things down.

1 Like
6

Thanks for the steps.

I checked IPv6 on and off and still get dropped packets or errors like this with IPv6 checked:

TimeoutError(): Automatic update check failed. Please check system network settings.

or

ping: google.ca: Name or service not known

and 2001:4860:4860::8888 gives me

ping: connect: Network is unreachable

Pinging without IPv6 checked gives this intermittently

ping: google.ca: Temporary failure in name resolution

This is what I get with continous ping:

--- google.ca ping statistics ---
153 packets transmitted, 94 received, 38.5621% packet loss, time 153527ms
rtt min/avg/max/mdev = 1.267/2.061/2.774/0.335 ms

No issues with other computers on my network.

7

I’m guessing these results are with toggling ‘IPv6’ - what happens if you leave it on?

1 Like
8

Not any better :frowning:

--- google.ca ping statistics ---
263 packets transmitted, 130 received, 50.5703% packet loss, time 265355ms
rtt min/avg/max/mdev = 1.719/2.470/3.563/0.282 ms

It was rock solid on TrueNAS Core so not sure if something happened from the multiple migrations to the latest Scale.

9

wtf… 50% packet loss? I mean… 100% would be so much easier to figure out.

Alright, so it is only affecting your NAS & only after you went from Core to Scale… What was your upgrade path? Was it Core straight to latest scale? I think iX recommends a specific version path when upgrading from Core to Scale…

It could be worth investigating a config backup & a clean install.

1 Like
10

I followed the recommended path to upgrade to Scale. I also now recalled doing a clean install of Electric Eel (24.10) before upgraded to Fangtooth (25.04) due to this issue, using config backup. Would the config backup from Core to Scale bring any issues?

I have set a static IP on my modem/router, and it seems to detect the same MAC between Core and Scale. So unless the modem/router is acting up…

11

…I’m nearing the end of my usefulness - I’d try to disable anything (apps, vms, smb, bridges, etc.) until we only have eno1 active. If it was my nas having the issue, I’d personally have no issues manually deleting interfaces until only eno1 is active.

With only eno1 active, I’d then focus on figuring out wtf I need to get ipv4 and ipv6 work on it with 100% success. Afterwards I’d slowly turn things up until I again run into issues.

With this all being said, I’m out of specific ideas & wish you the best, but you need someone smarter than me to help remotely any further :frowning:

1 Like
12

Well, you led me in the right direction!!!

I pinged the router and see the same dropped packet issues. So I decided to look through my router/modem with 20+ unique connected devices to see if there could be a conflict and found that the router assigned another device with the same static IP. I have no idea why the router would do that… I set that device with a different static IP and hope the router won’t decide to keep producing conflicts…

Now I also realized why one of my Amazon echoes (the conflict) that I rarely use would have issue as well!

THANKS A LOT.

1 Like