Help! Failed to sync TRUENAS catalog, and apps have no access to internet (Electric Eel)

EthanTheGoondu · November 23, 2024, 4:51pm

I have come across the following error in my system:

[EFAULT] Failed to clone 'https://github.com/truenas/apps' repository at '/mnt/.ix-apps/truenas_catalog' destination: Cloning into '/mnt/.ix-apps/truenas_catalog'......

I am unable to use git to clone any repo from github, I cannot curl the https site of github, but I can ping and dig github.com (So I assume DNS works fine). Additionally, my apps seem to have no access to the internet (Except for tailscale which I can use for some reason remotely).

Other things I have tried:

Checked the date via the date command, appears to be the correct date
Tried a different network port
Set the nameserver to my router and the cloudflare/google ones
Unsetting pool and setting it back again
Plenty of reboots

If anyone has more suggestions on what to try or if anyone knows what the problem could be, let me know, thanks!

SmallBarky · November 23, 2024, 6:35pm

Do you have a firewall blocking access? This is usually all networking related

EthanTheGoondu · November 23, 2024, 6:50pm

I left my router with default settings (and my server has worked on this network before), and if the router does have a firewall affecting this then I think I should also have issues with other devices on the same network? So far my other devices seem fine in git cloning any what not. Not sure how to determine if it’s the router’s fault.

Also, might be relevant, my router is a nokia ont xs-2426x-a. I think it’s pretty rare and I only have it because my isp provided me with it.

bacon · November 23, 2024, 6:55pm

Would be helpful if you could post the output of the following commands (run them in System → Shell):

curl -vv google.com
ip route
ip addr

EDIT:
also try the following (in case you’re having ipv6 issues):

curl -vv -4 google.com

EthanTheGoondu · November 23, 2024, 7:02pm

Certainly!

curl -vv google.com
*   Trying [2404:6800:4003:c11::71]:80...
*   Trying 142.251.12.100:80...
* Connected to google.com (142.251.12.100) port 80 (#0)
> GET / HTTP/1.1
> Host: google.com
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Location: http://www.google.com/
< Content-Type: text/html; charset=UTF-8
< Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-N_ATvdBoXz6oXd9Rwlicdw' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< Date: Sat, 23 Nov 2024 18:57:24 GMT
< Expires: Mon, 23 Dec 2024 18:57:24 GMT
< Cache-Control: public, max-age=2592000
< Server: gws
< Content-Length: 219
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
* Connection #0 to host google.com left intact

ip route               
default via 192.168.10.1 dev eno1 proto static 
172.16.0.0/24 dev docker0 proto kernel scope link src 172.16.0.1 linkdown 
172.16.1.0/24 dev br-a19d6f587053 proto kernel scope link src 172.16.1.1 
172.16.2.0/24 dev br-4855a477731a proto kernel scope link src 172.16.2.1 
172.16.5.0/24 dev br-239a3ea16c76 proto kernel scope link src 172.16.5.1 
172.16.6.0/24 dev br-466e0afa6a18 proto kernel scope link src 172.16.6.1 
172.16.7.0/24 dev br-c1e1a9b48af6 proto kernel scope link src 172.16.7.1 
172.16.8.0/24 dev br-0c24d8162c23 proto kernel scope link src 172.16.8.1 
172.16.9.0/24 dev br-6be5a4b8de73 proto kernel scope link src 172.16.9.1 
172.16.11.0/24 dev br-2cc9a940332f proto kernel scope link src 172.16.11.1 
172.16.12.0/24 dev br-8663f0d8007d proto kernel scope link src 172.16.12.1 
172.16.13.0/24 dev br-4e1c4c6337f5 proto kernel scope link src 172.16.13.1 
172.16.15.0/24 dev br-949f146aa77c proto kernel scope link src 172.16.15.1 
172.16.16.0/24 dev br-bce25433d49a proto kernel scope link src 172.16.16.1 
192.168.10.0/24 dev eno1 proto kernel scope link src 192.168.10.110

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1c:86:0b:2d:02:8a brd ff:ff:ff:ff:ff:ff
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 10:c3:7b:a1:db:c4 brd ff:ff:ff:ff:ff:ff
    altname enp0s25
    inet 192.168.10.110/24 brd 192.168.10.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 2406:3003:2005:fa3:12c3:7bff:fea1:dbc4/64 scope global dynamic mngtmpaddr 
       valid_lft 172700sec preferred_lft 172700sec
    inet6 fe80::12c3:7bff:fea1:dbc4/64 scope link 
       valid_lft forever preferred_lft forever
4: br-c1e1a9b48af6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:d3:7c:5a:18 brd ff:ff:ff:ff:ff:ff
    inet 172.16.7.1/24 brd 172.16.7.255 scope global br-c1e1a9b48af6
       valid_lft forever preferred_lft forever
    inet6 fe80::42:d3ff:fe7c:5a18/64 scope link 
       valid_lft forever preferred_lft forever
5: br-a19d6f587053: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:7e:32:a6:5a brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.1/24 brd 172.16.1.255 scope global br-a19d6f587053
       valid_lft forever preferred_lft forever
    inet6 fe80::42:7eff:fe32:a65a/64 scope link 
       valid_lft forever preferred_lft forever
6: br-466e0afa6a18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:72:d9:5f:0d brd ff:ff:ff:ff:ff:ff
    inet 172.16.6.1/24 brd 172.16.6.255 scope global br-466e0afa6a18
       valid_lft forever preferred_lft forever
    inet6 fe80::42:72ff:fed9:5f0d/64 scope link 
       valid_lft forever preferred_lft forever
7: br-4e1c4c6337f5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:48:e1:b4:d5 brd ff:ff:ff:ff:ff:ff
    inet 172.16.13.1/24 brd 172.16.13.255 scope global br-4e1c4c6337f5
       valid_lft forever preferred_lft forever
    inet6 fe80::42:48ff:fee1:b4d5/64 scope link 
       valid_lft forever preferred_lft forever
8: br-0c24d8162c23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:8f:80:4a:2c brd ff:ff:ff:ff:ff:ff
    inet 172.16.8.1/24 brd 172.16.8.255 scope global br-0c24d8162c23
       valid_lft forever preferred_lft forever
    inet6 fe80::42:8fff:fe80:4a2c/64 scope link 
       valid_lft forever preferred_lft forever
9: br-bce25433d49a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:3f:49:49:c2 brd ff:ff:ff:ff:ff:ff
    inet 172.16.16.1/24 brd 172.16.16.255 scope global br-bce25433d49a
       valid_lft forever preferred_lft forever
    inet6 fe80::42:3fff:fe49:49c2/64 scope link 
       valid_lft forever preferred_lft forever
10: br-239a3ea16c76: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:b9:09:d0:7d brd ff:ff:ff:ff:ff:ff
    inet 172.16.5.1/24 brd 172.16.5.255 scope global br-239a3ea16c76
       valid_lft forever preferred_lft forever
    inet6 fe80::42:b9ff:fe09:d07d/64 scope link 
       valid_lft forever preferred_lft forever
11: br-4855a477731a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:77:53:58:51 brd ff:ff:ff:ff:ff:ff
    inet 172.16.2.1/24 brd 172.16.2.255 scope global br-4855a477731a
       valid_lft forever preferred_lft forever
    inet6 fe80::42:77ff:fe53:5851/64 scope link 
       valid_lft forever preferred_lft forever
12: br-6be5a4b8de73: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:d5:59:95:d2 brd ff:ff:ff:ff:ff:ff
    inet 172.16.9.1/24 brd 172.16.9.255 scope global br-6be5a4b8de73
       valid_lft forever preferred_lft forever
    inet6 fe80::42:d5ff:fe59:95d2/64 scope link 
       valid_lft forever preferred_lft forever
13: br-8663f0d8007d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:d6:61:88:71 brd ff:ff:ff:ff:ff:ff
    inet 172.16.12.1/24 brd 172.16.12.255 scope global br-8663f0d8007d
       valid_lft forever preferred_lft forever
    inet6 fe80::42:d6ff:fe61:8871/64 scope link 
       valid_lft forever preferred_lft forever
14: br-949f146aa77c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:82:e4:4f:53 brd ff:ff:ff:ff:ff:ff
    inet 172.16.15.1/24 brd 172.16.15.255 scope global br-949f146aa77c
       valid_lft forever preferred_lft forever
    inet6 fe80::42:82ff:fee4:4f53/64 scope link 
       valid_lft forever preferred_lft forever
16: br-2cc9a940332f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:b3:a1:3c:b3 brd ff:ff:ff:ff:ff:ff
    inet 172.16.11.1/24 brd 172.16.11.255 scope global br-2cc9a940332f
       valid_lft forever preferred_lft forever
    inet6 fe80::42:b3ff:fea1:3cb3/64 scope link 
       valid_lft forever preferred_lft forever
17: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:f9:9e:9b:a3 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.1/24 brd 172.16.0.255 scope global docker0
       valid_lft forever preferred_lft forever
19: veth04b8c22@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-2cc9a940332f state UP group default 
    link/ether a2:7c:c1:4e:b0:a1 brd ff:ff:ff:ff:ff:ff link-netnsid 6
    inet6 fe80::a07c:c1ff:fe4e:b0a1/64 scope link 
       valid_lft forever preferred_lft forever
21: vethb297b04@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-8663f0d8007d state UP group default 
    link/ether 92:a8:1b:62:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::90a8:1bff:fe62:4/64 scope link 
       valid_lft forever preferred_lft forever
23: vetha939a3c@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-4e1c4c6337f5 state UP group default 
    link/ether 96:9d:56:6f:8d:58 brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::949d:56ff:fe6f:8d58/64 scope link 
       valid_lft forever preferred_lft forever
25: veth2a32c47@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-0c24d8162c23 state UP group default 
    link/ether 06:b4:6c:0c:8c:72 brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::4b4:6cff:fe0c:8c72/64 scope link 
       valid_lft forever preferred_lft forever
27: veth20f48d6@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-4855a477731a state UP group default 
    link/ether 3e:71:e8:c8:69:5c brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::3c71:e8ff:fec8:695c/64 scope link 
       valid_lft forever preferred_lft forever
29: vethab0af56@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-a19d6f587053 state UP group default 
    link/ether 6e:c2:e8:12:46:a4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::6cc2:e8ff:fe12:46a4/64 scope link 
       valid_lft forever preferred_lft forever
31: vethdc67df2@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c1e1a9b48af6 state UP group default 
    link/ether 3e:bf:4d:d2:7c:76 brd ff:ff:ff:ff:ff:ff link-netnsid 10
    inet6 fe80::3cbf:4dff:fed2:7c76/64 scope link 
       valid_lft forever preferred_lft forever
33: veth7917c2a@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6be5a4b8de73 state UP group default 
    link/ether 6e:b6:94:d0:79:de brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::6cb6:94ff:fed0:79de/64 scope link 
       valid_lft forever preferred_lft forever
35: veth9fce6a1@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-466e0afa6a18 state UP group default 
    link/ether 0a:24:f6:41:39:d4 brd ff:ff:ff:ff:ff:ff link-netnsid 8
    inet6 fe80::824:f6ff:fe41:39d4/64 scope link 
       valid_lft forever preferred_lft forever
37: veth2ba44f8@if36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-949f146aa77c state UP group default 
    link/ether 6a:3e:c8:83:65:37 brd ff:ff:ff:ff:ff:ff link-netnsid 7
    inet6 fe80::683e:c8ff:fe83:6537/64 scope link 
       valid_lft forever preferred_lft forever
39: veth601f2f7@if38: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-bce25433d49a state UP group default 
    link/ether 6e:15:75:dc:ff:cd brd ff:ff:ff:ff:ff:ff link-netnsid 9
    inet6 fe80::6c15:75ff:fedc:ffcd/64 scope link 
       valid_lft forever preferred_lft forever
41: vethcc6f816@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-239a3ea16c76 state UP group default 
    link/ether de:7c:17:ce:fc:85 brd ff:ff:ff:ff:ff:ff link-netnsid 11
    inet6 fe80::dc7c:17ff:fece:fc85/64 scope link 
       valid_lft forever preferred_lft forever

curl -vv -4 google.com
*   Trying 142.251.12.139:80...
* Connected to google.com (142.251.12.139) port 80 (#0)
> GET / HTTP/1.1
> Host: google.com
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Location: http://www.google.com/
< Content-Type: text/html; charset=UTF-8
< Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-bJJ6n44qQYgZdgrYrZttaA' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< Date: Sat, 23 Nov 2024 19:01:01 GMT
< Expires: Mon, 23 Dec 2024 19:01:01 GMT
< Cache-Control: public, max-age=2592000
< Server: gws
< Content-Length: 219
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
* Connection #0 to host google.com left intact

As you might guess my machine is 10.110, my router is 10.1, and I’m using the eno1 interface. I appreciate the help!

bacon · November 23, 2024, 7:05pm

Looks like you have internet connectivty.

Can you try a https url next:

curl -vv https://github.com > /dev/null

EthanTheGoondu · November 23, 2024, 7:15pm

Yup, it’s very odd. My apps still don’t seem to work, and I know for sure curling the https site will lead to a failure. Here is the output:

curl -vv https://github.com > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 20.205.243.166:443...
* Connected to github.com (20.205.243.166) port 443 (#0)
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3137 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=github.com
*  start date: Mar  7 00:00:00 2024 GMT
*  expire date: Mar  7 23:59:59 2025 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo ECC Domain Validation Secure Server CA
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: github.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x55fb7b609ce0)
} [5 bytes data]
> GET / HTTP/2
> Host: github.com
> user-agent: curl/7.88.1
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:01:50 --:--:--     0* Recv failure: Connection reset by peer
* OpenSSL SSL_read: Connection reset by peer, errno 104
* Failed receiving HTTP2 data
  0     0    0     0    0     0      0      0 --:--:--  0:01:51 --:--:--     0
* Connection #0 to host github.com left intact
curl: (56) Recv failure: Connection reset by peer

On my laptop it looks like this:

curl -vv https://github.com > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 20.205.243.166:443...
* Connected to github.com (20.205.243.166) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [315 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [3137 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=github.com
*  start date: Mar  7 00:00:00 2024 GMT
*  expire date: Mar  7 23:59:59 2025 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo ECC Domain Validation Secure Server CA
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: github.com]
* h2 [:path: /]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x14280c600)
> GET / HTTP/2
> Host: github.com
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/2 200
< server: GitHub.com
< date: Sat, 23 Nov 2024 19:10:07 GMT
< content-type: text/html; charset=utf-8
< vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Language, Accept-Encoding, Accept, X-Requested-With
< content-language: en-US
< etag: W/"b6d537d8e2e603df39a5bbfd370579c5"
< cache-control: max-age=0, private, must-revalidate
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'; base-uri 'self'; child-src github.com/assets-cdn/worker/ github.com/webpack/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com github.githubassets.com edge.fullstory.com rs.fullstory.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com www.youtube-nocookie.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com github.githubassets.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.com/assets-cdn/worker/ github.com/webpack/ github.com/assets/ gist.github.com/assets-cdn/worker/
< set-cookie: [Redacted]
< set-cookie: [Redacted]
< set-cookie: [Redacted]
< accept-ranges: bytes
< x-github-request-id: E4CA:8171D:108ED3:12E115:67422895
<
{ [4101 bytes data]
100  150k    0  150k    0     0  1928k      0 --:--:-- --:--:-- --:--:-- 2088k
* Connection #0 to host github.com left intact

bacon · November 23, 2024, 7:43pm

That is indeed odd. The handshake completed without issues but you are getting a connection reset (TCP RST) shortly after.

Can you check if curl -vvv https://google.com leads to the same “Connection reset by peer” error?

EthanTheGoondu · November 24, 2024, 3:13am

Here it is:

curl -vvv https://google.com
*   Trying [2404:6800:4003:c1a::71]:443...
* Connected to google.com (2404:6800:4003:c1a::71) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.google.com
*  start date: Oct 21 08:36:57 2024 GMT
*  expire date: Jan 13 08:36:56 2025 GMT
*  subjectAltName: host "google.com" matched cert's "google.com"
*  issuer: C=US; O=Google Trust Services; CN=WR2
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: google.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5600b8a10ce0)
> GET / HTTP/2
> Host: google.com
> user-agent: curl/7.88.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 301 
< location: https://www.google.com/
< content-type: text/html; charset=UTF-8
< content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-r-qIQcCvJs17TZpibm3qSw' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< date: Sun, 24 Nov 2024 03:07:00 GMT
< expires: Tue, 24 Dec 2024 03:07:00 GMT
< cache-control: public, max-age=2592000
< server: gws
< content-length: 220
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
* Connection #0 to host google.com left intact

I also could get a response for this:

curl -vvv https://example.com
*   Trying [2606:2800:21f:cb07:6820:80da:af6b:8b2c]:443...
* Connected to example.com (2606:2800:21f:cb07:6820:80da:af6b:8b2c) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=Los Angeles; O=InternetCorporationforAssignedNamesandNumbers; CN=www.example.org
*  start date: Jan 30 00:00:00 2024 GMT
*  expire date: Mar  1 23:59:59 2025 GMT
*  subjectAltName: host "example.com" matched cert's "example.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: example.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x561e206fece0)
> GET / HTTP/2
> Host: example.com
> user-agent: curl/7.88.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200 
< accept-ranges: bytes
< age: 203098
< cache-control: max-age=604800
< content-type: text/html; charset=UTF-8
< date: Sun, 24 Nov 2024 03:08:14 GMT
< etag: "3147526947+gzip"
< expires: Sun, 01 Dec 2024 03:08:14 GMT
< last-modified: Thu, 17 Oct 2019 07:18:26 GMT
< server: ECAcc (sed/589E)
< vary: Accept-Encoding
< x-cache: HIT
< content-length: 1256
< 
<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <style type="text/css">
    body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
        
    }
    div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
    }
    a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
    @media (max-width: 700px) {
        div {
            margin: 0 auto;
            width: auto;
        }
    }
    </style>    
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>
* Connection #0 to host example.com left intact

bacon · November 24, 2024, 9:16am

The apps not having internet access could be a different issue. But could you tell us which apps are having problems connecting to the internet and what the error messages are?

Also could you tell us when the issue first appeared? And were there any hardware or software changes that correlate with the issue appearing?

From what I’ve seen so far I don’t think the issue is on the truenas host itself. It only seems to affect github so far. But it’s difficult to tell with the limited information so far.

Does your laptop use a VPN or proxy server to connect to the internet? Can you compare the WAN IP of your laptop with that of truenas? You can do that by doing the following:

Open https://ifconfig.co on your laptop and note the IP Address field
Execute curl ifconfig.co on the TrueNAS host

You do not have to post the IP here, just compare the two.

If the issue is external, it might resolve itself. Try curl https://github.com from time to time to verify the issue is still present.

If you have a proxy server available, you can try connecting via proxy.

EthanTheGoondu · November 24, 2024, 1:00pm

IP addresses are the same, I got no VPN in use. As far as I know, this has been happening since 2 days ago (Hard to find proper records)

For hardware, there have been no changes since the last time everything was working (I did end up changing NICs for troubleshooting this issue specifically but that did not help so I’m back to where I started hardware wise), as for apps qbitorrent is able to find peers but unable to make any downloads (I am using the ubuntu distro iso torrent as a test for this), the *arr suites are unable to search anything (They just tell me no results for anything when I try to search), and metube hangs when loading after I paste a url in. Only thing that seems to work is tailscale for some reason (so SMB via tailscale works too), I believe it’s using wireguard so maybe that protocol has no issues?

Also to update, my successful entry into google seems to be a fluke since now when I try again I am unable to get through:

curl -vvv https://google.com
*   Trying 172.253.118.100:443...
* Connected to google.com (172.253.118.100) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.google.com
*  start date: Oct 21 08:36:57 2024 GMT
*  expire date: Jan 13 08:36:56 2025 GMT
*  subjectAltName: host "google.com" matched cert's "google.com"
*  issuer: C=US; O=Google Trust Services; CN=WR2
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: google.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x56425344ece0)
> GET / HTTP/2
> Host: google.com
> user-agent: curl/7.88.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Recv failure: Connection reset by peer
* OpenSSL SSL_read: Connection reset by peer, errno 104
* Failed receiving HTTP2 data
* Connection #0 to host google.com left intact
curl: (56) Recv failure: Connection reset by pee

This makes me extra confused (I did reboot a few times since I last tried google), but I also was able to dump a pcap which shows client packets being unable to get through (Or the response from the server is not coming back) casing a few retransmits:

I think I do have access to a proxy so I might try that, besides that I might get pcaps from my other apps or even factory reset my ONR. I think I might try reassigning my own ip within my network too (or try using DHCP) since the ONR I have is rather new. I appreciate the help!

EthanTheGoondu · November 24, 2024, 5:31pm

I have some news. I have tried resetting the configuration for TrueNAS and everything works now. My current theory is my ONR is causing issues with my device. Currently I am unable to change the static ip to any other address, so I think the ONR is adamantly enforcing the ipv4 reservation I made for my system despite me deleting it (I wanted to try DHCP for a new IP). I will now either factory reset the ONR and load in the backup config I made, or start from scratch and slowly restore my configs in the machine.

EthanTheGoondu · November 25, 2024, 2:39am

Another update. Since I reset my TrueNAS config, I was assigned 10.90 for IP as I used dhcp. When I tried to changed it to 10.110 (My previous IP), everything breaks again, so I’m inclined to blame my ONR on this. A factory reset of my ONR can confirm this, so I will try that

EthanTheGoondu · November 27, 2024, 12:52pm

I have solved the issue! Turns out my modem has some issues with DHCP reservation, where I deleted it but it still has issues routing packets back to my NAS sometimes. When I did a factory reset and didn’t make a DHCP reservation, everything works correctly, apologies to everyone for the false alarm! But next time for others facing this issue, you might want to check on your router!