Hi,
Firstly, thanks for your help in posting @Johnny_Fartpants and @Captain_Morgan!
I am using TrueNAS Scale Dragonfish 24.04.2.3 - I was recently caught in a long-term power outage (thanks to hurricanes). Unfortunately, I was on vacation then and couldn’t power down the system beforehand.
Upon return, I found the system on with all its subsystems (k3s, SMB, NFS, SSH, etc.) working as expected. However, I am unable to access the WebUI via HTTPS or HTTP. I’m not 100% sure which commands to run to check most, but I ran a netstat and saw that 443 isn’t even in LISTENING state, but 81 is (the port I decided to use HTTP on since it was unused. There was a reason for switching to 81, but don’t ask what it was; I did it over a year ago when I was first playing with the system and never fixed it because I use HTTPS normally.)
(No info could be read for "-p": geteuid()=950 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 10.10.1.3:179 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:32401 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:32600 0.0.0.0:* LISTEN -
tcp 0 0 10.10.1.3:5357 0.0.0.0:* LISTEN -
tcp 0 0 10.10.2.3:5357 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:53853 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:50051 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN -
tcp 0 0 10.10.1.3:50051 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:10010 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:43293 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5901 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:33337 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:6444 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:6999 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:81 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:33461 0.0.0.0:* LISTEN -
tcp6 0 0 :::32400 :::* LISTEN -
tcp6 0 0 :::30004 :::* LISTEN -
tcp6 0 0 :::29644 :::* LISTEN -
tcp6 0 0 :::29643 :::* LISTEN -
tcp6 0 0 :::29642 :::* LISTEN -
tcp6 0 0 :::29653 :::* LISTEN -
tcp6 0 0 :::29652 :::* LISTEN -
tcp6 0 0 :::58421 :::* LISTEN -
tcp6 0 0 :::20244 :::* LISTEN -
tcp6 0 0 ::1:179 :::* LISTEN -
tcp6 0 0 :::44525 :::* LISTEN -
tcp6 0 0 :::10250 :::* LISTEN -
tcp6 0 0 :::43907 :::* LISTEN -
tcp6 0 0 :::9696 :::* LISTEN -
tcp6 0 0 :::6443 :::* LISTEN -
tcp6 0 0 :::2049 :::* LISTEN -
tcp6 0 0 :::445 :::* LISTEN -
tcp6 0 0 :::139 :::* LISTEN -
tcp6 0 0 :::111 :::* LISTEN -
tcp6 0 0 :::81 :::* LISTEN -
tcp6 0 0 :::53 :::* LISTEN -
tcp6 0 0 :::21 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
Also, I have tried switching to one of the older versions via GRUB (24.04.1.1) but that results in everything becoming unusable. When I say unusable I mean that the subsystem services (k3s and the like) all do not come online.
Here are a couple of additional commands I just thought to re-run:
admin@truenas[~]$ sudo systemctl status nginx
[sudo] password for admin:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
Drop-In: /etc/systemd/system/nginx.service.d
└─override.conf
Active: active (running) since Tue 2024-10-15 20:16:40 EDT; 13h ago
Docs: man:nginx(8)
Process: 9763 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
Main PID: 4746 (nginx)
Tasks: 2 (limit: 76368)
Memory: 3.6M
CPU: 83ms
CGroup: /system.slice/nginx.service
├─4746 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
└─9764 "nginx: worker process"
Oct 15 20:16:40 truenas systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
Oct 15 20:16:40 truenas systemd[1]: Started nginx.service - A high performance web server and a reverse proxy server.
Oct 15 20:17:13 truenas systemd[1]: Reloading nginx.service - A high performance web server and a reverse proxy server...
Oct 15 20:17:13 truenas nginx[8798]: 2024/10/15 20:17:13 [notice] 8798#8798: signal process started
Oct 15 20:17:13 truenas systemd[1]: Reloaded nginx.service - A high performance web server and a reverse proxy server.
Oct 15 20:17:22 truenas systemd[1]: Reloading nginx.service - A high performance web server and a reverse proxy server...
Oct 15 20:17:22 truenas nginx[9763]: 2024/10/15 20:17:22 [notice] 9763#9763: signal process started
Oct 15 20:17:22 truenas systemd[1]: Reloaded nginx.service - A high performance web server and a reverse proxy server.
admin@truenas[~]$ sudo systemctl status middlewared
● middlewared.service - TrueNAS Middleware
Loaded: loaded (/lib/systemd/system/middlewared.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-10-15 20:15:31 EDT; 13h ago
Main PID: 1986 (asyncio_loop)
Tasks: 71 (limit: 76368)
Memory: 1.4G
CPU: 14min 50.252s
CGroup: /system.slice/middlewared.service
├─ 1986 middlewared
├─ 1997 /usr/bin/python3 -c "from multiprocessing.resource_tracker import main;main(11)"
├─ 2077 /usr/bin/python3 -c "from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=12, pipe_handle=46)" --multiprocessing-fork
├─ 9708 /usr/bin/python3 /usr/bin/websockify --web /usr/share/spice-html5/ --wrap-mode=ignore :5901 0.0.0.0:5900
├─427742 "middlewared (worker)"
├─428800 "middlewared (worker)"
├─429419 "middlewared (worker)"
├─431488 "middlewared (worker)"
└─432072 "middlewared (worker)"
Oct 15 20:15:16 truenas systemd[1]: Starting middlewared.service - TrueNAS Middleware...
Oct 15 20:15:31 truenas systemd[1]: Started middlewared.service - TrueNAS Middleware.
Oct 15 20:16:29 truenas middlewared[4178]: umount: /var/lib/systemd/coredump: not mounted.
Oct 15 20:17:17 truenas middlewared[1986]: libvirt: QEMU Driver error : Domain not found: no domain with matching name '29_PBS1'
Oct 15 20:51:12 truenas TNAUDIT_MIDDLEWARE[1986]: @cee:{"TNAUDIT": {"aid": "4cfd833f-aff4-4174-b0a9-68ddb38270f9", "vers": {"major": 0, "minor": 1}, "addr": "127.0.0.1", "user": "root", "sess": "a0550a74-b79c-4e0b-aa18-79f2491505a2", "time": "2>
Oct 16 09:33:42 truenas TNAUDIT_MIDDLEWARE[1986]: @cee:{"TNAUDIT": {"aid": "3bea2747-414d-47e4-ac24-06b5adf97b37", "vers": {"major": 0, "minor": 1}, "addr": "127.0.0.1", "user": "root", "sess": "d7a44200-7474-47f9-9f5d-8d534f0cc667", "time": "2>
Oct 16 09:42:15 truenas TNAUDIT_MIDDLEWARE[1986]: @cee:{"TNAUDIT": {"aid": "b8928382-fbab-4765-ba49-8e743daaa2a5", "vers": {"major": 0, "minor": 1}, "addr": "127.0.0.1", "user": "root", "sess": "816e0369-ec9a-458b-8609-8f8231a173cf", "time": "2>
Nginx and middlewared are both seemingly working. I found a post on the old forums that said to run a command to restart middleward via the TrueNAS cli, I believe it was
service restart middlewared
But that resulted in an error about the Namespace middlewared not being found. Perhaps I ran it incorrectly.