Web GUI not loading - middleware not found

TrueNAS General
1

Hi all, been trying to fix this without doing a restart. At the moment the apps are running in k3s as expected but the UI stopped working last night and the middleware daemon won’t restart.

my-name@truenas:~$ sudo service middlewared status
× middlewared.service
     Loaded: not-found (Reason: Unit middlewared.service not found.)
     Active: failed (Result: signal) since Wed 2024-04-03 19:00:31 IST; 21h ago
   Duration: 1d 8h 33min 37.483s
   Main PID: 1821 (code=killed, signal=KILL)
        CPU: 2h 31min 6.857s

Apr 03 04:02:49 truenas cron[1821]: 231128_1.1.2
Apr 03 04:02:49 truenas cron[1821]: 231128_2.0.4
Apr 03 04:02:49 truenas cron[1821]:
Apr 03 04:02:49 truenas cron[1821]:
Apr 03 04:02:49 truenas sudo[1184792]: pam_unix(sudo:session): session closed for user root
Apr 03 19:00:19 truenas systemd[1]: Stopping middlewared.service - TrueNAS Middleware...
Apr 03 19:00:31 truenas systemd[1]: middlewared.service: Main process exited, code=killed, status=9/KILL
Apr 03 19:00:31 truenas systemd[1]: middlewared.service: Failed with result 'signal'.
Apr 03 19:00:31 truenas systemd[1]: Stopped middlewared.service - TrueNAS Middleware.
Apr 03 19:00:31 truenas systemd[1]: middlewared.service: Consumed 2h 31min 6.857s CPU time.

my-name@truenas:~$ sudo service middlewared start
Failed to start middlewared.service: Unit middlewared.service not found.

Running this my-name@truenas:~$ sudo journalctl -u middlewared -n 100 -f shows nothing more useful, just an update that ran at 4am on a heavyscript cron.

The only change made to the system outside of the apps would be installing nixstats which I use for monitoring (and ideally logging but I haven’t gotten that working on TNS yet.)

NMAP shows that ports 80 and 443 are open (used by nginxproxy) and 8282 open (http UI) and 8443 closed (https UI). On both my browser and in-private window I get the same “Connecting to TrueNAS … Make sure the TrueNAS system is powered on and connected to the network.” message.

Thanks in advance! Let me know if any other info needed

2

Update: I attempted a restart, got an error:

[FAILED] Failed to start systemd-journal-flush.service - Flush Journal to Persistent Storage
# lines skipped
You are in rescue mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode.

Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.

Press Enter to continue.

On pressing enter

Reloading system manager configuration
Starting default.target
Failed to start default.target: Operation refused, unit may not be isolated.

At this point I’m tempted to re-install 23.10 and start again. I have a backup of my v22 config since I just upgraded, though through an install and not the upgrade path as that also did not succeed.