Unable to update TrueNAS scale [SOLVED]

I am running TrueNAS SCALE 24.04.0
No VM’s or apps. have NFS shares.

Last night i received several alert messages saying stuff like:

  • Failed to check for alert ScrubPaused: Failed connection handshake
  • Failed to check for alert Quota: Failed connection handshake
  • Failed to check for alert ScrubPaused: Failed connection handshake
  • Failed to check for alert ZpoolCapacity: Failed connection handshake

But all alerts were cleared as well. This was a bit concerning, so i wanted to check it out but my TrueNAS was really slow to respond. Be it GUI, shell or via IPMI (as in, takes a while to respond to interactions and loading).

I tried rebooting the system but it took like 10-20 minutes to shutdown (which from my experience, is very long). Booting up took about 5-10 min (within expectancy). But still, the GUI was rather slow in loading pages.

I didn’t see any alerts, i did however see lots of journal message while shutting down saying stuff like

xxx.service: killing process yyyy with signal SIGKILL
xxx.service: Processes still around after SIGKILL. Ignoring
Failed with result ‘Timeout’
Unit process yyyy remains running after unit stopped

(Not sure where these are logged, so apologies i can’t copy past them)

I tried updating my system to, but this failed after 2 hours, it was very slow to extract the update.
This is the error i got at the end:

Error: [EFAULT] Traceback (most recent call last):
File “< frozen runpy >”, line 198, in _run_module_as_main
File “”, line 88, in run_code
File “/tmp/tmpbsewpk4c/truenas_install/main.py”, line 794, in
File “/tmp/tmpbsewpk4c/truenas_install/main.py”, line 589, in main
enable_system_user_services(root, old_root)
File “/tmp/tmpbsewpk4c/truenas_install/main.py”, line 126, in enable_system_user_services
configure_serial_port(root, database_path(old_root))
File “/tmp/tmpbsewpk4c/truenas_install/main.py”, line 114, in configure_serial_port
advanced = query_config_table(“system_advanced”, db_path, prefix="adv
File “/tmp/tmpbsewpk4c/truenas_install/main.py”, line 106, in query_config_table
return query_row(f"SELECT * FROM {table}", database_path, prefix)
File “/tmp/tmpbsewpk4c/truenas_install/main.py”, line 94, in query_row
sqlite3.OperationalError: database is locked

I did notice something while updating. i checked my resources, CPU and memory were at 50% at best, but when i checked my disk capacity in shell with df -h i noticed that the /dev/loop0/ was at 100%, which was mounted to /tmp/tmpbsewpk4c. not sure if this matters, as it is a temporary location ( i am not that familiar with loop0).

  1. I read about the SWAP issue, could this be related?
  2. Any way to troubleshoot this slow responsiveness?
  3. Is there any other way i can update the system?
    • i saw there is a manual option to update, but would that help when it struggles to extract it?

Appreciate any help you can give!

Yeah could be related. If you’ve got sufficient memory, run swapoff -a and then try upgrading again.
Worst case you might need to reboot the box and run that at the start if things are too slow for swap to properly flush back into memory.

I managed to update the system after running swapoff -a. It still took about an hour to update the system, but at least it wasn’t resulting in an error at the end.
currently running version

System seems to be more responsive now. After the update, the reboot took several minutes (at least allot quicker compared to earlier).
Currently not seeing anything weird going on, but will keep an eye on it the next few days.

Thanks for the help @essinghigh

1 Like