Tried an upgrade last evening. Must have missed something. Failed (after a long time) to load middleware.
Here’s the error in the middleware.log after failure:
UpdateService.get_trains():113 - Failed to comapre trains ‘TrueNAS-SCALE-ElectricEel’ and TrueNAS-SCALE-FangTooth’
Rolled back to old ElectricEel for now. Any thoughts on what I missed?
There was an issue initially on Tuesday, but this was corrected that evening. what are you seeing in the UI on the update page?
The corrected update file which was released is available below if you’d prefer to use the manual update.
https://download.sys.truenas.net/TrueNAS-SCALE-Fangtooth/25.04.0/TrueNAS-SCALE-25.04.0.update
I ran the update yesterday (Wednesday, 16 April 2025) around 13:00 NYC time (17:00 UTC)
Update page shows:
Upgrade TrueNAS-24.10.2.1 → TrueNAS-25.04.0
The actual message on the cosole was
middlewared: setting up plugins (audit)
It hung there for many minutes before posting about not being able to start middleware. I did try clearing out the /audit directory. Didn’t help.
I have the exact same issue upgrading from the latest electric eel to fangtooth. Tried upgrading with both the GUI and manually using the file linked above.
I have a Xeon-e3-1220 v3 cpu with 16GB RAM on an Asrock E3C224D2I, running two zraid1 pools each with 3 HDDs.
Tried a few different ways including deleting /audit and unsetting app pool before upgrading, but each time the same.
The only way to get past this was to do a fresh reinstall and then import configs. Must be something left from previous upgrades that was interfering.
Audit gets stuck on 49/115 and never comes up. I backed up into the previous version. Hopefully they come out with a fix.
I tried again last night. First trimmed audit directory. Same issues. I did notice on boot I got errors from grub (GRUB error 150, “unsupported encryption,” ).
In the recovery shell, I saw that 7 python processes were running. For the heck of it I tried systemd restart middlewared.service and got some errors about python3 processes still running.
I’ve seen no response addressing these issues.
The middleware process not starting is the kind of thing we want to diagnose and fix pretty quickly. Can you send us a ticket / debug file from the system? Might be helpful if you can show the output of /var/log/middlewared.log as well.
I mounted
boot-pool/ROOT/25.04.0-1/var on /mnt/var
boot-pool/ROOT/25.04.0-1/var/log on /mnt/var-log
middlewared.log is very sparse. Only two lines with last evenings date/time.
What log files would you need?
-rw-r----- 1 root root 1180434 Apr 22 12:38 middlewared.log
drwxr-xr-x 2 root root 2 Apr 22 17:30 journal
drwxr-x— 2 _chrony _chrony 2 Apr 22 17:46 chrony
drwxr-xr-x 2 root root 3 Apr 22 17:46 sysstat
drwxr-x— 2 root adm 4 Apr 22 18:09 audit
-rw-r–r-- 1 root root 6164 Apr 22 18:20 fallback-middlewared.log
-rw-rw-r-- 1 root utmp 4992 Apr 22 18:24 wtmp
Just to be clear, when you say you “mounted” you mean you are doing things in a recovery shell? If that’s the case then all the services not running would be expected, lots of boot up process is skipped there.
Is the GRUB error about unsupported encryption preventing you from booting into the normal boot-environment? Have you made any changes manually to the boot pool in any point during its history? I.E. running zpool upgrade or similar?
No, I restarted the server in EE and then mounted the zfs volumes created last evening by changing their mount points (I will change them back).
I did find this in truenas_verify.TrueNAS-25.04.0.log
2025-04-22 22:09:07.748425+00:00: 2 discrepancies found.
/var/lib/nfs/state: got uid 0, expected: 107
/var/lib/nfs/state: expected: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, got: 9d9f290527a6be626a8f5985b26e19b237b44872b03631811df4416fc1713178
Ok, if you are back up into EE then a bug ticket with attached debug will be the best thing to let us see whats going on in this case. It’ll have a lot more details that might reveal clues here.
Where do I find the debug file?
You can click the smiling face icon at the top right toolbar for “Send feedback” and submit a bug report there with attached debug file. Otherwise System → Advanced → Save Debug
Thanks. Submitted. Sorry for being a bit slow.
No worries. We’re taking a look at the ticket, please feel free to talk with the engineering staff there and we’ll see if we can get to the bottom of this.
I created a ticket from the application but I can’t reply with the requested information. It keeps saying I have to sign in, but there is no way to create a user or to sign in.
NVM I found it. Hard to see… Man I’m not functioning well today
I’m seeing the same issue.
I had upgraded from Core earlier today to 24.04 and I was expecting things to go bad with that but it went perfectly so was not expecting it to fail updating Scale → Scale 
I have booted back into 24.04 and submitted an issue. Hopefully there is something useful in there but let me know if I can provide any further info.
