Upgrade TrueNAS-24.04.2 -> TrueNAS-24.04.2.5 or Upgrade TrueNAS-24.04.2 -> TrueNAS-24.10.1 seems to complete but never boots to the new upgraded environment after reboot

TrueNAS General
1

Issue(s)

  • Upgrade TrueNAS-24.04.2 → TrueNAS-24.04.2.1
  • Upgrade TrueNAS-24.04.2 → TrueNAS-24.04.2.5
  • Upgrade TrueNAS-24.04.2 → TrueNAS-24.10.1

Both of these upgrade attempts appear to complete successfully but upon reboot, it never moves past TrueNAS-24.04.2

Observations

Over the CLI, here is the output from running cli followed by system update update over SSH. I enabled .stacks with the hopes of catching silent error messages but nothing…

...
.stacks - Enable/disable printing stack traces for errors
[your_truenas_hostname_here] system update> .stacks
Errors stack trace display: disabled
[your_truenas_hostname_here] system update> .stacks enable
Invalid value: 'enable'. Should be 'on' or 'off'
[your_truenas_hostname_here] system update> .stacks on
[your_truenas_hostname_here] system update> update
[0%] ...
[0%] Retrieving update manifest...
[0%] Verifying existing update...
[50%] Reading update file...
[50%] Verifying rootfs.squashfs...
[50%] Verifying truenas_install/dhs.py...
[50%] Verifying truenas_install/__main__.py...
[50%] Verifying truenas_install/fhs.py...
[50%] Verifying truenas_install/__init__.py...
[50%] Running pre-checks...
[50%] Starting installer...
[50%] Creating dataset...
[50%] Extracting...
[51%] Extracting...
[52%] Extracting...
[53%] Extracting...
[54%] Extracting...
[55%] Extracting...
[56%] Extracting...
[57%] Extracting...
[58%] Extracting...
[59%] Extracting...
[60%] Extracting...
[61%] Extracting...
[62%] Extracting...
[63%] Extracting...
[64%] Extracting...
[65%] Extracting...
[66%] Extracting...
[67%] Extracting...
,,[68%] Extracting...
[69%] Extracting...
[70%] Extracting...
[71%] Extracting...
[72%] Extracting...
[73%] Extracting...
[74%] Extracting...
[75%] Extracting...
[75%] Performing post-install tasks...
[75%] Copying previous configuration...
[76%] Migrating configuration database...
[77%] Running autotune...
[85%] Preparing NVDIMM configuration...
[85%] Preparing GRUB configuration...
[90%] Updating initramfs...
^[[C[95%] Updating GRUB...
[97%] Configuring FIPS...
[100%] Configuring FIPS...
true
[your_truenas_hostname_here] system update> ?

After the host reboots, the new boot environment (the upgraded one) is never triggered. The TrueNAS SCALE host remains on the original boot version as if it never upgraded per https://<your_truenas_server>/ui/system/boot. I’ve rebooted a second time after the upgrade “completes” (it doesn’t really). I’ve changed the old to “Active” and then selected the new one to “Active” boot environment and rebooted and it’s stuck like this…

At every reboot, this Critical error message appears in the UI notification pane,

Failed to sync TRUENAS catalog: [EFAULT] Failed to clone 'https://github.com/truenas/charts.git' repository at '/mnt/tank/ix-applications/catalogs/github_com_truenas_charts_git_master' destination: [EFAULT] Failed to clone 'https://github.com/truenas/charts.git' repository at '/mnt/tank/ix-applications/catalogs/github_com_truenas_charts_git_master' destination: Cloning into '/mnt/tank/ix-...
2025-01-03 13:06:37 (America/New_York)
Dismiss

Once I manually go to https://<your_truenas_server>/ui/apps/available > “Refresh Catalog”, the error message goes away since catalog.sync_all succeeds…
Note: I do not use TrueCharts at all. This one is for the ix native scale apps.

Troubleshooting

  • Did you try via the WebUI over at https://<your_truenas_server>/ui/system/update? Yes
  • Did you try via the CLI/shell using cli followed by system update update? Yes
  • Did you try via the WebUI using the Install Manual Update option over at https://<your_truenas_server>/ui/system/update/manualupdate? Yes. I downloaded both .update files from the official source over at https://download.truenas.com/TrueNAS-SCALE-Dragonfish/24.04.2.5/?wrap=1
  • Are there any unresolved hardware failure? No. Everything is reporting healthy

Ask to the community

  • Any thoughts on how to debug the upgrade process to find out what failed?
  • Any non conventional methods to figure out what’s going on? Should I force delete some artifact somewhere and try again?

Thank you for reading my post! I look forward to any help :slight_smile: