Problems upgrading TrueNAS SCALE 25.04.2.6 -> 25.10.1

TrueNAS General
1

Having converted this from CORE using the progressive update method detailed in the documentation (Core 13.3 → Scale 24.04 → Scale 25.04), I’m having difficulty continuing to upgrade to 25.10:

[EFAULT] Error: Command ['chroot', '/tmp/tmpbosykf88', 'mkdosfs', '-F', '32', '-s', '1', '-n', 'EFI', '/dev/sdp1'] failed with exit code 1: mkdosfs: unable to open /dev/sdp1: Device or resource busy

It looks like /dev/sdp1 is currently mounted as /mnt/boot, which is probably the cause of the issue. Any idea how I should proceed?

2

Is everything you need in the 25.04 version? Make sure you have a current system configuration download. You can try deleting the 25.10 boot environment, if there is one, and trying to upgrade again. Another option could be a clean install of 25.10 and restoring the system configuration.

Are you attempting to upgrade 25.04.2.6 directly to 25.10.1 or did you do 25.10 and then upgrade to latest 25.10 version? Do you have enough freespace on your boot pool?

Kind of guessing since you didn’t post details for your system

3

I did grab a configuration download before attempting the upgrade. I am attempting to upgrade using the built-in upgrade in the WebUI, switching train from the current “TrueNAS SCALE Fangtooth 25.04 [release]” train to “TrueNAS SCALE Goldeneye 25.10”, which is saying it will Upgrade TrueNAS-25.04.2.6 -> TrueNAS-25.10.1

As to system specs:

  • IBM x3650 M3 chassis (16xSFF drive bays)
  • Dual Xeon X5677
  • 72GiB of DDR3 ECC RAM
  • Boot Volume: 120GiB SATA SSD
  • Container/VM Storage: 6(RAIDZ1)x1TB SAS Drive (Seagate, IBM branded)
  • Storage: 2x5(RAIDZ1)x5TB (SMR) SATA Drive (Seagate)

According to zpool list:

NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Alcatraz      5.44T  3.05T  2.39T        -         -    47%    56%  1.00x    ONLINE  /mnt
DataDrives    45.3T  43.8T  1.57T        -         -    53%    96%  1.00x    ONLINE  /mnt
freenas-boot   118G  9.54G   108G        -         -      -     8%  1.00x    ONLINE  -

So there should be plenty of space.

4

DataDrives is showing at 96% full. Recommendation is below 80% usage. 50% if block storage. I don’t think that is the issue here but that is considered the ZFS danger zone leading to crashes and possible pool loss.
Did it create a boot environment for 25.10 or it’s stopping before that. Options could be attempting an upgrade by downloading the Manual Update and trying that or booting an install image of 25.10 and doing an upgrade. I think it should give you that option in the menus. Have the pools been scrubbed recently?

5

Yes, both pools are set to scrub weekly. I am aware that 96% is bad, and I am intending to move the data off to another machine/array, but currently 5TB is the largest capacity HDD you can get in a 2.5” size (Yes, I know about 8TB QVO SATA SSDs and/or 15/31TB Enterprise SAS SSDs, no I don’t have the budget for that).

There does not appear to be a dataset created for 25.10 in the boot drive (it looks like that’s what it was trying to do?), it hasn’t rebooted to start the install proper this is all within the webui at this point.

6

I would try using the manual download option first to see if it was just the automatic method that was getting stuck. If that doesn’t work, it would be a fresh install with 25.10 and reloading the system configuration.
I would go with that release and then do the upgrade to 25.10.1 if that is successful. You will lose the previous boot environments for 25.04 and earlier with fresh install, though.

As long as you were aware of the full pool, you should be okay. I’ve had a few users stuff their pool and then we couldn’t get back in or we had to work at deleting files. System gets unresponsive with much more data.

I should add that upgrading to 25.10 puts you in the Early Adopter category at this time. Once they release the next version and move 25.10 to General status on the software status page, then you could choose that. If you upgrade now, you might be stuck in Early Adopter. I don’t know if you can down grade to more conservative status. This is a recent change on the ‘upgrade trains’ shown.

image
image1321×839 17.5 KB

7

Okay, attempting a reboot gets me to GRUB with a “25.10.1” revision, but trying to boot it gets the message “25.10.1 not found”

Loading Linux 6.12.33-production+truenas ...
error: file `25.10.1' not found.
Loading initial ramdisk ...
error: you need to load the kernel first.

Press any key to continue...

Pressing the key takes me back to the GRUB menu, where fortunately, there is also a menu option for 25.04, which I can use to boot into the earlier version.

It looks like I may be trying a manual update, or even booting from 25.10 install media and install/upgrading from there.

8

…And attempting a manual update gets me all the way up to:

[EFAULT] Error: Command ['chroot', '/tmp/tmpb_8ntmwm', 'efibootmgr', '-c', '-d', '/dev/sdp', '-p', '1', '-L', 'TrueNAS-0', '-l', '/EFI/debian/grubx64.efi'] failed with exit code 5: Could not prepare Boot variable: No space left on device
9

Do you have a lot of previous boot environments? I am confused because you show plenty of space with zpool list on the boot-pool. If you aren’t in a hurry, waiting for other users to post may help. I’m not sure where it could be having ‘space’ trouble.

We could try checking the partitions for the boot device but not sure what to really look at or where the GRUB is supposed to be.

10

Okay, so going into the (very early) EFI system for this server and deleting some defined boot options, along with booting directly from the 25.10 ISO and installing fresh seems to have worked. I’m now waiting for the system to finish restarting after importing my previous (25.04) configuration, which is currently sitting at ix-zfs.service/start and ix-netif.service/start, which have been running for the last 5 minutes.

11

Hello, I am in the same situation.

The upgrade from TrueNAS-25.04.2.6 → TrueNAS-25.10.1 fails with the error: Could not prepare Boot variable: No space left on device

Down here the details:

[EFAULT] Error: Command [‘chroot’, ‘/tmp/tmpqbhk60c7’, ‘efibootmgr’, ‘-c’, ‘-d’, ‘/dev/sde’, ‘-p’, ‘2’, ‘-L’, ‘TrueNAS-0’, ‘-l’, ‘/EFI/debian/grubx64.efi’] failed with exit code 5: efibootmgr: ** Warning ** : Boot0011 has same label TrueNAS-0
Could not prepare Boot variable: No space left on device
More info…
Error: Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 515, in run
await self.future
File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 560, in __run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 174, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/update.py”, line 287, in update
await self.middleware.call(‘update.install’, job, os.path.join(location, ‘update.sqsh’), options)
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1005, in call
return await self.call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 731, in call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 624, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/update/install_linux.py", line 50, in install
self.middleware.call_sync(“update.install_scale”, mounted, progress_callback, options)
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1041, in call_sync
return methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/update/install.py", line 61, in install_scale
self.execute_truenas_install(mounted, command, progress_callback)
File "/usr/lib/python3/dist-packages/middlewared/plugins/update/install.py", line 92, in _execute_truenas_install
raise CallError(result or f"Abnormal installer process termination with code {p.returncode}")
middlewared.service_exception.CallError: [EFAULT] Error: Command [‘chroot’, ‘/tmp/tmpqbhk60c7’, ‘efibootmgr’, ‘-c’, ‘-d’, ‘/dev/sde’, ‘-p’, ‘2’, ‘-L’, ‘TrueNAS-0’, ‘-l’, ‘/EFI/debian/grubx64.efi’] failed with exit code 5: efibootmgr: ** Warning ** : Boot0011 has same label TrueNAS-0
Could not prepare Boot variable: No space left on device

The boot pool is free at 98% and scrubbed.

Reinstalling from scratch seems an overkill solution to me, do you have any suggestion to solve this issue?

Thanks,

Michele

12

What are you using for a boot device?

@HoneyBadger Any ideas? Should this be a Bug Report issue?

13

Probably. This seems like a potential UEFI bug though, the “no free space” is a red herring looking up a couple lines:

Unless it’s specifically talking about “space left in the EFI volume”

14

@mikdima @Anaerin Would you two submit bug reports and dump files from your working versions of TrueNAS? It’s the smile icon on the upper right of the TrueNAS GUI, marked Feedback, Report a Bug. You can add a link to this thread when you submit all your info.
Please post back with a link to the created bug report.

15

@SmallBarky : I am using a 2.5” SATA SSD drive.

After this failed update I restarted the NAS but it didn’t boot.
When I had the chance, I attached a monitor to the NAS and I found out that TrueNas was stuck in the Grub menu, offering to boot with the version 25.10.1 (which did not complete).

To restore it, I had to attach a keyboard and choose the version 25.04.2.26. After the first boot (and a check under System→Boot in the UI) the boot menu was restored.

16

I think doing Report a Bug from your working version with the debug dump attached would be the best option and linking the created ticket in this thread. That at least gets all your info to TrueNAS and the Jira system. Maybe check for system bios updates for the motherboard?

Added link to ticket Jira

17

@SmallBarky I can’t add links to the this thread, but the ticket is NAS-139541

18

@SmallBarky : well, I created the ticket from there. Also: I can’t upload the debug dump because I have no permissions on the link I received (see the ticket on Jira). :S

And, no, there are no BIOS updates for my motherboard.

19

Did you do the bug report from the TrueNAS GUI and choose to attach the debug dump at that time or did you use the link at the top right of the forum, Report a Bug? I don’t have access to see and would have to ask someone else to check Jira.

20

@HoneyBadger Could you check on the linked Jira above. Seems it won’t let user attach debug or is sending a funny link. Looks like ticket created from GUI, if I read the Jira comments.