I upgraded from TrueNAS scale 24.04 to 24.04.1 today and my main storage pool is now basically destroyed. The devices show up in the ‘unassigned disks’ section and when I click ‘add to pool’, the devices properly show that they are part of the pool.
After the upgrade everything was initially good. I decided to upgrade the ZFS pools through the GUI as it had been a while and I remembered reading about a significant ZFS change to allow better pool expansion. All 3 pools reported that the upgrades completed without reporting any issues.
A few hours later I rebooted and then I was stuck at the ‘Welcome to Grub’ screen. I fought this for about 2 hours with no success.
I tried suggestions to enable CSM Legacy support, this worked but when I got back into TrueNAS, the above pool issue happened.
I’ve moved back to having CSM completely disabled (as it was the whole time I’ve been running TrueNAS) but the disks still show up as unassigned and still show as part of the pool in the add disks screen.
If I try to import the pool, nothing shows up to import. If I try to add the “unassigned” disks to the existing pool, the pool does not show up as an option to select though the pool does show on the storage tab. At this point, I’m unsure how to proceed.
After changing to CSM Legacy support, my LSI 9207-8i started showing a BIOS screen at boot and had a message about a configuration change. Did something with this ruin my pool? I know a common question with these cards is if they are flashed to IT mode, and yes both of mine are.
I also see that the broken pool is still showing that it can be upgraded for newer ZFS flags, though that should have already been done after the upgrade when I upgraded all the pools. I’m wondering if somehow the disks upgraded but TrueNAS ?forgot? and now it’s causing a problem.
I have 3 pools total, 2 are reporting fine and don’t report that they have a ZFS flag upgrade available. The problem pool is showing 0 of the 21 devices assigned and all 21 are showing “(Media)” after their names in the ‘add to pool’ screen
I’ve tried uploading a backup that I made before the 24.04.1 upgrade and that didn’t change anything.
Thanks for any suggestions you might have and I’m happy to provide any more info that I can.
Using “add to pool” screen is not the correct way to deal with a pool that is missing. In fact, it can be the worse way to trouble shoot if someone accidentally creates a new pool using those disks.
One thing that is generally not well dealt with in TrueNAS, (yet, it’s getting better), is that upgrading the boot pool is generally not a good idea. Especially with SCALE. This can lead to an unbootable system.
Further, upgrading pool(s) without clear purpose can prevent certain recovery processes. Like booting older TrueNAS versions because that older version does not support the new, active pool feature(s).
My recommendation on pool feature upgrades is:
Don’t upgrade unless you are looking for a specific new pool feature.
If possible, verify that all your older TrueNAS boot environments support the desired new pool feature. And if you do upgrade the pool, and their is a boot environment that does not support that feature(s), then you might as well remove that boot environment. (It is no longer usable with your pool(s)).
Upgrade to only those new pool features you desire via the command line. You can leave the others disabled.
Alright, so it looks like some drives aren’t being seeing by TrueNAS at all, so it i something with the hardware. The zpool status didn’t show anything about the problem pool, but zpool import did show the problem pool and listed a few drives as unavailable, so this is what is causing the problem. Thank you so much for getting me a lead to start hunting.
Also, thankfully I didn’t upgrade the boot pool, only my data pools. I also never intended to add the disks to a pool through the ‘add to pool’ GUI, I just wanted to show that the drives still showed as part of the pool. I appreciate the info and I guess I got a little too excited seeing the Upgrade button on the Scale GUI, where I was used to doing it on the CLI before. Paired with the article I read about adding single drives to ZFS with the new flag and I certainly jumped the gun.
Time to hunt some hardware issues now. Thanks again for the help, panic made me forget about actual CLI commands.
Here is another command you can use, (SCALE / Linux specific), for disk trouble shooting; lsblk
It probably won’t list the ZFS pool relationship, but, if you are checking for drive existence, it can help.
TrueNAS SCALE, (& Core), are not the easiest NAS software to trouble shoot. They were originally written for the Enterprise Data Center market, as was ZFS. TrueNAS can and does work well for many home and small office users, though during trouble shooting knowledge of ZFS and the underlying OS, (Linux for SCALE and FreeBSD for Core), is very useful.
