ZFS Mount and TrueNAS Configuration and Apps

TrueNAS General
1

Hello All, I completely borked my TrueNAS instance and magically fixed it but wondering what the next step for me is.

Long story short, I had an old SATA m.2 with TrueNAS on it, I decided to upgrade and move it to an nvme that was sitting around. I put both in separate m.2 slots on my motherboard, logged into my instance on the sata and changed my boot-pool to a mirrored pool. I went ahead and removed the sata and it booted up properly with no issues. Then I got another nvme of the same size and attempted to resilver the mirrored pool so it was no longer degraded. It crashed half way through the resilver and dropped into the grub menu. I tried all sorts of things including scrub using the old sata m.2 with no luck. In fact the scrub messed up the grub config on the original disk as well. Fast forward a few days and trying to reinstall grub via live disk and finally giving up, decided to reinstall TrueNAS, upload my config, and copy the App data onto the new boot-pool so I don’t have to reinstall all the new apps since the old boot-pool was still intact just no grub config to load the OS. I ended up going into the linux shell of the new install, mounted the the old boot-pool as old at /mnt/old and as I tried to mount all dataset I ran zfs mount -a quickly after I realized that it mounted everything at the root level. Thinking that I corrupted my fresh install, I decided to restart the machine before putting any more effort into it (in case I have to start over) and vola everything came back on, all apps were running without issue, my cloudflared tunnels are working fine.

Here is my assumption: Somehow when I mounted the old boot-pool onto the root level of the new boot-pool it ended up overwriting all the directories needed to “revive” the old instance.

Here are my questions:

  1. When I checked the storage usage of the boot-pool and old , old has significantly higher usage. I don’t know if this is due to all the many kernels since I started from 24 and upgraded to 25 over time? or all the data is still ONLY on old and are just linked to the new boot-pool? I guess one easy way to find out is to remove the old m.2 and see if everything comes back up but I’m worried it’ll ruin the setup again.

  2. Is there a way to know which pool app images are being loaded from?

  3. If we are able to determine this is not a permanent “fix” is there a way to make this permanent?

Side note for anyone who has a corrupted grub and does not have their config backed up, this may be helpful:

  1. Remove your original boot disk from the machine.
  2. Attach a new disk
  3. Make a fresh install of TrueNAS (preferably the same version as your original installation)
  4. Attach the original boot disk along with the new boot disk
  5. It’ll boot into initramfs but it’ll be stuck since there are now 2 boot-pools available.
  6. Do zpool import to get a list of boot-pools available
  7. Do zpool import <ID> is the id of the old boot-pool from step 6. You may need to add -f
  8. type exit

Above steps should use the grub from the new install to load initramfs and from there initramfs can load up the OS from the old boot-pool with no issues.

2

Update: Decided to risk it and the “fix” has persisted. So I guess problem solved.