Pool swap - Replicated pool and .ix-apps mount point problem

TrueNAS General
,
1

Hoping someone can help me out a bit here. I’ve been trying to move my main pool to a new set of disks. Everything looks good until I import the new pool (clone of the old pool) using the UI. At that point it looks like a new /mnt/.ix-apps folder is created by the system instead of mounting the ix-apps dataset at that path. The new pool should be the same as the old one. Same datasets, same name. I’m wondering if something in TrueNAS is using the old pool’s ID instead of it’s name to find that dataset.

I’ll break down exactly what I’ve done:

On Electric Eel 24.10.0

Setup

  1. Attach new disks to system and make a new pool (newtank) that will replace the old one.
  2. Stopped all apps and share services.
  3. Create a manual recursive snapshot of tank.
  4. Run a recursive replication task sending tank to newtank. “Include Dataset Properties” enabled, and the manual snapshot selected. This ran successfully.

Now I have tank and newtank, and in the Datasets UI they look identical (except for size, and the Roles icons). Time to swap them.

  1. Export both pools from the UI. Uncheck delete saved configs. The warning does says that Docker: tank will be interrupted when I export tank, maybe that was a problem.
  2. Rename tank → oldtank from cli:
    a. zpool import tank oldtank
    b. zpool export oldtank
  3. Rename newtank → tank from cli:
    a. zpool import newtank tank
    b. zpool export tank
  4. Import tank from the UI. “Should” be done.

The Dataset and Apps UI sections are erroring now:

  • Failed to load datasets [Errno 2] No such file or directory: '/mnt/.ix-apps/app_configs'
  • FileNotFoundError [Errno 2] No such file or directory: '/mnt/.ix-apps/app_configs'

In the shell I see that a new /mnt/.ix-apps folder has been created. It’s obviously missing data from the ix-apps dataset on the pool:

$ ls -lAh /mnt/.ix-apps
total 5.0K
drwxr-xr-x  4 root root  4 Nov  3 11:54 .
drwxr-xr-x  6 root root  7 Nov  3 11:54 ..
drwx--x--- 12 root root 13 Nov  3 11:54 docker
drwxr-xr-x  8 root root 15 Nov  3 11:54 truenas_catalog

$ ls -lAh /mnt/tank/ix-apps
total 27K
drwxr-xr-x  5 root root    5 Oct 31 23:04 app_configs
drwxr-xr-x  3 root root    3 Nov  3 10:30 app_mounts
drwx--x--- 12 root root   13 Nov  1 12:41 docker
-rw-r--r--  1 root root 7.0K Oct 31 23:38 metadata.yaml
drwxr-xr-x  8 root root   15 Nov  1 11:25 truenas_catalog
-rw-r--r--  1 root root 7.9K Oct 31 23:38 user_config.yaml

If I try to work around it and symlink .ix-apps to the folder on tank, the UI pages load but then apps won’t start. I know I’m going to break things if I keep messing around though.

I can reverse my import/export/rename steps and reboot the system and it’s back to normal with the old drives. So now I have 2 copies of the pool, but only 1 will work with truenas when imported. Which makes me think there is a reference to the pool ID somewhere that needs to be updated (if possible). Am I on the right track?

1 Like
2

I managed to resolve this. The issue wasn’t some secret ID, it was just that the dataset mountpoint property was not copied over when the pool was replicated. Updating that got everything back into working order with the new drives.

With original (working) pool attached:

  • Apps UI > Unset pool
  • Export old pool
  • Import new pool from CLI
  • Set mountpoint for ix-apps dataset on new pool to /mnt/.ix-apps
    • check current mountpoint, it will probably be /mnt/tank/ix-apps:
    • zfs get mountpoint tank/ix-apps
    • update it to the correct location:
    • zfs set mountpoint=/mnt/.ix-apps tank/ix-apps
  • Export new pool from CLI
  • Import new pool from UI
  • Apps UI > Choose Pool. It will say Initializing pool but if the mountpoint has been updated correctly all your apps will re-appear when it’s done.

Hope this helps someone else.

1 Like
4

Yep.

For future reference, I cover the mountpoint in this tutorial:

1 Like