Middleware reconstructing the mount tree at boot?

TrueNAS General
1

Im looking for guidance before I attempt to edit the TrueNAS database or reinstalling.

Environment:

  • TrueNAS SCALE 25.04.2.6
  • Pools:
  • boot-pool
  • big_tank
  • little_tank
  • Originally migrated from another TrueNAS system several months ago.
  • Apps currently limited to Dockge and Scrutiny.

Problem:
All datasets under my pools are being mounted as /mnt/mnt/big_tank/... and /mnt/mnt/little_tank/... instead of /mnt/big_tank/… and /mnt/little_tank/…
This causes numerous issues including app-related mount fal;ures and an inability to perform certain ZFS operations.

The strange part (yet, important part) is when booting from a Live environment and importing the pool (ie zpool import -f big_tank). All dataset mountpoints appear correct (/mnt/big_tank, /mnt/big_tank/config, etc) There are no /mnt/mnt mountpoints stored in ZFS.

Running zfs get -r mountpoint big_tank shows only normal /mnt/big_tank/… paths.

What Ive found is within TrueNAS itself:

midclt call pool.query | jq ‘. | {name,path}’

returns

{
“name”: “big_tank”,
“path”: “/mnt/big_tank”
}

which is correct. However:

midclt call pool.dataset.query

contains numerous reference to /mnt/mnt/big_tank and /mnt/mnt/little_tank, and all those paths appear throughout the middleware output.

A few other things I think are important to know:

  1. zpool get altroot shows

big_tank altroot /mnt
little_tank altroot /mnt

which appears normal for SCALE

  1. findmnt, mount, and /proc/self/mountinfo all show datasets actively mounted under /mnt/mnt/... after TrueNAS boots.
  2. Attempting to change mountpoints from SCALE results in errors such as:

cannot unmount ‘/mnt/mnt/…’: no such pool or dataset

  1. The same datasets import and mount normally while outside of TrueNAS

Things Ive tried.

  • Rebooting
  • Disabling Incus
  • Removing old docker/incus remnants
  • Manual unmount attempts
  • mounting and unmounting via ZFS
  • Verifying all datasets and mountpoints from a Live boot environment
  • Verified no /mnt/mnt mountpoints are actually stored in ZFS metadata

I have to beleive that the pools and ZFS metadata is healthy, but TrueNAS middleware has cached or stored incorrect dataset paths somewhere in its configuration database and its reconstructing the mount tree incorrectly at boot.

My Questions:

  1. Has anyone seen SCALE create a /mnt/mnt mount hierarchy like this?
  2. Is there a known middleware or database issue that can cause dataset paths to be duplicated?
  3. Is there a supported way to force middleware to rebuild dataset metadata from ZFS?
  4. Has anyone successfully repaired this without a full reinstall and configuration rebuild?

Honestly any guidance would be appreciated, Ive been working on this for days. I have backups and can provide any additional command outputs if needed.

Thank you,
Ryan

2

We’ve seen cases where zfs pools created on different os’s like proxmox or unraid get mounted under /mnt/mnt when imported in truenas.

If my memory is correct this post has solutions to fix it