Cleanup after re-installing apps on a different pool

Hi everyone,

Introduction

About a year ago I’ve done a fresh install of TrueNAS Scale (I think 24.10, but perhaps also 24.04 first) on my server (it was running TrueNAS Core before). Some months ago I’ve upgraded to 25.04 (currently 25.04.2.1).

For a long time, my TrueNAS server only had a RAIDZ2 of 8 HDDs with a lot of data, 1 Linux VM and 2 apps (Plex and Immich).
But some weeks ago, I’ve finally installed a small SSD pool (mirror) to run my apps and VMs on it.

The Linux VM was easily migrated to the SSD pool. But migrating the apps seems less obvious…

After a lot of messing around, I found below post of Stux that explains well how a migration can be done.

However, since I’ve learned the hard way best not to mess around too much outside of the TrueNAS GUI, and since I only have 2 apps, I decided to reinstall the apps from scratch on the new SSD pool, but keeping the data.

So this is what I did in the end

  1. I’ve created new datasets for my applications on the SSD pool.
  2. I’ve copied all application data from the HDD pool to the new datasets on the SSD pool.
  3. I’ve done “unset pool” in the Apps configuration in the TrueNAS GUI.
  4. I’ve done “choose pool” in the Apps configuration in the TrueNAS GUI and selected the SSD pool.
  5. I’ve installed Immich and Plex as brand new installations, setting the same configuration as my old applications, pointing to the new host paths, aka my new datasets on the SSD pool with a copy of my application data on the HDD pool.

This seems to have worked well, as my apps are properly running and haven’t lost any data/config.

The actual problem

So far so good, but then I wanted to look into cleaning up everything application related from my HDD pool. I wasn’t surprised to see many files / folders to be still there.
However, I wasn’t expecting so many datasets still being mounted. I also don’t really know how to validate for 100% sure that they are no longer needed or used, before I try to remove them.

Below is a list of application related datasets and mount

root@truenas-master:/home/admin# zfs list | grep ix
master-pool/ix-applications                                                    13.5G  43.9T  1.36M  /mnt/master-pool/ix-applications
master-pool/ix-applications/catalogs                                            118M  43.9T   118M  /mnt/master-pool/ix-applications/catalogs
master-pool/ix-applications/default_volumes                                     205K  43.9T   205K  /mnt/master-pool/ix-applications/default_volumes
master-pool/ix-applications/k3s                                                13.4G  43.9T  13.2G  /mnt/master-pool/ix-applications/k3s
master-pool/ix-applications/k3s/kubelet                                         785K  43.9T   785K  legacy
master-pool/ix-applications/releases                                           6.18M  43.9T   222K  /mnt/master-pool/ix-applications/releases
master-pool/ix-applications/releases/immich                                    2.25M  43.9T   222K  /mnt/master-pool/ix-applications/releases/immich
master-pool/ix-applications/releases/immich/charts                             1.45M  43.9T  1.45M  /mnt/master-pool/ix-applications/releases/immich/charts
master-pool/ix-applications/releases/immich/volumes                             597K  43.9T   205K  /mnt/master-pool/ix-applications/releases/immich/volumes
master-pool/ix-applications/releases/immich/volumes/ix_volumes                  392K  43.9T   256K  /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes
master-pool/ix-applications/releases/immich/volumes/ix_volumes/library            0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/library
master-pool/ix-applications/releases/immich/volumes/ix_volumes/pgBackup           0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/pgBackup
master-pool/ix-applications/releases/immich/volumes/ix_volumes/pgData             0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/pgData
master-pool/ix-applications/releases/immich/volumes/ix_volumes/profile            0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/profile
master-pool/ix-applications/releases/immich/volumes/ix_volumes/thumbs             0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/thumbs
master-pool/ix-applications/releases/immich/volumes/ix_volumes/uploads            0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/uploads
master-pool/ix-applications/releases/immich/volumes/ix_volumes/video              0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/video
master-pool/ix-applications/releases/plex-master                               3.57M  43.9T   222K  /mnt/master-pool/ix-applications/releases/plex-master
master-pool/ix-applications/releases/plex-master/charts                        2.90M  43.9T  2.77M  /mnt/master-pool/ix-applications/releases/plex-master/charts
master-pool/ix-applications/releases/plex-master/volumes                        461K  43.9T   205K  /mnt/master-pool/ix-applications/releases/plex-master/volumes
master-pool/ix-applications/releases/plex-master/volumes/ix_volumes             256K  43.9T   256K  /mnt/master-pool/ix-applications/releases/plex-master/volumes/ix_volumes
master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/config        0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/config
master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/data          0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/data
master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/logs          0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/logs
master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/transcode     0B  43.9T   205K  /mnt/master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/transcode
master-pool/ix-apps                                                            7.39G  43.9T   273K  /mnt/.ix-apps
master-pool/ix-apps/app_configs                                                14.8M  43.9T  14.8M  /mnt/.ix-apps/app_configs
master-pool/ix-apps/app_mounts                                                 72.8M  43.9T   222K  /mnt/.ix-apps/app_mounts
master-pool/ix-apps/app_mounts/immich                                          71.5M  43.9T   256K  /mnt/.ix-apps/app_mounts/immich
master-pool/ix-apps/app_mounts/immich/backups                                  69.9M  43.9T  69.7M  /mnt/.ix-apps/app_mounts/immich/backups
master-pool/ix-apps/app_mounts/immich/library                                   205K  43.9T   205K  /mnt/.ix-apps/app_mounts/immich/library
master-pool/ix-apps/app_mounts/immich/pgBackup                                  205K  43.9T   205K  /mnt/.ix-apps/app_mounts/immich/pgBackup
master-pool/ix-apps/app_mounts/immich/pgData                                    205K  43.9T   205K  /mnt/.ix-apps/app_mounts/immich/pgData
master-pool/ix-apps/app_mounts/immich/profile                                   205K  43.9T   205K  /mnt/.ix-apps/app_mounts/immich/profile
master-pool/ix-apps/app_mounts/immich/thumbs                                    205K  43.9T   205K  /mnt/.ix-apps/app_mounts/immich/thumbs
master-pool/ix-apps/app_mounts/immich/uploads                                   205K  43.9T   205K  /mnt/.ix-apps/app_mounts/immich/uploads
master-pool/ix-apps/app_mounts/immich/video                                     205K  43.9T   205K  /mnt/.ix-apps/app_mounts/immich/video
master-pool/ix-apps/app_mounts/plex-master                                     1.05M  43.9T   256K  /mnt/.ix-apps/app_mounts/plex-master
master-pool/ix-apps/app_mounts/plex-master/config                               205K  43.9T   205K  /mnt/.ix-apps/app_mounts/plex-master/config
master-pool/ix-apps/app_mounts/plex-master/data                                 205K  43.9T   205K  /mnt/.ix-apps/app_mounts/plex-master/data
master-pool/ix-apps/app_mounts/plex-master/logs                                 205K  43.9T   205K  /mnt/.ix-apps/app_mounts/plex-master/logs
master-pool/ix-apps/app_mounts/plex-master/transcode                            205K  43.9T   205K  /mnt/.ix-apps/app_mounts/plex-master/transcode
master-pool/ix-apps/docker                                                     6.82G  43.9T  6.82G  /mnt/.ix-apps/docker
master-pool/ix-apps/truenas_catalog                                             497M  43.9T   497M  /mnt/.ix-apps/truenas_catalog
ssdmaster-pool/ix-apps                                                         2.51G   859G   112K  /mnt/.ix-apps
ssdmaster-pool/ix-apps/app_configs                                              948K   859G   948K  /mnt/.ix-apps/app_configs
ssdmaster-pool/ix-apps/app_mounts                                                96K   859G    96K  /mnt/.ix-apps/app_mounts
ssdmaster-pool/ix-apps/docker                                                  2.30G   859G  2.30G  /mnt/.ix-apps/docker
ssdmaster-pool/ix-apps/truenas_catalog                                          217M   859G   217M  /mnt/.ix-apps/truenas_catalog
root@truenas-master:/home/admin#
root@truenas-master:/home/admin# mount | grep "ix-app"
master-pool/ix-applications on /mnt/master-pool/ix-applications type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/catalogs on /mnt/master-pool/ix-applications/catalogs type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/default_volumes on /mnt/master-pool/ix-applications/default_volumes type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases on /mnt/master-pool/ix-applications/releases type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/k3s on /mnt/master-pool/ix-applications/k3s type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich on /mnt/master-pool/ix-applications/releases/immich type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/plex-master on /mnt/master-pool/ix-applications/releases/plex-master type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich/volumes on /mnt/master-pool/ix-applications/releases/immich/volumes type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich/charts on /mnt/master-pool/ix-applications/releases/immich/charts type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/plex-master/volumes on /mnt/master-pool/ix-applications/releases/plex-master/volumes type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/plex-master/charts on /mnt/master-pool/ix-applications/releases/plex-master/charts type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich/volumes/ix_volumes on /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/plex-master/volumes/ix_volumes on /mnt/master-pool/ix-applications/releases/plex-master/volumes/ix_volumes type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich/volumes/ix_volumes/uploads on /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/uploads type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich/volumes/ix_volumes/profile on /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/profile type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich/volumes/ix_volumes/thumbs on /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/thumbs type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich/volumes/ix_volumes/pgBackup on /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/pgBackup type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich/volumes/ix_volumes/library on /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/library type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich/volumes/ix_volumes/pgData on /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/pgData type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/immich/volumes/ix_volumes/video on /mnt/master-pool/ix-applications/releases/immich/volumes/ix_volumes/video type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/transcode on /mnt/master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/transcode type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/config on /mnt/master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/config type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/logs on /mnt/master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/logs type zfs (rw,noatime,xattr,posixacl,casesensitive)
master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/data on /mnt/master-pool/ix-applications/releases/plex-master/volumes/ix_volumes/data type zfs (rw,noatime,xattr,posixacl,casesensitive)
ssdmaster-pool/ix-apps on /mnt/.ix-apps type zfs (rw,noatime,xattr,posixacl,casesensitive)
ssdmaster-pool/ix-apps/docker on /mnt/.ix-apps/docker type zfs (rw,noatime,xattr,posixacl,casesensitive)
ssdmaster-pool/ix-apps/truenas_catalog on /mnt/.ix-apps/truenas_catalog type zfs (rw,noatime,xattr,posixacl,casesensitive)
ssdmaster-pool/ix-apps/app_mounts on /mnt/.ix-apps/app_mounts type zfs (rw,noatime,xattr,posixacl,casesensitive)
ssdmaster-pool/ix-apps/app_configs on /mnt/.ix-apps/app_configs type zfs (rw,noatime,xattr,posixacl,casesensitive)
overlay on /mnt/.ix-apps/docker/overlay2/d182cc217b762f81680a877bcec30f8b3cbeeebe12873b8c00978a90ae5cfdd3/merged type overlay (rw,relatime,lowerdir=/mnt/.ix-apps/docker/overlay2/l/KVEKJBGJHKAZF46RFH4Z6SNTX4:/mnt/.ix-apps/docker/overlay2/l/XX7MOS4Y6LU3AUG6XMRR7CMIAA:/mnt/.ix-apps/docker/overlay2/l/ZFBXM3Z74NRJVKRSXUQVQJMIBA:/mnt/.ix-apps/docker/overlay2/l/CIOYTJRVRA4MNDDPRYVECDD5WP:/mnt/.ix-apps/docker/overlay2/l/7UWF3H3QJ2ZLBCOGCUFA4HE6ER:/mnt/.ix-apps/docker/overlay2/l/2YGBH6QW5OY2GJ7FFZGSNFRNAQ:/mnt/.ix-apps/docker/overlay2/l/6IPBNEKRB3R7AR4NW5J52N5LCE,upperdir=/mnt/.ix-apps/docker/overlay2/d182cc217b762f81680a877bcec30f8b3cbeeebe12873b8c00978a90ae5cfdd3/diff,workdir=/mnt/.ix-apps/docker/overlay2/d182cc217b762f81680a877bcec30f8b3cbeeebe12873b8c00978a90ae5cfdd3/work)
overlay on /mnt/.ix-apps/docker/overlay2/4703c51a1f53783eaaa1bf9d5498f47bdc188e0e135b07158aa07cfce51deb06/merged type overlay (rw,relatime,lowerdir=/mnt/.ix-apps/docker/overlay2/l/PZOIAKXAEN3J4Q57FV24ESLB65:/mnt/.ix-apps/docker/overlay2/l/VR6QO2X5RNH7RAY7BVDITLDG3F:/mnt/.ix-apps/docker/overlay2/l/ALCFS6TVCLL7HFJTRRLYR36HH3:/mnt/.ix-apps/docker/overlay2/l/KPKRJ5LYPEFNKWW7OTE6MAPGGJ:/mnt/.ix-apps/docker/overlay2/l/Y4UDVYFX4FGODZGBULX6GQ76RW:/mnt/.ix-apps/docker/overlay2/l/OGJQG3ECVAFBWPRDQGMY5I3HLY:/mnt/.ix-apps/docker/overlay2/l/E7Y37AJT4QYFR5J2KJ6WKZ36J3:/mnt/.ix-apps/docker/overlay2/l/XST7O5JMMX6HKKNX6N2WRKMY3G:/mnt/.ix-apps/docker/overlay2/l/3ZGLEKU5Q5NJR6QOCOXBTVJEXF:/mnt/.ix-apps/docker/overlay2/l/LO7CKPTMLC45G2R25T3GC2KUOQ:/mnt/.ix-apps/docker/overlay2/l/FCHLTZMUT462IK3VG4JYXVDXDC:/mnt/.ix-apps/docker/overlay2/l/G5U5IHCEN73ZYUEUOKAZQTENSF,upperdir=/mnt/.ix-apps/docker/overlay2/4703c51a1f53783eaaa1bf9d5498f47bdc188e0e135b07158aa07cfce51deb06/diff,workdir=/mnt/.ix-apps/docker/overlay2/4703c51a1f53783eaaa1bf9d5498f47bdc188e0e135b07158aa07cfce51deb06/work)
overlay on /mnt/.ix-apps/docker/overlay2/227005352a1d49ad8f2c619a1ef5114ff42d0a54861679c4a77ab4ae731e340b/merged type overlay (rw,relatime,lowerdir=/mnt/.ix-apps/docker/overlay2/l/R6DLMBPFZGKYR2ORNAX4FR635B:/mnt/.ix-apps/docker/overlay2/l/NQCUMFGDMWWAMVBGFCH7TRM6ZF:/mnt/.ix-apps/docker/overlay2/l/4PXIQSVDFN63USQFJC5AMDQLVH:/mnt/.ix-apps/docker/overlay2/l/TOWXKD4HLVUOWUZ3NWMIPD35ID:/mnt/.ix-apps/docker/overlay2/l/POADACGCYDX6ZTN43PUJEMERSI:/mnt/.ix-apps/docker/overlay2/l/VSM7JKJRM2MNX54YHDVC2YD4TY:/mnt/.ix-apps/docker/overlay2/l/KGLUJNV5K526HRXUIVTBFWB2AU:/mnt/.ix-apps/docker/overlay2/l/M7Q64IYBNA7FGABYQJ7BET3F2R,upperdir=/mnt/.ix-apps/docker/overlay2/227005352a1d49ad8f2c619a1ef5114ff42d0a54861679c4a77ab4ae731e340b/diff,workdir=/mnt/.ix-apps/docker/overlay2/227005352a1d49ad8f2c619a1ef5114ff42d0a54861679c4a77ab4ae731e340b/work)
overlay on /mnt/.ix-apps/docker/overlay2/4a6e8ad4d86f58f18d21cc237b914af726e88cb3b0e2c78b6c72c1f39dcb45ef/merged type overlay (rw,relatime,lowerdir=/mnt/.ix-apps/docker/overlay2/l/NZCR7AP2SIHCTDBBLVWMPAWDZ3:/mnt/.ix-apps/docker/overlay2/l/KYKQSGDH272Y3ZXBE3UNLRY2PW:/mnt/.ix-apps/docker/overlay2/l/N3OCZZOTQFO4UE4AVFX47L52K4:/mnt/.ix-apps/docker/overlay2/l/3LSEAFVKR3XVO22OAQL7SNK6EQ:/mnt/.ix-apps/docker/overlay2/l/4KNUWLMLZIQKJ7PABUL4UH2XUU:/mnt/.ix-apps/docker/overlay2/l/GPUE4EKLNAOYGOMAZYUIR74F6K:/mnt/.ix-apps/docker/overlay2/l/GS3RBVQFUWMQWFZWVWCC2UCRZG:/mnt/.ix-apps/docker/overlay2/l/F32GIT32AFDDQFGWVRTWU7HFS4:/mnt/.ix-apps/docker/overlay2/l/UMUGOA46O4P67K6G7FF67VXCXT:/mnt/.ix-apps/docker/overlay2/l/C5URFJZQAWRDWUNSIHCBPFP57P:/mnt/.ix-apps/docker/overlay2/l/7WCOGQ73UN662M2NO2WIHDN7WE:/mnt/.ix-apps/docker/overlay2/l/ANRXQGVL3P2WTPIFZ7TLRWC4NJ:/mnt/.ix-apps/docker/overlay2/l/T3C3KF7BDA7T4KRVZ3VXWU2JGZ:/mnt/.ix-apps/docker/overlay2/l/LGNDREBBYUUAJGV33E2VN3QPA5:/mnt/.ix-apps/docker/overlay2/l/K53CGD4NYIGGXSQX27WCZCZ2CW:/mnt/.ix-apps/docker/overlay2/l/6IUSTM5PAYZ65WYCKS6XP4O2TU:/mnt/.ix-apps/docker/overlay2/l/NOKF2ZDWNSPZTFKCID4STPS723:/mnt/.ix-apps/docker/overlay2/l/6SNZ623BVIGYHSXQWBRNOSASSW:/mnt/.ix-apps/docker/overlay2/l/CEXSQM3UHUALR6MQSFDQGRNLXS:/mnt/.ix-apps/docker/overlay2/l/YMDHPZLOBSZCCA4H74MJTWR6VH:/mnt/.ix-apps/docker/overlay2/l/LUK2SRQ7YCDZJ6PNR7L76GKTZ3:/mnt/.ix-apps/docker/overlay2/l/M7Q64IYBNA7FGABYQJ7BET3F2R,upperdir=/mnt/.ix-apps/docker/overlay2/4a6e8ad4d86f58f18d21cc237b914af726e88cb3b0e2c78b6c72c1f39dcb45ef/diff,workdir=/mnt/.ix-apps/docker/overlay2/4a6e8ad4d86f58f18d21cc237b914af726e88cb3b0e2c78b6c72c1f39dcb45ef/work)
overlay on /mnt/.ix-apps/docker/overlay2/21ff40f7d9917f2b74dbdbe80b8e7fca587f36f742369e1196b08e5e1e950704/merged type overlay (rw,relatime,lowerdir=/mnt/.ix-apps/docker/overlay2/l/2WYE7Z2JKOMXU35YPZK2RD3VIL:/mnt/.ix-apps/docker/overlay2/l/S2IAJPI2MFIQ2QBRVKD7QRN77R:/mnt/.ix-apps/docker/overlay2/l/HS7BOYW4DFWRPHTNEXNTH5OHTT:/mnt/.ix-apps/docker/overlay2/l/CFEBEMDWSHNHIKLQDQD3ARLZCM:/mnt/.ix-apps/docker/overlay2/l/FPTWCZ7ONGWOEMBHH2OPXW2AWJ:/mnt/.ix-apps/docker/overlay2/l/NBMSDS36OIT6CA5FFYETIEB7VK:/mnt/.ix-apps/docker/overlay2/l/K3E6I7ZEYHG7XKNBUPSVWIKC4P:/mnt/.ix-apps/docker/overlay2/l/IB4QCUKVK7UOCBX5PYR4BR6DOJ:/mnt/.ix-apps/docker/overlay2/l/54PHYXN6RVYNWUSZHSGPG3U6Z5:/mnt/.ix-apps/docker/overlay2/l/46YPCJVUQRL2S4UDVWXZEO53WR:/mnt/.ix-apps/docker/overlay2/l/P4CUSNVMTNL7XCA4M6J3EDR36A:/mnt/.ix-apps/docker/overlay2/l/YLP55YUDZDZPPQFP4DRXWYRP2G:/mnt/.ix-apps/docker/overlay2/l/M74WJRFP3FN3MVX3QLYQRW2CND:/mnt/.ix-apps/docker/overlay2/l/ORVROMQFMAHHCUQMEX24AILIGO:/mnt/.ix-apps/docker/overlay2/l/F7XIF7GVWAIVPCOPP5Z4BX46QD:/mnt/.ix-apps/docker/overlay2/l/GB3HDZXDWP3XI2FOOGW3OZWYVV:/mnt/.ix-apps/docker/overlay2/l/VL2LXIRWDLTRQ6L5DYQZFRXAIK:/mnt/.ix-apps/docker/overlay2/l/WEFYONORATHRCVLLKHSMH64YZB:/mnt/.ix-apps/docker/overlay2/l/UYE3URZQKSEJFNTMWPUWX25AE6:/mnt/.ix-apps/docker/overlay2/l/SEHBE7QSPCW2D46NTNBAEA5JA2:/mnt/.ix-apps/docker/overlay2/l/A5ABGGKWE4QKKBI4CXL3MVD77V:/mnt/.ix-apps/docker/overlay2/l/7YQ76GDLSLKN3DHWHYLEJB64EB,upperdir=/mnt/.ix-apps/docker/overlay2/21ff40f7d9917f2b74dbdbe80b8e7fca587f36f742369e1196b08e5e1e950704/diff,workdir=/mnt/.ix-apps/docker/overlay2/21ff40f7d9917f2b74dbdbe80b8e7fca587f36f742369e1196b08e5e1e950704/work)
root@truenas-master:/home/admin#

As you can see, there are some datasets / mounts for the new SSD pool (ssdmaster-pool), but the majority is for the old HDD pool (master-pool).

Below some further investigation on the actual usage

root@truenas-master:/home/admin# grep -R "/mnt/master-pool/ix-applications" /proc/*/mounts 2>/dev/null   | sed -n 's|^/proc/\([0-9]\+\)/mounts:.*$|\1|p'   | sort -un   | while read -r pid; do ps -p "$pid" -o pid=,comm=,cmd=; done   | awk '!/\[.*\]/'   | awk '{print $2}'   | sort   | uniq -c   | sort -nr | head -50
      6 middlewared
      5 containerd-shim
      5 bash
      4 docker-proxy
      2 systemd
      2 sort
      2 python3
      2 nginx
      2 netdata
      2 chronyd
      1 zed
      1 wsdd.py
      1 winbindd
      1 wb-idmap
      1 uniq
      1 truenas_audit_h
      1 systemd-udevd
      1 systemd-machine
      1 systemd-logind
      1 systemd-journal
      1 syslog-ng
      1 sudo
      1 su
      1 sshd
      1 smbd-notifyd
      1 smbd-cleanupd
      1 smbd
      1 smartd
      1 python.d.plugin
      1 nscd
      1 nmbd
      1 lxcfs
      1 libvirtd
      1 head
      1 gssproxy
      1 dockerd
      1 dbus-daemon
      1 cron
      1 containerd
      1 cli
      1 certmonger
      1 awk
      1 avahi-daemon
      1 auditd
      1 asyncio_loop
      1 (sd-pam)
root@truenas-master:/home/admin# grep -R "/mnt/master-pool/ix-apps" /proc/*/mounts 2>/dev/null   | sed -n 's|^/proc/\([0-9]\+\)/mounts:.*$|\1|p'   | sort -un   | while read -r pid; do ps -p "$pid" -o pid=,comm=,cmd=; done   | awk '!/\[.*\]/'   | awk '{print $2}'   | sort   | uniq -c   | sort -nr | head -50
root@truenas-master:/home/admin# grep -R "/mnt/.ix-apps" /proc/*/mounts 2>/dev/null | grep ":master-pool"

Looking at all of this, these are my conclusions (please let me know if I’m wrong!)

  1. Both master-pool/ix-apps and ssdmaster-pool/ix-apps have /mnt/.ix-apps as mountpoint, so they can’t be mounted simultaniously.

  2. From the mount command, I can see that only the ssdmaster-pool/ix-apps dataset and its children are currently mounted and the master-pool/ix-apps dataset and its children are NOT mounted.

  3. From /proc/*/mounts I also conclude that no processes are currently using the master-pool/ix-apps datasets.

  4. So I THINK it is safe to remove the master-pool/ix-apps dataset and it’s children. Can someone please confirm? Or is there perhaps some hidden dependency from TrueNAS that makes removing them not a good idea?

  5. master-pool/ix-applications IS still mounted and IS also used by many processes, even though it seems no longer “used” by TrueNAS applications, as reinstalling my apps on ssdmaster-pool didn’t create any ssdmaster-pool/ix-applications datasets at all???

  6. I have no clue what to do with this one… Should I remove these datasets? Can I even remove them, as they are used by processes?

I don’t think you actually listed processes that use those datasets. I am not exactly sure what your dense one-liner does, but it never looks at open files. I’d guess you only listed process that can see the mountpoints, but you didn’t check if the processes actually read/write any files from/to them.

Using fuser -m /mnt/.. is probably closer to what you want.

If I recall correctly, ix-applications is the dataset used in the old app system. They shoudn’t have any use after the migration to the docker based system.

Instead of removing the dataset, maybe try some non-destructive ways first. You could set it to read-only (via GUI). You could also rename the dataset or just disable automount - but these options don’t seem to be available in the GUI.

1 Like

I am a bit surprised you didn’t get any datasets in ssdmaster-pool/ix-apps/app_mounts after reinstalling the apps. What storage type did you use when reinstalling the apps? Did you use host path or ixVolume?

1 Like

I gotta admit that I got that “brilliant” one-liner from Copilot, without really understanding what exactly it does :thinking:

I did try with the fuser command at first, but that didn’t even return any processes (besides the kernel) for my ssdmaster-pool/.ix-apps mountpoint that contains running applications. So I considered my fuser command “invalid” because of that…

root@truenas-master:/home/admin# fuser -vm /mnt/master-pool/ix-applications/
                     USER        PID ACCESS COMMAND
/mnt/master-pool/ix-applications:
                     root     kernel mount /mnt/master-pool/ix-applications
root@truenas-master:/home/admin# fuser -vm /mnt/.ix-apps
                     USER        PID ACCESS COMMAND
/mnt/.ix-apps:       root     kernel mount /mnt/.ix-apps
root@truenas-master:/home/admin#

I also found some topics about ix-applications maybe being the “old app system”, but I didn’t really find (yet) any clear instructions on how to properly remove it and if this is advisable?

For my new app installations, I only used host paths and tmp folders, never ixVolumes.

Things can be a bit tricky. fuser is correct in this case. There are no processes with open files in ssdmaster-pool/.ix-apps. If you do something like this:

cd /mnt/.ix-apps
fuser -mM /mnt/.ix-apps

This will show the shell process, because the shell process has the folder open as current working directory.

If you do fuser -mM /mnt/.ix-apps/docker you would get the docker process.

In general, fuser might not be that useful for you - it can only tell you if there are processes which currently have open files within a dataset. I don’t think it recursively goes across mountpoints, so you have to be sure to pick the correct mountpoint.

Good! That also explains why you don’t have any app_mounts datasets. If none of your hostpaths use the old ix-applications folders then you’re all good.

In my installation I have removed the ix-applications datasets without any repercussions.

There was a section in the manual about ix-applications, but I can’t find it now. The most official comment I could find is this:

1 Like

Most app docs have moved over to apps.truenas.com.

Specifically you’re looking for https://apps.truenas.com/getting-started/app-storage/#ix-apps-dataset

2 Likes

I read all posted above and decided to try something mainly because of reading your official document from TrueNAS. It said that “ix-applications is only relevant if you want to be able to revert to 24.04 and you can delete the dataset if you don’t”.

So I just tried to delete the dataset in the GUI, without even stopping my apps. And it worked ok. TrueNAS didn’t complain about it being in use and after a reboot, my apps still worked fine.

Thanks both!

Now only this dataset remains:

master-pool/ix-apps

Any suggestions on this?

I’d say remove master-pool/ix-apps. I agree with your conclusion that those aren’t mounted. Might have to use CLI if they aren’t shown in the GUI. zfs destroy -nvr master-pool/ix-apps should do the job (With the ‘n’ option that command only prints what it would do. Remove the ‘n’ to do the real, irrecoverable deletion).

1 Like
root@truenas-master:/home/admin# zfs list | grep ix
ssdmaster-pool/ix-apps                                         2.52G   859G   112K  /mnt/.ix-apps
ssdmaster-pool/ix-apps/app_configs                              948K   859G   948K  /mnt/.ix-apps/app_configs
ssdmaster-pool/ix-apps/app_mounts                                96K   859G    96K  /mnt/.ix-apps/app_mounts
ssdmaster-pool/ix-apps/docker                                  2.30G   859G  2.30G  /mnt/.ix-apps/docker
ssdmaster-pool/ix-apps/truenas_catalog                          222M   859G   222M  /mnt/.ix-apps/truenas_catalog
root@truenas-master:/home/admin#

Ah… That feels so much better :slight_smile:

Not sure how to set 2 solutions