Cleaning up a mistaken snapshot clone

Novice home user, need help un-tangling a failed failed attempt to fix an issue, and then actually fixing the problem I was trying to solve in the first place.

I have a TrueNAS Scale server running, among other things, a Jellyfin server. Earlier today, I noticed all of my saved playlists had been cleared. The playlists themselves all still existed, but none of them had any songs in them. I know that in the filesystem those playlists are just config files listing the songs to include, so I though I could use snapshots to get them back. I have snapshots from before they cleared out, but didn’t want to roll all my other settings and stuff back to that point. So, I figured I’d just build a clone from the snapshots, go to the directory in the clone where the playlists are stored (/ix-applications/releases/jellyfin/volumes/ix_volumes/config/data/playlists/), copy out the old playlist config files, and use them to rebuild.

I cloned a snapshot of Apps/ix-applications/ from March to a new dataset – Apps/rollback – but have ran into a few issues.

  1. I guess I misunderstood what snapshots are, because this data set doesn’t seem to actually contain what I’m looking for? I assumed a cloned snapshot of FOLDER from DATE would just be everything that was in FOLDER on DATE, but this looks completely different, and doesn’t have the playlist config files I was looking for.

  2. But now I can’t delete it, because when I try I get this error

At the very least, I’d appreciate if someone could help me delete this snapshot clone, since it’s clearly not what I need. And if anyone has ideas on how to accomplish what I’m actually trying to do here, I’d appreciate help with that as well.

Thanks!

When you decide to continue, create a checkpoint for your pool (which you will discard after you’re finished):

zpool checkpoint <poolname>

Keep in mind that if you do have to rewind to the checkpoint, you will lose everything you did after you created it.

Ideally, you’re not downloading or saving files (or running services) while you’re troubleshooting this.

The dataset Apps/ix-applications is its own dataset, which is a parent to other child datasets underneath. It’s likely a very “hollow” dataset that serves as a “placeholder” for the important datasets underneath.

This is why you’re not seeing the files in the clone.


Before you attempt to “clone” anything, if all you need to do is retrieve files from a snapshot, you can browse and copy files from the snapshot itself.

Just browse to the “hidden” special location /mnt/<path><to><dataset>/.zfs/snapshot/, where you will find a list of snapshots presented as directories for easy navigation.


What is the state of clones on your pool?

zfs list -r -t filesystem -o name,origin Apps

I tried running both of the commands you suggested in the truenas shell under System Settings, and neither of them worked. it returned “command not found” for both zfs and zpool.

Screenshot_20240604_203429

is there a different shell or terminal I need to run these in?

as for looking around in hidden directories, what’s the best way to do that? should i just cd into them in the truenas shell?

Do you know how to enter an SSH session?

It’s better to paste text instead of screenshots of text.

Not really, I’ve always done everything through the TrueNAS web ui.

I assume step 1 is enabling the SSH service in the TrueNAS web ui though.

As long as you can copy and paste the text, rather than always post a screenshot.


Replace zfs and zpool with /usr/bin/zfs and /usr/bin/zpool respectively.

Prepend the commands with sudo for operations that require root privileges.

still didn’t work. copied from truenas shell

admin@truenas[~]$ sudo /usr/bin/zfs list -r -t filesystem -o name,origin Apps
[sudo] password for admin: 
sudo: /usr/bin/zfs: command not found
admin@truenas[~]$ sudo /usr/bin/zpool checkpoint Apps 
sudo: /usr/bin/zpool: command not found
admin@truenas[~]$ 

Sorry.

/usr/sbin/

worked this time. zfs list below

most of these are the standard directories created for truenas apps. notes on the ones that aren’t

  • Apps/immich and sub-directories is storage for an Immich cloud photos library. I wanted them to be stored on my Apps SSD for performance, and be backed up regularly to my spinning disks, but had trouble getting the backup to work when the photos were stored with the rest of immich in Apps/ix-applications. Putting them here was a clunky workaround, but it did work. Unrelated to the issue I’m trying to fix here.

  • Apps/ix-applications/releases/jellyfin/volumes/ix_volumes/config-27-clone & Apps/rollback are both snapshot clones i made when troubleshooting how to do this.

  • Apps/vpn is just a folder with .conf file for my VPN. Makes it easier to set up apps to use the VPN.

admin@truenas[~]$ sudo /usr/sbin/zfs list -r -t filesystem -o name,origin Apps
[sudo] password for admin: 
NAME                                                                                        ORIGIN
Apps                                                                                        -
Apps/immich                                                                                 -
Apps/immich/library                                                                         -
Apps/immich/pgBackup                                                                        -
Apps/immich/pgData                                                                          -
Apps/immich/profile                                                                         -
Apps/immich/thumbs                                                                          -
Apps/immich/uploads                                                                         -
Apps/immich/video                                                                           -
Apps/ix-applications                                                                        -
Apps/ix-applications/catalogs                                                               -
Apps/ix-applications/default_volumes                                                        -
Apps/ix-applications/k3s                                                                    Apps/rollback@ix-applications-backup-system-update--2024-03-05_00:59:50
Apps/ix-applications/k3s/kubelet                                                            -
Apps/ix-applications/releases                                                               -
Apps/ix-applications/releases/ddns-updater                                                  -
Apps/ix-applications/releases/ddns-updater/charts                                           -
Apps/ix-applications/releases/ddns-updater/volumes                                          -
Apps/ix-applications/releases/ddns-updater/volumes/ix_volumes                               -
Apps/ix-applications/releases/ddns-updater/volumes/ix_volumes/data                          -
Apps/ix-applications/releases/filebrowser                                                   -
Apps/ix-applications/releases/filebrowser/charts                                            -
Apps/ix-applications/releases/filebrowser/volumes                                           -
Apps/ix-applications/releases/filebrowser/volumes/ix_volumes                                -
Apps/ix-applications/releases/filebrowser/volumes/ix_volumes/config                         -
Apps/ix-applications/releases/homarr                                                        -
Apps/ix-applications/releases/homarr/charts                                                 -
Apps/ix-applications/releases/homarr/volumes                                                -
Apps/ix-applications/releases/homarr/volumes/ix_volumes                                     -
Apps/ix-applications/releases/homarr/volumes/pvc-4e4ad4b1-672f-44e6-8fe2-8a558980ebf9       -
Apps/ix-applications/releases/homarr/volumes/pvc-56e6078a-ffa0-4bcf-8b3d-986e3610a646       -
Apps/ix-applications/releases/homarr/volumes/pvc-bd7be45c-e8c7-4bc9-8ad5-de845e89bf4d       -
Apps/ix-applications/releases/immich                                                        -
Apps/ix-applications/releases/immich/charts                                                 -
Apps/ix-applications/releases/immich/volumes                                                -
Apps/ix-applications/releases/immich/volumes/ix_volumes                                     -
Apps/ix-applications/releases/immich/volumes/ix_volumes/library                             -
Apps/ix-applications/releases/immich/volumes/ix_volumes/pgBackup                            -
Apps/ix-applications/releases/immich/volumes/ix_volumes/pgData                              -
Apps/ix-applications/releases/immich/volumes/ix_volumes/profile                             -
Apps/ix-applications/releases/immich/volumes/ix_volumes/thumbs                              -
Apps/ix-applications/releases/immich/volumes/ix_volumes/uploads                             -
Apps/ix-applications/releases/immich/volumes/ix_volumes/video                               -
Apps/ix-applications/releases/jellyfin                                                      -
Apps/ix-applications/releases/jellyfin/charts                                               -
Apps/ix-applications/releases/jellyfin/volumes                                              -
Apps/ix-applications/releases/jellyfin/volumes/ix_volumes                                   -
Apps/ix-applications/releases/jellyfin/volumes/ix_volumes/cache                             -
Apps/ix-applications/releases/jellyfin/volumes/ix_volumes/config                            -
Apps/ix-applications/releases/jellyfin/volumes/ix_volumes/config-27-clone                   Apps/ix-applications/releases/jellyfin/volumes/ix_volumes/config@27
Apps/ix-applications/releases/jellyfin/volumes/ix_volumes/transcodes                        -
Apps/ix-applications/releases/joplin                                                        -
Apps/ix-applications/releases/joplin/charts                                                 -
Apps/ix-applications/releases/joplin/volumes                                                -
Apps/ix-applications/releases/joplin/volumes/ix_volumes                                     -
Apps/ix-applications/releases/joplin/volumes/ix_volumes/pgBackup                            -
Apps/ix-applications/releases/joplin/volumes/ix_volumes/pgData                              -
Apps/ix-applications/releases/nginx-proxy-manager                                           -
Apps/ix-applications/releases/nginx-proxy-manager/charts                                    -
Apps/ix-applications/releases/nginx-proxy-manager/volumes                                   -
Apps/ix-applications/releases/nginx-proxy-manager/volumes/ix_volumes                        -
Apps/ix-applications/releases/nginx-proxy-manager/volumes/ix_volumes/certs                  -
Apps/ix-applications/releases/nginx-proxy-manager/volumes/ix_volumes/data                   -
Apps/ix-applications/releases/plex                                                          -
Apps/ix-applications/releases/plex/charts                                                   -
Apps/ix-applications/releases/plex/volumes                                                  -
Apps/ix-applications/releases/plex/volumes/ix_volumes                                       -
Apps/ix-applications/releases/plex/volumes/ix_volumes/config                                -
Apps/ix-applications/releases/plex/volumes/ix_volumes/data                                  -
Apps/ix-applications/releases/plex/volumes/ix_volumes/transcode                             -
Apps/ix-applications/releases/qbittorrent                                                   -
Apps/ix-applications/releases/qbittorrent/charts                                            -
Apps/ix-applications/releases/qbittorrent/volumes                                           -
Apps/ix-applications/releases/qbittorrent/volumes/ix_volumes                                -
Apps/ix-applications/releases/qbittorrent/volumes/ix_volumes/config                         -
Apps/ix-applications/releases/qbittorrent/volumes/pvc-ac1a473b-2a2a-4ef1-9457-4c80967c124f  -
Apps/rollback                                                                               -
Apps/vpn                                                                                    -

Lastly, i suspect this doesn’t matter, but my Jellyfin app is the stock one from TrueNAS.

The dataset Apps/ix-applications/k3s is a clone.

I’m not familiar with it. Is it a standard iX dataset, or did you inadvertently create it via cloning from a snapshot?

Meanwhile, Apps/rollback is a regular dataset, not birthed as a clone from any snapshot.

It’s like you mixed up certain steps. The “dataset” Apps/ix-applications/k3s is in fact a “clone” from a snapshot that exists on Apps/rollback.

that’s possible. again, i’m a novice, and i was troubleshooting this, but that’s not a directory i would have created myself by choice.

Apps/rollback “has dependent clones” per the screenshot in my initial post - at Apps/ix-applications/k3s & Apps/ix-applications/k3s/kubelet.

But to go back to my original issue - it sounds like you think /mnt/Apps/.zfs/snapshot/ might have the playlist config files I’m looking for, but it’s a hidden directory. should i just cd into it with the truenas shell and look around?

Sure, but you still have this strange issue of mysterious clones.

Maybe I just don’t understand SCALE.

Is <poolname>/rollback something you created, or is it a default iX dataset?


That will be mostly empty.

You need to access the “hidden” .zfs/snapshot directory for the specific dataset.

Such as,

Apps/ix-applications/releases/jellyfin/volumes/ix_volumes/config/.zfs/snapshot/

Well, you clearly understand SCALE better than i do at least.

Apps/rollback is not a default directory. I created it by selecting a snapshot i wanted to clone, selecting “Clone to New Directory.”

  • When i selected that, TrueNAS suggested a directory to create, but it was buried deep in apps/ix-applications.

  • I had tried cloning a snapshot to a new directory before, and used the default directory truenas suggested (Apps/ix-applications/releases/jellyfin/volumes/ix_volumes/config-27-clone), but found i wasn’t able to get into it and look around.

  • So, when i tried a second time, i overwrote the directory suggestion, and told TrueNAS to clone it to Apps/Rollback instead.

  • I was able to get into this clone, but i found it didn’t contain what i was looking for. So, i decided to stop troubleshooting on my own, and create an account here to ask for help.

I appreciate your help on this. I understand now that i may be able to find and restore the playlist’s I’m looking for by using the TrueNAS shell to CD into hidden ZFS directories. I might not have time to do that for a few days though. I may need to come back and ask for more help if i run into issues there. Even if i find them, i may still need some help cleaning up the cloned datasets my troubleshooting created. Can we leave this thread open for a little bit linger, or should i just open a new one if i need further assistance?

Thanks!

1 Like

You can ask in here. No need to start a new thread.[1]


  1. Under the assumption that starting a new thread might earn you “points”, since I’m trying to thin out the competition. :smirk: ↩︎

Update: I was able to restore the playlists over the weekend. I got all the old playlist.xml files for each playlist from a snapshot clone by CD ing into it in the TrueNAS shell, copied them out to a directory i could more easily access in a GUI, and used them to rebuild what I’d lost.

A note to anyone in the future with a similar issue who finds this in a search - just overwriting the current playlist.xml files on the playlists with the ones i’d recovered didn’t fix this, likely due to Jellyfin recently enabling public and private playlists. Previously, all these playlists were public, but when tried just overwriting their blank playlist.xml files with the ones i’d recovered, they all switched to private. At this time, there’s no way to change an existing playlist from private to public in Jellyfin’s GUI, and i didn’t want to poke around in Jellyfin’s huge database file and figure out how to do it in there. therefore, i deleted the existing playlists, created new public ones with the same names, and then copied in the old playlist.xml files to set their contents.

now, i could use some help cleaning up the mess i made troubleshooting this. there are two snapsnot clone datasets i don’t need anymore, and would like to delete, but can’t figure out how to remove in Truenas’s gui.

/mnt/Apps/rollback/

  • I can see this one in Truenas’s datasets gui, but when i try to delete it i get this error
[EFAULT] Failed to delete dataset: cannot destroy 'Apps/rollback': filesystem has dependent clones use '-R' to destroy the following datasets: Apps/ix-applications/k3s/kubelet Apps/ix-applications/k3s
  • does this mean i need to rm -r those two datasets in the truenas shell first, then delete /mnt/Apps/rollback in the gui?

/mnt/Apps/ix-applications/releases/jellyfin/volumes/ix_volumes/config-27-clone/

  • i can’t see this one in the datasets gui, since it won’t let me see sub directories of mnt/Apps/ix-applications.

  • so should i just rm -r this in the truenas shell too? or is there a better way to do it?