I have an Emby server on my Synology NAS. I also have a TrueNAS, which is overflow storage. I mount it to the Synology so Emby can see the files. Had to shut everything down recently and upon booting the mount failed. That magically resolved itself, but it gets weirder.
When remounting, I discovered a bunch of files were in the mount’s target folder, so it wouldn’t let me mount there. I looked and it was mostly just media NFO files. Those sometimes get stuck when media gets deleted, so I just wiped the whole folder and mounted. But…when re-ingesting, Emby proceeded to write new NFO files for hundreds of media items, suggesting that the ones left over in the mount target had somehow been removed from the TrueNAS? I have no idea what’s going on here.
I have a snapshot from before the shutdown, so fingers crossed I can just roll back to that, but I’m very confused.
EDIT: Should I snapshot now? It’s bad data but if I’m going to attempt a rollback it might be a good idea?
Rollback is a dangerous operation that causes any configured replication tasks to fail. Replications use the existing snapshot when doing an incremental backup, and rolling back can put the snapshots out of order.
I’m not sure what this means? The suggested alternative doesn’t seem workable for me because I don’t even know what data is missing or altered, I just want to set the whole thing back where it was on October 1.
yes, snapshot now before you touch anything. a rollback destroys everything newer than the snapshot you roll back to, snapshots included, and itll break replication running off that dataset. safest move is cloning the pre-shutdown snapshot to a temp dataset and copying what you need out of it. and the folder you wiped was synology-local from when the mount was down, so your truenas data is almost certainly fine as is.
those leftover nfo files in the mount target were probably local on the synology while the share was unmounted. apps write into the empty folder when the mount is down, so wiping that local dir was fine.
emby rewriting hundreds of nfos after remount doesnt always mean the media on truenas got deleted. it often just rescan-rewrites sidecars.
b4 any rollback:
-take a fresh snapshot now like u said
-zfs list -t snapshot | grep -i yourdataset
-cd into the old snap under /mnt/POOL/DATASET/.zfs/snapshot/SNAPNAME and see if the media + nfos are still there
-zfs diff POOL/DATASET@SNAP | head -100
if only nfos are missing, copy them back from the snap. full rollback nukes everything written after that snap and can break replication, so only do it if whole folders of media are actually gone.
You can also make a checkpoint of the pool before doing anything that involves rolling back to a snapshot or destroying snapshots or files. zpool checkpoint mypool zpool status mypool | grep checkpoint
Worst-case scenario if something goes horribly wrong or you make a mistake, you can rewind the entire pool back to the checkpoint and you’ll be back to where you are right now before you had done anything.
So I need to copy the snapshots to elsewhere then? I didn’t see anything about that in the documentation.
I don’t know what that means, though.
Like I said, I have no idea what I need to replace…would cloning it require the amount of disk space the snapshot says it occupies, or the amount of space the data on the disk occupies? Because if it’s the latter I don’t have that much free space.
The Date Added on several movies is today. Custom genres have been wiped from one of them.
Now, there are other nfo files with a modified date of today which do show the correct added date, so it’s unclear why they’re listed as having been modified today, and maybe it isn’t an issue, but even so.
And that’s just what I’ve found. Who knows what else is going on.
What do these steps do?
Is there a quick explanation of checkpoints and how they work? The link is to a lengthy thread which links to another lengthy thread…
Okay, I think I’ve figured out why there were files on the Synology. The scheduled task ran to back up the TV intro times, and save it to the NFO for that media item. Obviously those NFOs got created as part of that process, because the intro backup task was being told the items still existed.
The end result is that all my TV intro/credits times have been wiped, and the NFOs backing them up have been wiped, so I definitely need to revert to the old NFO files so I can then restore them.
I don’t understand why a handful of movies came along for the ride though, or why the NFOs got overwritten once I reconnected the mount.
The commands were showing you how to list the Snapshots for your dataset, telling you how to CD (change directory) into a snapshot and how to list the differences between the snapshot and the data.
That is a basic summary.
You can look up commands on the system by using the MAN pages. man zpool will bring up the manual page for the zpool command. You can also search the internet. One resource for the ZFS commands is OpenZFS Documentation — OpenZFS documentation
Okay…and now what? It’s just listing a whole lot of files and folders, though it does seem much shorter than the list that I get when I sort by Date Modified in Windows Explorer. There doesn’t seem to be any indication of what the difference is (is it altered, is it missing in the snapshot, is it missing in the pool), and it’s very difficult to read because spaces are rendered as “\0040.” I don’t know what the “head -100” bit does, the documentation you linked to doesn’t mention it on the Snapshots page and just searching the documentation for “head” returns waaaay too many irrelevant results to sift through. Is it limiting it to 100 results? Because otherwise the list of altered files seems way too short.
I know for a fact there are files on the dataset now that aren’t in the snapshot—new artwork was downloaded for some of my movies which had no artwork available when I added them—that is not listed when I run the diff command.
The head part is listing the first 100 lines, like you thought. You can type man head into the shell and it will bring up the manual page for that command.
You might try looking for online videos of ZFS and comparing snapshots or something along those lines. It is a bit hard to describe over the forums.
Not sure what the point is of running diff at all TBH. We’d already established that there have been files modified. If we were listing files that only existed in one or the other I’d get that, but files that exist in both and have been updated since the dismount are less relevant given there are too many of them to manually copy them over.
I can’t figure out how to multi quote on mobile…the problem at this point is I’m on day four of no real access to my media server and I just want to get the affected files back to where they were before the dismount ASAP.
Rolling back is no worse than restoring from a backup of the same snapshot, and you apparently have no replication scheme to be broken by a rollback.
If you’re satisfied that no file written after the snapshot was taken is worth keep, just proceed. (Making a checkpoint, just in case, might still be a good idea.)
I don’t even know what this replication thing everyone keeps talking about is. The documentation kinda implies creating snapshots is involved in it, but it’s unclear.
I have made a point of not touching the TrueNAS for anything not related to this snapshot process since the issue occurred.
Made a checkpoint as soon as @winnielinnie mentioned them. What I’m not clear on is whether there’s data I need to move off the TrueNAS relating to the checkpoint, or if it should just stay on the TrueNAS. Something was mentioned earlier about needing to copy over snapshots.
You’re not going to do anything with the pool checkpoint. Ideally, you would be fine even if you didn’t take a checkpoint. The only reason to create one is to be able to rewind your entire pool back to a specific point in time if something goes horribly wrong with rolling back snapshots or deleting files, datasets, or snapshots.
To know exactly what that point in time is, you can check with zpool status:
zpool status -v mypool | grep checkpoint
Everything done after that time will be gone if you rewind your pool back to the checkpoint.
Once you’re done doing what you want to do, and you are sure everything is fine, you can discard the checkpoint:
zpool checkpoint -d mypool
You’ll probably never end up resorting to a pool rewind, since that would only be necessary for emergency situations or if something goes wrong.
