Several of my jail roots disappeared

TrueNAS General
1

Hi,

This is almost certainly user error and lack of experience, but I’m worried about a “trial and error” approach to fixing this making it worse, so grateful for any advice.

I am on TrueNAS-13.0-U6.1 and this morning I noticed one of my jails (“web”) was down. I couldn’t work out why but I restarted it and that worked.

At the same time, I noticed two old jails that are no longer user. I deleted those via the jails web UI.

Now, I have a whole bunch of jails (including the one that was down unexpectedly this morning) that refuse to start. The error message is:

“[Errno 2] No such file or directory: ‘/mnt/NAS/iocage/jails/web/root/etc/rc.conf’”

(Here, ‘web’ is the name of the jail, but it’s the same for the other jails).

And indeed, the directory /mnt/NAS/iocage/jails/web/root does not exist:

root@freenas:/mnt/NAS/iocage/jails/web # ls
config_backup.json	config.json		fstab

The other thing I have noticed is that all the “dead” jails are on 11.2-RELEASE-p9 and all the jails that work are on other releases. But that could be a red herring since most of my jails were created at the same time and I’ve only updated/subsequently created a handful of them.

I am trying to understand what might have happened.

I am also trying to find the best way to get it back. I have cloud backups that would allow me to re-create and re-configure the relevant services, but I also assume it should be possible to do this via snapshots.

For example, in Storage → Snapshots in the UI I have a daily snapshot in for “NAS/iocage/jails/web” (and the other jails).

But I have also never acutally tried to restore data from a snapshot (lucky me!) so I’m not sure how to best go about it, and some of the documentation confuses me enough that I’d be worried about making it worse. :slight_smile:

Thank you,
Martin

2

I think you should consider to upgrade TN on 13.3, and upgrade all jails.
But depending on what exactly you are running (11.2 Is really really old) this can be a bit tricky

3

Would that get my files back? The question here is why the ‘root’ directory under all the jails disappeared. I’d be nervous about starting an upgrade in the middle of trying to restore files. Note it’s only some of the older jails that are on 11.2, TrueNAS itself is 13.0.

4

Probably Is related to the jail type (base or clone).

Exactly, what service are affected? Have you mounted data outside jails via mount point?
Because in this case, normally a clean install of jails with same mount point Is preferred (but you are on 13.0…)

6

The files are gone. I think I need help restoring via previous snapshots.

What I mean is the whole root file system of the jail would normally be in a folder called ‘root’ under ‘/mnt/NAS/iocage/web’ and that whole folder is missing. On the ‘web’ jail and multiple other jails.

7

I think what is happening is that the ‘root’ datasets under several of the jails have disappeared.

image
image2572×966 244 KB

So for example flickrbackup and hass are newer jails that work. logging, media, node-red, mosquitto, and web are all jails that stopped working.

I can’t explain why though. My theory is I deleted another jail (not shown in this list) through the UI and somehow it nuked a whole bunch of datasets. But why?

8

Clone jail are tethered to the original release they were created, that’s why i assume somehow deleting those old jails put you in this situation. I faced a similar problem, but i can’t say you more, i lack too.
So you probably need restore deleted jails too.

Restore from snapshot should be easy via GUI (find the one you need restore and confirm), but if you want to be 100% sure in what u doing try replicate the snap on another datasets and check data.
Thats why i was talking about to directly ricreate those jails, but if your data were inside you can’t do that without a prevoius restore

9

What does this reveal?

iocage list -l
10

Same thing as the web interface. A list of jails, many of which are down (and won’t start).

11

To see which ones are clone jails and base jails…