How do you backup app data in ElectricEel?

I’m a new user of TrueNAS, started with the previous version of scale, installed apps using default settings, so have an ix-applications dataset, then upgraded to ElectricEel.

What did the upgrade process do? Did it just do a move of the data from ix-applications to ix-apps dataset and mount it as /mnt/.ix-apps? What are the few gigs of stuff left behind?

Is the ix-applications dataset definitely safe to delete now?

Why is the ix-apps dataset hidden and not displayed in the UI anywhere?

How can I back it up? I need to wipe my main data pool that ix-apps is on, recreate the pool and then restore ix-apps. How?

(I’m backing all my own datasets up by having them SMB shared, mounted on another machine, and running rsync on that other machine.)

Someone from iX explained that they clone the latest snapshot and then promote the clone to convert whatever has to be converted. (Look up “cloning” in the documentation.)
That way an untouched copy is preserved if the user wants to revert to Dragonfish.

I suppose so, if you’re confident that you’ll never go back to Dragonfish, and are also ready to delete previous boot environments.
Take some time for this decision…

Why do you “need” to do that?

What does it mean that I still have a few GB of data visible in my ix-applications dataset, that contains files from my apps that AFAICT upgraded just fine and now have ~10x as much data in /mnt/.ixapps?

I accidentally created my pool as DRAID2 instead of RAIDZ2.

Lets confirm this first.

ix-applications may be the volume in use

ix-apps may be the cloned copy used for rollback if needed. Its probably hidden to avoid it being used.

The expectation is that this rollback process is rarely used… but it was possible for testing.

I can check during the week, but if anyone else can confirm that would be useful.

After that we would need to confirm if rsync can be used to backup… we would normally recommend zfs send/receive… its more transparent.

NOTE: THIS IS INCORRECT. SEE BELOW.

But what does the following mean? (All my apps are off, so I shouldn’t be generating any data over time.)

$ df -h /mnt/.ix-apps
Filesystem      Size  Used Avail Use% Mounted on
NVMes/ix-apps    12T  1.0M   12T   1% /mnt/.ix-apps

$ sudo du -sh /mnt/.ix-apps
263G    /mnt/.ix-apps

$ df -h /mnt/NVMes/ix-applications 
Filesystem             Size  Used Avail Use% Mounted on
NVMes/ix-applications   12T  2.0M   12T   1% /mnt/NVMes/ix-applications

$ sudo du -sh /mnt/NVMes/ix-applications
264G    /mnt/NVMes/ix-applications

$ sudo du -sh /mnt/.ix-apps
263G    /mnt/.ix-apps

$ sudo find /mnt/.ix-apps -type f | wc -l
6285246

$ sudo find /mnt/NVMes/ix-applications -type f | wc -l 
6298288

Why do none of the numbers match up? Why do they claim to be mounts of different filesystems?

I can’t trust this at all. Just looking inside ix-applications and seeing stuff is in there doesn’t provide enough information to know it’s the “active” dataset.

Because they are. .ix-apps != .ix-applications
As for numbers… “space” is a complicated notion with ZFS, or any other CoW filesystem, and df or du only report on abstractions, not on the low level situation inside ZFS.

Thinking about it, I’ll note that upgrading turned all apps on, and I had to turn them off again, so it’s most likely that ix-applications is current because it produced files during that on period, and .ix-apps is static for roll-back purposes only, as you suggest.

2 Likes

But being off by a factor of 10 makes the Datasets UI pretty useless and very misleading doesn’t it? It telling me my ix-applications was only 2.75GiB is what made me think my stuff must have been moved out of there in the first place.

Is it a bug?

Which line says 2.75GiB?

Clones are complicated, because they are not full copies… they include links to other data.

The screenshot of the UI.

Linux reports the apparent space used by a directory… ignoring compression, deduplication, cloning.

ZFS and TrueNAS report the actual space used… not including RAID overheads.

So it’s a bug then? Because I sudo tar -czvpf my ix-applications dataset to aid backing up and restoring it, and that was a 127GiB file. Since I don’t have deduplication on, there’s no way “actual space used” could only have been 2.75GiB.

No, I think its getting the benefits of the clone process… which is a type of deduplication.

The UI should tell users how much space their datasets are using. That’s what it purports to do. If it only shows how much space has been used since a hidden clone was taken as part of an upgrade, then it’s a bug.

They should do whatever they need to do (eg. add the size of the clone to what is displayed now) to make the UI report a sensible, meaningful number.

it’s most likely that ix-applications is current

.ix-apps is the one in use in 24.10 and the previous ix-applications is retained in case you need to roll back. A fresh install of 24.10 does not create an ix-applications dataset.

Where is this documented?

And having wiped my only data pool, which moved .ix-apps to my boot pool (leaving it apparently empty, by the way), how do I now get .ix-apps on my new data pool and restore my app data to that location?

It’s mentioned in a few spots in: TrueNAS Apps | TrueNAS Documentation Hub, but we can work on a more focused section explaining the file structure.

As for the second question, I’m not entirely sure how you would go about fixing that. So you had one data pool containing both the apps pool and general storage (not recommended) and you already wiped that pool? Did you recreate the pool using the same name?

I have a boot pool on my OS drive, and a data pool on my data drives for everything else. Is that really not recommended? It’s not normal? Anyway, yes.

Yes, already wiped, recreated with the same name.

I’ve already extracted my backed up app data to an ix-applications dataset I created.

$ df -h /mnt/.ix-apps 
Filesystem                  Size  Used Avail Use% Mounted on
boot-pool/ROOT/24.10.0/mnt  108G  128K  108G   1% /mnt

$ ll /mnt/.ix-apps
total 1
drwxr-xr-x 2 root 2 Oct 31 08:14 ./
drwxr-xr-x 4 root 4 Nov  3 23:08 ../

$ df -h /mnt/NVMes/ix-applications 
Filesystem             Size  Used Avail Use% Mounted on
NVMes/ix-applications   31T  194G   31T   1% /mnt/NVMes/ix-applications

What if I wiped my boot drive now, installed 24.04.2.2, recovered from a config backup I have for that version, and then did the upgrade to 24.10? Will 24.04.2.2 see the ix-applications dataset I created as if it was the one it had originally created, and then will the upgrade process see it and do the right thing?

Or will the config not apply correctly because it’s not the exact same pool? Same name, but different definition.

Or how about I just install 24.10 from scratch, which would presumably put .ix-apps on my data pool, and then do the documented (where?) process for recovering app data from a backup in 24.10? If it’s not documented, just move the contents of my ix-applications dataset to inside .ix-apps?

Ah, reading the apps docs:

ixVolumes are not recommended for permanent storage volumes, they are intended for use as rapid storage for a test deployment of the container. We recommend adding datasets and configuring the container storage volumes with the host path option.

I guess the recommendation is that .ix-apps is not supposed to be backed up or recovered, and that if you want to keep any app data, you create your own dataset for it?

So perhaps I should just be installing my apps from scratch and telling them to use new datasets for their various storage needs, then splitting up my backed up app data by app and moving it to the appropriate new datasets?