TrueNas Scale needs a first party application backup and restore system

Apps and Virtualization
,
1

The whole purpose of a NAS to keep data secure and resilient to data loss. Currently there is no way to backup and restore applications.

Thoughts:

  • First party implementation by ix systems
  • Simple
  • Application independent
  • Backup and restore applications
    • Restoring/Backup application configuration and data set
    • Restoring application data
    • Restore applications even on a new Truenas Scale system from backup
  • End-user should be able to browse application data through a file explore

Application Backups/Restore should be plug-and-play. No other NAS software has implemented such a system that is open source. Third party isn’t good enough due to complexity.

React to the post with “thumbs up” or “down”

3 Likes
2

That’s apart of the scope of work we are doing in the new Apps implementation. Idea is to keep nearly all application config on-data pool, and if we have specific things land in our DB, provide an export / import functionality as well. We want to make it super simple to migrate an Apps workload to a new system and bring it back with minimal fuss.

6 Likes
3

That’s exciting to hear!

Once the team is further along with the design phase can you break that down from a technical perspective and the end-user experience for community feedback? Perhaps that could be here or a blog post?

4

Yep, I’ll be keeping folks updated as things land in the nightly images in the run-up to BETA. Once at BETA I will expect full docs to make an appearance as well.

3 Likes
5

One thing that would be important is to be able to back up and restore individual apps.

1 Like
6

That’s not always trivial. Backing up a Nextcloud container for example, correctly, involves setting maintenance mode, running a mysqldump or pgdump, etc. You wouldn’t get a good backup just doing a server level backup of some sort. It’s not the only app with special backup procedures either. And with going to docker, it can get even worse as someone may run a separate postgres database “server” container for many purposes, making it even more difficult. I suppose they can make a best effort but it won’t be a substitute for me for a container based backup of the special apps.

1 Like
7

I could see that being an issue for live backups. If the app is properly shut down before back up and as long as separate containers are accounted for that shouldn’t be an issue.

My concern depending on how the apps are containerized if they (multiple independent apps) share infrastructure/backends like databases. That’s more efficient but I’m not seeing an easy technical route for backup and restore.

8

Yeah it is complex. However nextcloud is a great example that frequently breaks highlighting my concern.

Imagine nextcloud breaks in a way that’s not easily recoverable. It would be awful to have to roll back all applications (configuration and data) to an earlier state when it’s only one that needs to be restored. To me that puts a lot of data at risk. For example X application commits data which will be lost. That data might be important and unknown at the time of restore by the admin. This risk grows with the number of applications on the server.

9

This comes down to

  1. snapshotting data + config for each “app”, ie a compose folder + data dataset
  2. taking care backup/restore on a app compose basis, ie where necessary, there might need to be a backup/restore service implemented in the compose stack.

Not sure if 2) would be undertaken. Its something I do organize for some of my compose apps.

I definitely organise 1) to happen, and in many cases this is sufficient…

10

Exactly what is needed for some apps. My nextcloud backup is taken within my container, it does the documented nextcloud backup steps and I never lose anything. Same for mariadb container and a few more.

11

@kris any further thoughts about the concerns and solutions here in relation to ongoing work with the backup/restore system?

I ask as sometimes it’s much harder to change design not if considered early in development.

12

Current design plan is two parts:

For custom installed compose apps, we expect to store the original YAML (in the case where a user supplied their own) on the data pool, as well as the host mounts and other persistent data. This can be used to migrate / deploy an existing app to a new system, assuming you move the persistent App data over via ZFS Replication or some other means.

For Apps installed via the UI, those are a bit different since we have a lot of extra meta-data to consider from the UI questions. Those will be stored in the TrueNAS configuration database and have an option to export / import those config files. (JSON or YAML not yet sure). With the configuration in the DB, it means losing the boot pool you can bring your apps configuration back if you restore your config file, and manually export them if needed in the case of migrating to a new system.

13

“Always keep a recent copy of the configuration file” is a fair ‘Best Practice’ requirement.

But, for the sake of the many, many casual users who will not hear of best practices until after disaster has occurred, it would be good if apps could be restored from their own pool alone, even if the boot pool is lost.

14

Another idea would be to routinely duplicate the configuration file on a pool, perhaps in the system dataset.

15

They’ve been doing that for ten years now. But they still refuse to expose it through the UI.

16

Yea, thats on my list of things to do. Backup configuration to pool periodically. On fresh install when pool is imported, prompt user if they want to restore the last saved configuration.

2 Likes
17

That’s been in the product for roughly ten years–every day at 0345, a backup is taken to /var/db/system/configs-(longhexnumber)/(TrueNAS release)/, which is in the .system dataset.

That’s the piece that’s missing despite open tickets and discussion on the old forum. In fact, it seems that when the pool is imported, those old configs are erased.

18

Yep, and we use that occasionally manually to this day :slight_smile:

The auto discovery / import piece is what is needed primarily here. Not that its crazy difficult to add, just need to make it a reality soon.

19

This is where I’m a little confused. When I hear persistent app data and app configuration pools that holds config files it seems monolithic.

Monolithic being defined as restoring and backing up all applications via their respective persistent data/configuration pools. Therefore you can only back up by replication or snapshot those entire pools.

If that is true the end user can’t restore an individual app without it affecting the rest of the applications?

This is a serious concern for me to begin to invest in truenas scale apps. A few posts above I talked about a scenario involving nextcloud describing my concerns with such a monolithic approach to back up and restore. This approach can lead to data loss that scales with the number of apps and users during restore.

If we’re able to manually export and import a specific apps configuration and data set why not make that the basis for backup and restore.

@kris I’m coming at this from an end user perspective but could you help clarify my concerns?

Edit: If entire pools can be backed up but restoring can happen per app’s configuration/persistent data my primary concern would be adequately addressed. I’m not sure that’s being proposed in the future backup restore system.

20

I am also looking for something like this. Coming from a full truecharts implementation where snapshots and database dumps could be scheduled with heavy script. I feel like I’m living on the edge now.
I will explore a bit more, but so far I am not seeing an equivalent for ix apps.