Local Backup to SATA or USB HDD

Feature Requests
,
1

Problem/Justification
Currently, there is no simple way of backing up to a local SATA or USB HDD.
It is quite cumbersome to use cold storage as a backup solution. With cold storage I mean drives which are stored offline somewhere off site.
Correct me if I am wrong or if I am missing something.

Let me start by describing how I used to do my backups:

  1. Have a couple of HDD’s stored off site on a shelf.
  2. Bring them to my NAS in more or less irregular intervals.
  3. Connect via SATA hot swap.
  4. Import the ZFS pool in the TrueNAS webui.
  5. Run a scrub if it has been a while since the last backup.
  6. Create manual snapshot with a specific naming scheme of all pools I want to back up.
  7. Manually run a replication task to copy the snapshots to the backup hdd.
  8. Manually delete all old snapshots, since there is no option to only keep a certain number of snaphots.
    There are only options to delete snapshots based on age. However, this is not an option for me, since I only create backups in irregular intervals.
  9. Then I export the pool via the webui, take out the hot swap sata drive and take it off site again.
  10. Rotate the HDD’s and take another one next time. There should be at least one stored off site at all times.

Since this was too much of a hassle for me, I decided to write a script that does all of that for me.
However, it is not 100% reliable and I would love to see such a feature implemented similarly in the webui.
I think it could be beneficial to a lot more people. All the threads I have read online about backing up to a local sata or USB HDD don’t give satisfactory and easy answers.

Here is a description of how my script works. The feature could be implemented similarly:

  1. Script gets started automatically on drive insertion by a udev rule.
  2. There is a config file with a list of source pools, destination pools and corresponding serials of the backup drive.
  3. If the serial number of the inserted drive matches one in the list, the script continues.
    (All of this would not be needed in the feature implementation, since a button in the webui to manually start the backup process would be just fine too.)
  4. The backup pool is imported
  5. The age of the latest snapshot is checked. If it is older than some threshold, a scrub is run.
  6. If the scrub succeeds, snapshots of the specified datasets are taken with a naming scheme.
  7. Snapshots are copied to the backup drive. (incremental, if not the first time. This is also why a scrub is needed)
  8. Old snapshots are deleted. Only a certain number of snapshots is kept at a time.
  9. Pool is exported
  10. Log is sent via email (this would not be needed in a real implementaion of this feature)

It would be great to have a UI element to configure, start and monitor such a backup process:
-Configure which datasets / pools should be copied to which backup HDD.
-Allow for the execution of custom scripts before and after the backup process? For example to create a truenas config backup or stop VM’s before backing up.
-Snaphot retention based on number of snapshots (not on age). This is required because of the irregular backup intervals. And in order to copy an incremental snapshot from source to destination, there needs to be the same base on both disks.
-Scrub and SMART test based on time since last backup to ensure integrity of backup pool before copying new snapshots.

Please tell me what you think about such an idea.

1 Like
2

I go through a very similar process.

id love to see your script if you are open to sharing to help float the boat!

3

Sure, I just published it on GitHub:

Let me know if you have any problems using it! It isn’t the prettiest, but it works for me for now.

If TrueNAS won’t implement such a feature natively and if there is demand for it, I would probably rewrite this script in python and try to use the TrueNAS API for snapshot creation and replication. I am currently using a script called zfs_autobackup for that.

1 Like
4

I have one lesson learned from supporting Multi-Report, the TrueNAS API can and will change over time. Some of the API features I use today are no longer listed in the online documentation and I know some of the API features are going away. Using the API is nice however my focus on the new Multi-Report v4.0 (due out next year) is to make the script stand on its own two feet, not be dependant on the TrueNAS API. This would allow other people who use Linux to use the script as well. Just something for you to consider, but the API might be the best way if it is only devoted to TrueNAS.

2 Likes
5

Thanks for the advice! And thanks for creating such an awesome script too!
I just started using Multi-Report along with scrutiny, after TrueNAS completely removed any GUI for scheduling and monitoring SMART tests in their new 25.10 Version.
I don’t get why they don’t put more emphasis on features like these. I mean they are crucial to keeping data safe (the core feature of TrueNAS).My feature request would probably only benefit home users, not their enterprise customers. So I kinda get why they don’t offer such a feature yet. But the whole SMART topic should be relatively important to everyone.

As a home user, I really do love TrueNAS and it does a lot of things right, but having no easy way to backup data to an external drive was almost a deal breaker for me. I just don’t want to backup to the cloud or have a second machine plugged in all the time, just for backups.

As for the TrueNAS API: I too noticed breaking changes over time. I already use it for pool import and export. According to the documentation, it should be possible to do all the snapshot related stuff with the API too, but I’d have to try it out to be sure.

1 Like
6

There is no simple way because this is a bad idea. Single HDDs can rot. USB hdds can fail in a weird and wonderful way, including corrupting the data while it’s being written — and yet your backup must be more reliable than the source. There is no conceivable scenario where this sins good idea.

Instead, setup replication to the remote system, or use one of the existing backup tools to create versioned backup on some cloud storage that provides data integrity guarantees.

7

I don’t see where this is a bad idea.
I don’t trust USB drives too, that’s why I only use SATA HDDs in a hot swap bay.
To detect data corruption, I run a scrub on the backup HDD before starting a backup. ZFS will store checksums along with the data, even on a single drive.

So data corruption will not be repaired, but definetly detected. So far, I have never had any ZFS errors on any of my backup drives. If I were to encounter any ZFS errors, I wouldn’t do an incremental backup. If the problem persists, I’d replace the disk. I also have 2 backup disks in rotation. One of them is kept off site at all times.

I am a home user and the only place to setup a secondary TrueNAS system would be at friends or families places or at my office. All in a radius of just a couple of kilometers from my home. I don’t want a lightning strike to fry both my main and backup nas, because of an overvoltage in the grid. That should be rather unlikely and I do have protection against that, but you never know. Cold storage (possibly in a faraday cage) seems more resilient against something like that. It’s also more resilient against cybersecurity threats, because a bad actor cannot reach a disk that is unplugged. And it is also the cheapest option in the long run.

3 Likes
8

I beg to differ. Any form of backup is better than no backup. Any additional backup is an improvement to the scheme.
An external backup on an ubiquitous USB drive which is used because it is convenient is better than a backup on an external SAS enclosure which gets neglected because it is too cumbersome to carry around.

copies=2 is you friend. Or, with some CLI shenanigans, a raidz1 on partitions.
Again, a backup with some rotten files is better than no backup at all.

These are not exclusive. Replication is typically to a live system. A USB drive is well-suited to make an offline backup—and possibly off-site as well.

6 Likes
9

Yes, but this is not the choice here. The choice here is jugling USB drives vs making a proper backup solution to live system (that is being scrubbed periodically), or to the offline facilities desfined for the purpose (like LTO tapes). HDD in a safe is not such facility.

So no, if jugling drives is implemented as a final measure, and not a stop-gap, until correct soluition is in place – it is actually worse than not having a backup, because it prevents one from actively working towards towards putting a correct solution in place.

Again, this is not what is being argued here.

Of course, until one day the disk won’t spin up, just because.

  1. Yes, backup shall be to a live system – which is actigvely maintained and monthly scrubbed.
  2. Offline backup if desired (why!?) shall be done to media designed for offline use. LTO tapes, or BlueRay M-disks come to mind. Not a hard drive, let alone bottom of the barrel one (because what else is there in USB enclosures?).
1 Like
10

Like I said, I only use SATA drives. The usage of USB drives is not mandatory in the final feature implementation, or it could come with a disclaimer.

I do my backups monthly, so I also scrub my backup drives monthly.

Why is an HDD not suited for that? It is not an archive. It is a backup. I do it about every month. HDDs will retain their data pretty well for up to around 5-10 years. So doing backups in the timeframe of months is well on the safe side.

Why isn’t this a correct solution?
For me it seems better to have 2 or more cold storage backup HDDs in rotation, that are resilient against cybersecurity threats or voltage spikes in the power grid.

Then someone who has 2 or more HDDs in rotation (like me) would still have the other one. The drive in your always running backup system can’t fail?

What you are referring to is called archiving. Not regular backups of changing data. M-Discs and Blu-Rays can only be written to once and tape also doesn’t allow for alot of write operations. Furthermore, a tape drive or Blu-Ray Burner + M-Discs are really expensive.

2 Likes
11

I guess I’m not doing a good job at trying to express this idea: by using single drive media as a backup you are betting that when you actually need one, it will spin up an will be viable. In essense, you are betting that your single drive is more reliable than your source system, and proper ZFS array.

I claim that this is not a good bet.

Single hard drives don’t provide any guarantess that the data written today will be retrieable tomorrow. You are going to be doing all that drive shuffiling only to discover potentially that when you array dies your drive also decided to quit that day, (or any other day since it was last written)

Ok, you maybe have 5 of those in rotation. But 5 drives in rotation provide significaltly lower overal reliability that even raidz1 of the same 5 disks.

The drive in runing system absolutely can fail, and you will take care of it right away. Your drive in a safe will be taken care at your next rotation, which mayb be weeks away. You are thus extending the window where you are “under-insured” so to speak.

It depends. You don’t have to do it yourself and buy all that hardware – you can pay, say, Amazon AWS for Glacier archive at $1/TB/month, with objeclock.

If your array dies – you’ll restore from there.

Again, only you know your usecase – but I personally have never encoutered a scenario where backup to sinlge drives in rotation is a good, let alone the best, approach. .

12

If your TrueNAS server IS THE BACKUP, then using multiple single disk pools for additional rotating cold backups is quite suitable. This is an easy way to clone your backups to get multiple copies.

I have 3 SATA disks that I rotate through, all are ZFS formatted and scrubbed before updating the backup. They are stored inside anti-static baggies, then inside a Seahorse case, (aka hard shell). In the past I was able to store them off site, but today I can’t easily do so. However, in the future I may do so again.

We are not talking Enterprise Data Center backups. My “day job” is Unix System Administrator and backups are critical to normal operation. We have fulls, incrementals, archivals for legal reasons, disaster recovery and of course off-site. While some archival and off-site backups are still done with tape, our disaster recovery and other off-site backups are done live, via network to remote disk.

It is a complicated world today. Some Enterprise Data Centers no longer use tape as a primary backup. Or are phasing tape out.

1 Like
13

That “more reliable” part is a requirement of your very own making.

Offline is the ultime protection againts intruders or cryptolocker, and therefore an essential part of a complete 3-2-1 scheme.

Often either a regular desktop drive (currently the case witch Seagate external drives being 24+ TB Barracuda HAMR) or a (white label variant of a) NAS drive (WD). Shucking such drives is an acknowledged way to build arrays at a discount.

And, contrary to SSDs, hard drives are suitable to retain data while offline. It is, after all, the same technology as in LTO tapes.

Let’s agree to disagree.

1 Like
14

Maybe I should have been more clear about my setup, which is similar to what a lot of other people seem to use at home.
I have 2x 16TB HDDs in a mirrored configuration. I only use two HDDs, because it is power efficient and it’s enough storage for me. My backup drives are also 16TB, so I can fit all of my data on a single backup HDD. If I had 5 in rotation, I would have 5 copies of my data. So all 5 have to fail to loose my data. In a raidz1, I could only loose 1 drive.

Below is list of threads that are all about this or a similar topic. There is a lot of demand for such a feature. (just look at the amount of comments, likes, views, and general interaction on all of these threads) One can find even more by searching for something along the lines of “TrueNAS backup to external drive”.
A lot of people also already use a similar approach for backup, that may be manual or error prone. So why not include such a feature natively to make the backup process people already use more safe? Each user has their own reason why backing up with this method may be the most suited in their situation. For example cost, or lack of bandwith to backup to a cloud provider, etc…
And all the different backup strategies are not exclusive. You can and should use different ones combined.
Instead of focussing on why this is a “bad idea” (I don’t see it as such in a lot of cases), I would much rather have people contribute to refining the feature request for different use cases. For some people encryption might be a priority, for others backing up VMs and containers safely by shutting them down first, etc…
Don’t get me wrong though. Every backup method has downsides and it is important to discuss them and be aware of them to be able to mitigate them. For now, I don’t see an argument that rules out this method of backing up completely.

Here a list of similar threads:
Although they don’t count as votes, it shows there is demand. Keep in mind that most users using a similar backup strategy will not even have posted about it. I myself used it for a long time and didn’t post about it until now. To some it may be even a complete deal breaker that TrueNAS does not support such a feature natively and easily. That should be one of the core features of a NAS.

https://forums.truenas.com/t/backups-to-disk-drives/8637
https://www.truenas.com/community/resources/how-to-backup-to-local-disks.26/
https://www.truenas.com/community/threads/backup-to-external-drive.97285/
https://forums.truenas.com/t/best-way-to-replicate-entire-pool-to-external-hard-drive/37396
https://forums.truenas.com/t/datensicherung-auf-eine-externe-hdd/38873
https://www.truenas.com/community/threads/how-to-make-use-of-external-drives-with-truenas-scale.100426/
https://forums.truenas.com/t/trying-to-backup-to-external-usb-drive-via-replication/5656
https://www.truenas.com/community/threads/zfs-send-to-external-backup-drive.17850/
https://www.truenas.com/community/threads/usb-drive-as-offline-backup.107555/
https://www.truenas.com/community/threads/backup-to-usb-to-take-home.105161/
https://www.truenas.com/community/threads/backup-to-external-drive.39399/
https://www.truenas.com/community/threads/backing-up-to-external-usb-from-freenas.77435/
https://www.truenas.com/community/threads/looking-for-best-practices-for-backup-to-external-usb-drive.106413/
https://www.truenas.com/community/threads/local-backup-on-external-hard-drive.107511/
https://www.truenas.com/community/threads/manually-on-demand-replicating-entire-main-pool-to-external-usb-drive-for-physical-offsite-emergency.79804/
https://www.truenas.com/community/threads/backing-up-a-data-set-to-an-external-hard-disk.114245/

https://www.reddit.com/r/truenas/comments/zpdetb/how_easy_is_it_to_backup_from_truenas_to_another/
https://www.reddit.com/r/truenas/comments/v5iee5/help_occasional_backup_to_external_drive/
https://www.reddit.com/r/truenas/comments/q5v1oj/can_you_backup_to_external_drive/
https://www.reddit.com/r/truenas/comments/149eglu/best_practices_for_backing_up_in_truenas/
https://www.reddit.com/r/truenas/comments/u7wqei/backup_truenas_data_to_usb_drive_for_offsite/
https://dekoning.work/blog/backup-to-external-hdd-truenas
https://forum.level1techs.com/t/choosing-an-external-hdd-for-backing-up-truenas-datasets/206952
1 Like
15

Fair point. But I still do not see a clear request where things could be simplified.
The process is:

  1. Plug in drive/enclosure.
  2. Import.
  3. Run manual replication.
  4. Optionally run SMART test and/or scrub.
  5. Export.

1 and 4 require user action. Defining the replication task for 3 is a one-off task; with a suitable policy for retention times, zettarepl should do the pruning of old snapshots on further backups… as long as there is still one common snapshot for incremental replication. Is there something worth automating to save a few clicks with 2 and 5?

16

For this method to work at an irregular interval, there would have to be an option to set the “Snapshot Lifetime” on the snapshot task based on number of snapshots and not based on time. Because taking backups with this method can happen at irregular intervals. If I were to set a snapshot lifetime of 4 weeks, but create the backup after 5, there would be no common snapshots anymore and incremental replication wouldn’t be possible.
The snapshot retention policy on the replication task should also include an option based on number of snapshots, not time.

So in general with snapshot creation and replication:
Add options for manual and irregular snapshot creation and replication. (based on number of snapshots to keep, not time)
It would also be nice to be able to create a snapshot task that does not run on a schedule. Because right now, you would have to always disable the snapshot task, to prevent it from accidentaly running based on a schedule.

Is something like that already possible?

You can look at my script which I posted earlier for a fully automated solution. I just plug in my drive. The script will scrub, backup, shutdown containers during snapshot creation, etc. automatically and I get an email when everything is done and I can unplug my drive. But that isn’t neccessary. I’d be happy if they would allow for a completely manual snapshot creation and replication not based on a schedule. Doing the other stuff like pool import and export or scrub in the gui is acceptable work.

17

All media, including the tapes used for decades for Enterprise backups suffer bit rot. Making a backup that you can verify (ZFS scrub) is better than no backup. Having a copy of that backup off-site is better than no backup. Understanding the limitations of any backup scenario is important to understand if you have the protection you need. While the 3-2-1 rule is best practice for Enterprise backups, non-Enterprise use cases may (will probably) have lesser requirements.

4 Likes
19

Hi all!

I don’t use the replication task as the Niob (OP) described but just do a simple SMB copy of the datasets to my single external NTFS-formatted HDD. Before the copy, I check and scrub the data quickly, format the external HDD, and, of course, don’t use the Windows Explorer, but “FastCopy” (software like TeraCopy). This is my off-site backup process every 6 months.

Nevertheless, the feature would be beneficial to me in the near future, as I’m moving away from the NTFS altogether. I still hang on to it as Windows is dominant in my region, and more importantly, in my family. Until I manage to get better internet and hardware to support remote replication, this will be my way of doing precious offline backups. Keep in mind that this feature would help low-budget users a lot, since we have very limited hardware.

1 Like
20

That looks awfully short, especially if backups are irregular.
I have tiered daily/weekly/monthly snapshots, and monthlies are kept for several years.

21

Perhaps, but I can justify it. (below).

Yes, but nothing is ever absolute, and for a vast number of use cases, online backup with properly configured permissions/locks provides the same practical level of protection as a tape in a bunker, at a significantly smaller cost (no juggling disks).

I strongly disagree with this (with the idea of shucking; not with its being an ack-ed way to get an illusion of cheap storage). Being acknowledged by masses does not mean it’s correct or worthwhile. The reason is twofold:

  • Drives are designed to work well in arrays. Several crappy drives in redundant configuration provide much better overall reliability than any single disk can ever dream to achieve, not to mention, making one would be so prohibitively expensive it would never be attempted, even if it was feasible. And yet, in this proverbial USB disk scenario it’s the single disk that must survive.
  • Drives that go to desktop drives are with a very few exceptions the crappiest of the crop – those that perhaps barely passed – much better to sell than at a discount than recycle. Binning exists.

As a result, using shucked or external drives as-is just compounds the problem.

If cost savings is a goal for a home user or small business – they ought to buy used drives: same price, but the left side of the bathtub curve completely avoided. But that’s a different topic.

And here I also disagree: this is technically correct, but an incomplete statement. Yes, both are magnetic recording, but there is a crucial difference: a disk has motors, actuators, processors, power management, thermals to manage, all that jazz. LTO tape is just tape in a box.

Magnetic media fails predictably: a few bad sectors here and there, nothing catastrophic. But everything else around it in a hard drive – not so much.

My, albeit personal, experience taught me not to expect a disk that was powered down to ever successfully spin up. I’d say 80/20 chances. Maybe this was a professional deformation fueled by WD Greens, and also maybe I’m exaggerating a bit – but if I do – it’s just a little.

While a disk spins and gets scrubbed – I trust it. A disk that is powered off – is a mystery. Maybe it’s already dead?

So no, it’s not a fair comparison.

– “No, you are wrong!” :slight_smile:

Seriously though, this is such a frequent topic that it’s worth it to get to the bottom of it and have something to link to in the future.

Absolutely, yes, I’m all for a data-driven approach. But as we see time and again, users focus on a scary but rare scenario (ransomware) but underestimate all the other more likely modes of failure. Because it feels right, does not mean it is.

This is where I also strongly disagree. I argue home backup must be more robust than enterprise backup, or at least just as robust. Because if my business burns in flames and all data is lost, I’ll just start another business. If I lose family photos from 5 generations, I’ll be quite upset to say the least. So you bet I’m implementing enterprise-level protection for my home data, times two, and a haphazard zoo of rotating single disks does not even make a list of potential candidates.

+1.

My backups go back decades, to the year I first learned about what it is. Main reason – you may not immediately notice that something went wrong (data rot, corruption, user error, etc.). Storage is cheap, and getting cheaper every year, so there is no reason to delete anything ever – especially deduplicated version history. It’s literally free.