I need help with how to backup TrueNAS ZFS-pool data to multiple USB drives

TrueNAS General
, ,
1

What could be the best/wise way to backup a ZFS pool’s data to multiple USB-drives?

Automatically continuous/reliable backups:

  • Full + incremental backup e.g. once a week (USB-drives are kept connected, but need to be “turned off”/ spin down when not in use)

  • Quartely Full backup to (drives that have their own power turned off after a backup so TrueNAS would see them as disconnected)
    – Maybe these could even be combined to avoid unnecessary double backups?

  • Data does not fit to a single USB drive (needs a smart way to split data to multiple drives).
    – Needs ability to alert when the space running low in the USB-drives that are used in the backup.

  • Need to support ability to add more USB-drives when data does not fit in the ones that are connected or used in the “backup cycle”.

  • Many USB-drives are DM-SMR (with no trim support) (Fast in sequential backup, not so ok in ZFS, or has this changed?)

  • USB-drives are diffrent size so maintaining a separate “backup zfs-pool” from them is not practical? Does exsample TrueNAS Scale support Btrfs (that support/handle diffrent size drives ok, but is it’s raid5 stable enough?) or someting else that this could be wise and supported way?

  • The USB-drives need to be able to be “spindown/sleep” when backup job is not running (to increase life and minimize energy consumption).

  • Would need to support easy removing the USB-drives (for offline backups).
    – Ideally when a disconnected USB-drive that is used for continous backups is connected back, it would automaticly work for backup. Without having to manually setting the drive back in use.
    – Would be nice to have automation support (smart power socket that could be controlled by the system to bring power back to USB-drive when its time to backup).

  • Needs ability to set how many backup copies are kept and delete the old ones automaticly.

  • Timing of backup jobs (ideally diffrent timings for difrent directories/files from the pool).

  • Needs to support keeping original file modifications / creation times and that they can be restored. ACL are plus, but not mandatory.

  • Compression support (must be able to restore individual files without having to extract all data)
    – mount compressed filesystem and/or view compressed files “like Windows can use zip files as a folder”) support.
    – Ideally compression/etc should help to use big (256MB) cluster or even files? in the filesystem of the USB-drive that suits SMR drives well so they wont slow down or waste space?

  • Open source

  • Ideally the solution would work in the old 12 series TrueNAS Core (help in backup/upgrade), latest TrueNAS Core and Scale.

  • GUI prefered, easy to edit conf files or script files+cron?
    –Ideally it can be made/instructed easy enough for even novice to operate safely (reduses mistakes even for pro’s in the long run :slight_smile:

If not possible to implement in TrueNAS alone?
– VM in the same host could be an option. Connect the USB-drives to this VM and use it to do the backup process if it can talk to TrueNAS ok?), recomendations and how?

Maybe I am over thinking this? :slight_smile: If the list is impossble to accomplish, how would you do it?
– Also what important did I miss?

ps. English is not my native language, sorry about the grammar.

2

I don’t know of a solution like that. Better to either:

Build a second NAS (on another site)

Use Storj.

3

SMR drives, by design, are NOT fast for large sequential operations such as your backup scenario.

1 Like
4

Some thoughts:

  • Using a stripe of disks could be an option. Of course, loss of a single disk would be fatal for the backup set. Loss of a block would only affect a single file, (though not if in redundant metadata).
  • You really don’t want DM-SMR drives.
  • While another file system on the backup disk than ZFS might be desired, TrueNAS’ official support for such other file systems is non-existent.
  • Using ZFS does allow scrubbing of the backup media to detect bit-rot and allows you to erase the failed file and write a new copy.
  • Multiple USB drives, in different enclosures can be problematic. Both from a reliability side and potentially serial number side, (which can prevent them from working at the same time). See:
    Why you should avoid USB attached drives for data pool disks

Some time ago, I thought about making a backup tool that sorted the files for backup based on size.

Thus, I could use multiple media, (back then, tapes & DVD-Rs), for my backups. It would backup the largest file first, on the first media and continue with the next largest files that fit until that media was full. Then start a new media with the largest file still needing to be backed up.

The benefit for that method was each media was independent of the others. Loss of one would not impact any file on another. (Unless a single file had to span multiple media… which I did not have thoughts on how to handle.)

This avoids the stripe of media problem.

5

Thanks for your input and sorry not best word choise from me.
–What I tryed to mean, that I have got good speeds of the SMR drives when backuping to them (140-80MB/Sec continius writing) when they are not a part of a ZFS/raid etc.

They are specially fast if the drive is new (Up to 190 MBps) or was been “refreshed” time to time.

I made also some testing in Windows for the drives,
There is huge difference when “Enable write caching on the device” is enabled (normally disabled for USB drives, but enabled for SATA drives).

When cache is enabled the drives starts to work like a “normal” drive, speeds go up form 28-65MB/sec to 77-103MB/sec when copying files and the drive heads stop jumping around.
–I wonder is it same for TrueNAS?

Exsample TeraCopy Buffer size has a big difference too (Test done copying files over network from SMB share (Data in TrueNAS ZFS pool) to the SMR-drive when it has been alredy filled with data before, so it has been slowed down)

When not using cache
Buffer size, avg speed, max speed

  • 32KB, avg 27MB/sec, max 28MB/sec (quite stable, but keeps drive busy long after write stops)
  • 64KB, avg 3,5MB/sec, max 3,6MB/sec
  • 128KB, avg 6MB/sec, max 7MB/sec
  • 256KB, avg 11MB/sec, max 13MB/sec
  • 512KB, avg 20MB/sec, max 27MB/sec
  • 1MB, avg 27MB/sec, max 28MB/sec (default)
  • 2MB, avg 42MB/sec, max 45MB/sec
  • 4MB, avg 51MB/sec, max 57MB/sec (fastest and wavy)
  • 16MB, avg 42MB/sec, max 49MB/sec (very jumppy)
  • 32MB, avg 16MB/Sec, max 26MB/sec (very jumppy)
  • 256MB, avg 11MB/Sec, max 14MB/sec (on/off moments)

Cache enabled
Buffer size, avg speed, max speed

  • 1MB, avg 68MB/sec, max 77MB/sec (default)
  • 2MB, avg 75MB/sec, max 87MB/sec
  • 4MB, avg 76MB/sec, max 86MB/sec
  • 16MB, avg 50MB/sec, max 65MB/sec (Some spikes up and down in speed)
  • 32MB, avg 38MB/Sec, max 49MB/sec
  • 256MB, avg 45MB/Sec, max 103MB/sec (drive heads starts to jumping around, writing stops time to time)

Quick tests! Not the whole story but shows that it might need quite good settings to get good speed out of the SMR-drives…

I have also directly used SMR-drives in the Truenas when they are NTFS formated, work ok when coying files manually, but NTFS not ideal. Not done so accurate test there, yet.

I wonder would 256MB cluster size work good for SMR drives, so the whole zone would be writed once and prevent fragmantation?

Have you (or anyone else) tryed SMRFFS? Does it work?
–Also it mentions that “SMR drives have a specific idiosyncrasy: (a) drive managed drives prefer non-interrupted sequential writes through a zone”…

6

Just clicking on the link provides a rather strong clue:

This repository was archived by the owner on Mar 25, 2021. It is now read-only.

1 Like
7

There is a difference between ZFS’ copy on write, (COW), write behavior and most other file systems.

ZFS will perform any writes, (data & metadata), to unused blocks on the storage. As a final step write transaction, the critical metadata is updated to free up the old blocks of data, (if any), and metadata pointing to the old blocks of data.

This Copy On Write behavior is especially noticeable on DM-SMR disks because they prefer long streams of writes. Which ZFS tries to do, but won’t if it’s free space is fragmented. Thus, ZFS on DM-SMR disks is slower than other file systems.

The other file systems that use COW, are:

  • BTRFS, Linux only, (though it is available on ReactOS, a MS-Windows work alike…)
  • BCacheFS, Linux only and barely usable compared to OpenZFS or BTRFS
  • Hammer1/2, DragonFly BSD only

One last note. ZFS was never intended to be the absolute fastest file system on the planet. (I know, a bit of an exaggeration.) Data integrity and online actions were the priority, (like zero boot time file system checks before use). So other file systems that work well with DM-SMR disks is irrelevant when discussing ZFS and it’s COW.

8

For me, a reliable and stable system is better than the fastest.

SMR-drives are a problem, but for backup they usually are the best value drives and because I alredy have them I amd stuck to use them and also want the solution to be future proof that it can work with SMR-drives as well.

Sofar looks like solution could be a use traditional filesystem that suits SMR drives and top of it use file compression to to split the data to multiple drives.
– Exsample If using RAR, it could even bring recovery support that could handle data loss errors and it also support checksum to check data validation.
– RAR is not open source :confused: (I do Wonder if any open source formats allow recovery support?

But would prefer more complete solution than just using a file compression…

Doen any of you have experience using any of these backup software to backup data from TrueNAS ZFS pool?

Future wishes

  • ZFS to get support for diffrent size drives in a raidz, raidz2 setup and hopefully SMR support too.

  • Wait for btrfs to get its raid5/6 to a stable point and hopefully SMR support as well.

9

If you care about the data, need a rock-solid, reliable approach, then do not do it with USB. The best solution is a direct SATA / NVME connection, i.e. bare-metal. There are HBAs with external connectors that allow you to attach external arrays with little fuss but which also ensure that ZFS has direct access to the drives themselves.

Can it be done with a TrueNAS? Sure, there are external arrays available that allow a eSATA / SFF connection from a TrueNAS and then it would be up to you to set alerts and expand the capacity of the external array as necessary.

Here is an DIY example with 25 drives. but smaller, complete solutions exist also. Thing is, if the external drives have to be transportable, you may need to transition to larger drives in the external array or start dedicating arrays to specific datasets.

Just no. I know some people here have made it work, but you’re setting yourself up for failure. The cost is almost the same, stick to CMR-only drives. DM-SMR is nothing but trouble, especially if a lot of data has to be moved, during scrubs, and worst of all during resilvers.

Good luck with that. If you find such a solution, please let us know. To my knowledge, no such solution exists? I am happy to be wrong.

What you may be able to do is to create a number of separate replication tasks that you will likely have to manually re-enable every time you attach an array.

You may be able to “marry” an array to a specific replication task with a clever script that detects a specific array and then enables the replication task of choice. That’s above my ability level, however.

Snapshot policies can do that automatically for you.

Ditto re: snapshots.

Automatic.

Snapshots work pretty much the same throughout the TrueNAS universe. I doubt newer version pools can be imported into older version systems however, as feature flags and ZFS options keep expanding. (i.e. TrueNAS Fangtooth pool → CORE import is likely a no-go, it will work the other way around, AFAIK).

Good luck, but as others have pointed out, you may be better off with a cloud-storage approach.

10

Support for different sized disks in a RAID-Zx and using all available disk space is likely never going to happen. Just too many problems.

That said, you can mix different disk sizes in a RAID-Zx vDev and that vDev will use the minimum common size available for all disks. And if the smallest disk(s) are replaced with larger, then the vDev will grow to the next smallest disk(s).

As for SMR support in ZFS, their are 3 types of SMR:

  • DM-SMR - Drive Managed
  • HA-SMR - Host aware
  • HM-SMR - Host managed

Most consumer are in the DM-SMR category. The Host Managed are probably not available to the general public as they require serious software drivers to make them work.

But, Host Aware, now that could be in ZFS’ future.

TrueNAS will not likely support BTRFS in the foreseeable future. Perhaps in a VM or some container…

One frustrating thing about BTRFS is that development is stagnant. OpenZFS gets lots of updates per month, though maybe only a few real features per year. But BTRFS has been waiting on reliable & stable support for RAID-5/6 for maybe 8 to 10 years. Just not enough developers care about RAID-5/6 for BTRFS.

Worse, some developers have moved on to BCacheFS as the “new” future file system for Linux. (Unless you use Red Hat Enterprise Linux, then it is Stratis.)

1 Like
11

So it looks like “pool like, filesystems like ZFS or btrfs” are “no go” for backups.

So what is a solution to backup to USB-drives?

Also generally reply to solution ideas provided for me, thanks from them, but in the case I am working on.

  • Cloud backup is not a working option
    – Security and regulation problems in the cloud rule them out for many data needed to be backuped.
    – “too much” data, saving and keeping it in cloud costly way too much and accessing in is too slow over the internet and pulling data out ads to even more costs.

  • Another ZFS box.
    – There is alredy another server with a ZFS-pool that data is synced, but this is not a backup solution.

Need a backup solution. One that can take backup to drives that can be disconnect from the system (Tape and also USB drives are good for that) and put them in a safe.
–Security and regulation need safe offline backups.

Also disconnect drives are usually the only ones where data is saved in a worst case ransomware attack so they are very much needed.

Openmediavault seems to have USB-backup solutions, like
https://www.openmediavault.org/?p=936
– The new plugin ‘USB Backup’ can be used to automatically synchronise a shared folder to an external USB storage device when the it is plugged in.
– Can this be done in TrueNAS as well?

12

As long as your dataset can fit on one drive/tape, there’s no issue with ZFS.

The issue is with multiple indivual drives.
I suppose you could write (or rather pipe) a zfs send stream to a file and then tar the file to multiple drives but that may get complicated quickly.

13

I did a 6 HDD RaidZ2 under USB, when I started.
(however, it was not under TrueNAS (Actually it was called FreeNAS that time), but under an Ubuntu server and ZFS for Linux.)
It was so instable that at least 3-4 times a week one of my HDDs were dropping from the array. I had to readd them, and run a scrub job.
It is 16 TB ( 6x4TB WD Blue drives). It took forever to resilver them.
I don`t think, it worth the effort one must put into supporting this system.

14

Part of the instability could have been one of these:

  • Too fast head parking, then taking too much time to un-park for any access
  • Too long TLER, Time Limited Error Recovery, (Seagate calls it something else). Basically desktop disks take extreme measures to read a failing sector, instead of letting RAID redundancy take care of the failing block.
  • Some desktop / laptop drives use SMR technology, which causes time delays even possibly to the point that ZFS considers the drive failed.
  • If all 6 HDD are on the same USB host port, it acts as a funnel, thus during scrubs or resilvers it can take what seems forever.
  • If USB is used without UASP on both sides, the old USB block protocol is slower, thus adding yet more delays.

Some of those factors also apply to SATA / SAS attached disks.

15

I agree for backups that exceed a single backup disk, ZFS is not quite suitable.

It would be really nice to have a backup solution that did what I described here;

I wish I had the energy to write up such a backup program. But, even though I was a computer programmer for close to 2 decades, that was more than 25 years ago. And to do this right, would take me a major effort.

16

The solution is: don’t do it. Backup to a second system running ZFS. If you want reliable automatic backups, that is.

17

It was split by 2 USB 3.0 ports (it was relatively old MSi MoBo) by 2x4 port passive USB 3.0 HUBs.
My drives were CMR drves, so that is not an issue now.
Finally I bought an IBM 1015 HBA, reflashed it to IT mode, and also a Fantec 8 bay 2U server case…
And since that time it is rock solid and stable.

1 Like
18

I agree.
If the USB drives are actual HDDs (as mine were), you can remove them from the cases and install to a PC case. (they are normal sATA drives insive the housing.)
The only thing, you should avoid is SMR drives.
(you can check it by google-ing their part numbers, after removed from theUSB cases.)
If the drives are thumb drives, I recommend to use them for any other prupose.

19

There isn’t an ideal solution to this scenario

However there are other options

  • Do the drives need to be new? - For Cold storage 2nd User Enterprise is an option (CMR)
  • Do the drives need to be usb or just removable? If the latter get something like a DS380 that has 8 removable bays

If the folders/directories are under ~14TiB then an option is

robocopy [folder] drive /mir

and have TWO drives [folder]A & [folder]B, running the copy either

TN to A & TN to B - at the same time - heavy network traffic

or

TN to A & A to B - with a 2 Sec delay (files will be cached on PC)

depending on your network speed

I use Ex datacentre drives and have A & B though your case may vary

20

I have used the Oyen Digital Mobius 5 series in the past and it usually worked well as DAS.

Not as a ZFS target but rather as a means to back up from the NAS via a Mac in a Mac-native format to the Mobius. This makes it easier to read / use the content remotely since my Mac has no ability yet to read ZFS pools.

I would NOT use this method in a unsupervised backup scenario, and I would NOT rely on it as a primary backup. Too much can go wrong.

But the Mobius 5 is relatively compact, contains a fan and a built-in PSU, offers SATA / USB 3 and contains a rudimentary RAID controller. For the money, it is superior to many other solutions out there.