Splitting the data in the pool to directory segments (so data can be split manually to fit multiple difrent size drives) and copy data from them separately to diffrent usb drives.
– When using NTFS, it works great even with SMR drives and allows using compression at the same time.
— When doing the copy with Teracopy I can also use checksum counting to test that the copy is ok and it can be verified.
– Drives can be disconnect when not in used and stored to safety.
→ But lot of short backs, I would really want to move on to better way of backuping.
If using Cobian Backup / Cobian Reflector I can also take 7zip compressed Full+Incremental backups and automate what to copy, where to copy and how many copies to keep, issue commads after the copy (sleep drives), etc.
– Its free, but not open source
— Heading to eol if it does not find programmers to continue the project
→ Maybe one could be interest to port it to Linux/Freebsd and make it part of Truenas?
I am having ideas of a script…
Where is a list of the drives used for backup,their “64-bit World Wide ID” and what backup group/plan the drives is parting.
–This allows to detect what to do to to all the backup drives when they are connected or needed to be mounted etc.
– In the first iteration go for manual before planning anyting automatic to do when usb drives is connected.
The use compression to split the data to multiple drives, (like we used compression in the past to split data to multiple floppy disks Compression also adds CRC so it can be verified.
I am not shure jet is it any good idea to try to use someting like
mergerfs to “combine” the disk.
– users are requesting mergerfs be made part of TrueNAS
“spanning drives” method to just add drives together, they would still nee to be able to work by them self.
– Any raid-0 type is out of the questions.
Also thinking that maybe file list + checksum would be good to be take and used to as a log/db where to compare changes so incremental backups could be easily taken whiteout needing to read all the backup disk involved.
This would really help that you would not need to connect all the usb drives to just backup the changes to new usb drive etc.
Still in the alpha stage of just thinking about it and wondering still that there has to be way someone has alredy been thinking about this in the past and solved it alredy way better than I can even imagine…
I few years back, still in the FreeNAS days, I had a perl program that would:
take a ZFS snapshot
prompt the user to mount a filesystem at the target location, for which I used a single disk. IIRC I used, at different times, ext4, zfs, and zfs with copies=2. These were SATA spinning-rust disks either plugged into eSATA ports, or using hotswap 3.5" bays
use tar(1) to create a multi-volume archive, optionally with fast compression. This would write the tarball in multiple parts to the target filesystem
after each part, check to see how much disk space is remaining; if it was less than the “part size”, then it would send an email to the operator that another disk was needed, and leave the tar pipeline paused
the operator would be expected unmount the disk, mount a new one in its place, and then run a command that tells the script it can continue
the script then continues to write out parts to the next disk, rinse and repeat until done
Note that there is no requirement that the removable disks are the same size, or necessarily of the same filesystem type; only that there is enough space on the set of removable disks to hold the entire tarball.
If this sounds like the traditional way to use a standalone tape drive, it’s not a coincidence.
The down side is that it was only good for a full backup and was time consuming (both writing the data and doing the disk swaps); I was using it only for DR copies. Trying to restore a subset of files using this method would be quite painful.
I’ve also used Bacula to do traditional file-based backups for FreeNAS systems at a Former Employer, again where the amount of data was much greater than the size of a single drive or tape. Luckily, that was using a tape library.
I’m now back in the situation where, like the OP, this is now a problem for me again so I may have to dust things off and figure out the best current approach. I’m not currently in a position where a tape library is in the budget, so it’ll probably be back to the disk swapping approach. I suspect that cloud storage for backups would be too expensive/slow if a full recovery is needed; pulling 80TB from AWS would be upwards of USD 7500 in network fees alone.
Maybe I’ll look at adapting the removable disk strategy to be a virtual library for bacula, possibly using consolidated backups if I’m crossing a slower network link. More thought is required.
I have a large volume shared as SMB on truenas SCALE and mounted on my Linux Desktop. I also have a hotpluggable SAS disk adapter plugged into USB as well on the Desktop. To backup the SMB share on truenas I run the following
sudo tar -v --create --tape-length=2790G --file=/dev/sda /media/shared/Media/
/dev/sda is the hot pluggable 3TB (see tape-length) disk and /media/shared is the SMB share.
When the first disk is full, it will prompt you to swap the disk. Once the next disk is ready, press Enter to continue at the prompt.
Note that this uses the entire target disk as a block device. It won’t create any partition nor file system on the target disk, so don’t partition or format the backed up disks.
To RESTORE/extract the files, connect the first backup disk and use
sudo tar xvf /dev/sda -C /
It will prompt you to swap the disk similar to the back up process.
This may not be a perfect solution for your use case but maybe you can modify it to make it work.
Been backing up to USB drives for years. One can automate, so, a script can detect what is attached and zfs import them. Then run the backup, then zfs export them. You can even use multiple sets, and based on the name attached the script can do the appropriate thing. I store my backup sets at a bank. It’s not my only backup method, it’s one of them.
