I have two external USB drives that I rotate to an offsite location. Both formatted as zfs, and using replication to update the backups.
But over the last few days I’ve come across a couple of threads on external backups where ext4 is used/recommended.
Is there any downsides to zfs/upsides for ext4 for this use case that I miss?
ZFS can be scrubbed and is checksummed in ways that ext4 dreams of. Plus, it can be used as a sneakernet replication that also allows you to go forward and backward in time, which can be super useful re ransomware.
Ext4 will only be whatever the latest version was when the backup was taken. But it is more easily mountable on plain vanilla Linux systems than ZFS, though that is much less of a factor today.
All things being equal, I’d stick to ZFS for those reasons.
Thanks! Those are the benefits I have in mind as well, for both zfs and ext4. The ones for zfs is more important to me, so I’ll stay with that then. 
You can even mount the drives on a Mac if necessary.
In addition to the other comments, I would add these as benefits for ZFS:
copies=2 for some datasets for that extra reliabilityFor the last, Linux does support whole disk or partition encryption for any file system, including EXT4. However, TrueNAS does not use such, and may not even include those packages. Thus, it may not be available.
In regards to snapshots on the backup disks. A common method for backups is ZFS Send & Receive and replicates the source to the destination. However, when destination snapshots matching source are not needed, you can use something like RSync. Then when done, make a new ZFS snapshot on the destination to indicate backup complete.
Using RSync for ZFS backups may seem strange, but it has some benefits. You don’t need any ZFS snapshot history for ZFS Send incremental backups. And yet, we can still get incremental backups using RSync.
Even if a block goes bad on a single disk ZFS pool, you can still restore the file from other sources. It is far more complicated to figure out what is impacted on EXT2/3/4 when a bad block occurs. In the worse case, you have to re-create the file system and re-copy all the files. ZFS generally allows the recovery to be as simple as restoring the affected file from another source.
This is one of those half complete features that the devs didn’t bother finishing properly.
Give me one good reason I shouldn’t add you to my ignore list?
Yes and no.
What I meant was restoring the whole file. I’ve done it on my miniature media server’s media pool for media files that ended up with bad blocks.
How this would work for a single disk pool backup, is that you would just copy over the entire file. Hey, it is even possible that RSync might do it for you during a backup if you used the block checksum compare feature.
Of course, under normal conditions a backup script would not use the RSync block checksum compare, as that would end up reading all blocks from both source and destination. Great for really verifying your copy, not so great at time. It would be more time efficient to use the file time & date stamps for the normal RSync backups.
Is that what happens when one configure a Replication task in the GUI?
Yes.
The low level term is ZFS Send & Receive, which existed years before FreeNAS or TrueNAS and it’s GUI.
Thanks! Will dig into the config options!
ZFS is undoubtfuly better than ext4 due to all these features that are mentioned here. However, the full power of zfs features manifest in the set of multiple hard drives (the more drives, the better)! Only in this case, one can benefit from the most reliable file system.
Now, pay attention to the start of the post:
External USB drives are not best choice for NAS systems. However, they are for worst-case scenario if one needs to evacuate quickly and can only take most important things. In that case, one can not create/recreate pools. A single drive that can easily be plugged into another computer is feasible solution.
Therefore, use ZFS in the NAS, but for a single, standalone drive, majority of its advantages just can not manifest. In that case, ext4 is better choice.
Exactly.
Have both zfs and ext4 USB drives for extra backups.
Came to the conclusion that I dont need all the snapshots or features, so Im currently testing ext4 with rsync and saving deleted files to a directory.
If TN had execute-command-before it takes a snapshot, or replication, I would almost prefer zfs, but the control and visibility I get with rsync is what Im after.
Now, being able to encrypt ext4 with veracrypt or cryptsetup would be perfect, but TN lacks such support!
What brings you to TrueNAS or ZFS?
If you’re not using ZFS snapshots, replications, or native encryption, why not use Synology, Unraid, or QNAP?
I have been very happy with ZFS in USB drives for certain critical backups that are small in size. (My primary backup is another matter.)
I have found particular happiness with ZFS and USB by using NVMe solid state drives over USB Attached SCSI. Vendors don’t like talking about that, but you can get your own drive and enclosure. Your USB controller needs to support UAS too.
I don’t believe it’s published anywhere, but I know that on TrueNAS Community Edition (SCALE) 24.10.x.x supports it, as I’ve been using it there. I’ve confirmed that by looking at the kernel-loaded driver. Haven’t tried on Fangtooth yet.
As usual, go for quality in the hardware and ZFS will thank you.
Tip when using the Replication tasks to manage the replication: make sure you enable the task, run it, then disable it. Otherwise the middleware will complain when you try to export the pool. Also mind which options are selected on that screen. Some are destructive.
These are the fastest replications I have across more than 150+ datasets across multiple machines.
I do have local, local-secondary and off-site ZFS Replication already.
What I am after is being able to fully control a local, ENCRYPTED, accessible backup of my data.
Too bad TN does not support cryptsetup or veracrypt. That’s what I’m working on now, having a full rsync backup on a veracrypt volume. May take a VM, will see.
Yes, already have ZFS on USB, but at the end of the day, the worst case scenario is a Windows machine with an external USB dock.
Answer:
Ensuring I do have access to my data (ext4 or NTFS), regardless of which direction IX, Linux or the world takes.
Yes, USB DOES work reliably!
I do have several systems with USB drives with ZFS Pools and they have been working flawlessly for backups.
True, a few older USB drive models can get disconnects, but with the right hardware, it does work.
Once we get most equipment to USB 10G or USB 4:
I wanna see the faces of all THOSE ignorant people that continue to bet against USB!
it’s just another cable!
How you implement the protocol over it, that’s a whole different thing!
I’m confused. By default, any dataset that you replicate which is already encrypted will remain encrypted on the backup device. Thus avoiding the need for further encryption. That holds true whether the backup is a local USB drive or a remote server.
So why not encrypt the dataset / pool from the get-go and avoid these additional encryption options that simply layer another potential failure point on top of ZFS, which is perfectly capable of reliable encryption?
It’s not just the cable, or the protocol. Rather, it’s a long chain of conditions, all of which have to be strong enough to deal with whatever data you’re throwing at them. Ideally, they do so reliably. And that’s the rub. While the protocol and cable might be perfect, there are additional links in that chain like the hardware and the drivers that might make for a bad hair day.
Whether it’s a USB chip that may be designed perfectly but not cooled adequately by whoever implemented it. Or a driver that works 99.9999% of the time, except when it doesn’t. Edge cases can make for all sorts of exciting nights trying to troubleshoot the cause of a checksum error.
So it’s not that USB cannot be a reliable transfer medium already, it’s that until it reaches the same reliability as a LSI HBA flashed to the latest IT mode firmware, the folk here are reluctant to endorse it because of how many times they have seen folk coming here looking for help with USB issues.
Ditto the broad endorsement of Intel and Chelsio NICs. I have no doubt that there may be some very reliable Aquantia or RealTek implementations out there but we have seen more than a few cases re: those ethernet chipsets reporting problems that Intel / Chelsio users don’t.
This is key. When I was researching USB Attached SCSI I found more than a handful of anecdotal reports and some git issues for drivers flagging certain AMD chipsets that can sometimes be unreliable with the protocol.
If you never have any issues on one AMD machine, or an Intel machine, and then suddenly things aren’t working on some other machine using some other chipset, you’re starting to troubleshoot really far off the path from ZFS or it’s typical stack.
And if you’re using the USB BOT/Mass Storage protocols, you aren’t dealing with anything ZFS would natively understand and have to pass off a lot of heavy lifting to drivers and protocols that weren’t necessarily designed or tested for the kind of workloads that are really easy to run on ZFS.
This really deserves multiple stars. It’s the same reason ZFS does not want a RAID controller, cheap USB SATA multiplier, etc. to get between it and the raw SATA / PCIe / whatever access to a particular drive. It’s a similar problem to Promox VM instances of TN not being given passthrough access to the hardware - it might just work fine for a while but eventually it is likely to bite you in the butt.
