Immutable backups with ZFS send (replication tasks)

TrueNAS General
1

Hi everyone :slightly_smiling_face:

Today I was finally looking into replication task to backup to an offsite TrueNAS system to protect against ransomware.

I created a replication task, that (I think) uses root on the local TrueNAS and the user “backup” on the remote system. Owner of the remote datatset is the user backup.
Snapshot retention policy is set to none.

Side note: none is a little bit misleading IMHO, the better word for it would be “keep forever”

That is all fine and well, but assuming the worst case happens and somebody gains root access to my local TrueNAS (which hopefully should not be easy, thanks to 2FA) would it not be easy for the attacker to just change the retention policy?

That is why I am wondering, how you guys do immutable backups?
Make Snapshots of the snapshots, on the remote?
Use pull, instead of push?

2

Offline backup?

1 Like
3

Your backups are more at risk of physical theft or destruction, rather than someone gaining access to your local network and TrueNAS root user account.

It makes sense to use a high quality, steel reinforced door and a lock that is hard to pick. After a certain point, you might want to consider someone breaking the back window, rather than continue to focus on the security features of your house’s front door.

1 Like
4

Along with human error. Sad but true. Checkout zpool checkpoints they could potentially help in all scenarios.

5

So I guess nobody thinks it is worth to go the extra mile and use immutable backups?

I was thinking about using Backblaze with lifecycle rules or rsync and WORM.
Or using pull instead of push, but with a user that has only read permissions on the source?

How would that work?

6

@etorix shared the easiest way to do this. Keep your backups unplugged and offline. Store them somewhere safe when they are not currently being used to receive a replication from the main pool.

If it’s not feasible, then yes, using “pull” instead of “push” for your replications can help.

Don’t forget that physical access trumps any kind of sophisticated software security.

2 Likes
7

I use many different backup methods, after all, a single copy might be corrupted, who knows. I don’t put all my eggs into one backup. I don’t want to lose my important data, ever. Thus, I use different tools for it too. And I do offsite too to protect against disasters like house destroyed (mine almost was recently, house got hit by lightning but fortunately not super powerful as it can be and I lost 20 some devices, took 6 weeks to fix all but one of them), theft, etc.

One of my backups is very simple. I replicate up to external USB, disconnect, and it goes to the bank safe deposit box every so often. I use kopia. I use truenas repllication to a vps. I use rsync for a few things even.

For me, I’d never trust a single backup or method. Maybe zfs has a random rare error and when I go to restore from a replicated copy, bad things happen. Or, maybe my backup job does something stupid, a glitch caused by it or me, and I lose my backup and the my system dies. So many possibilities.

2 Likes
8

cheers for all your great inputs! I think I will not worry too much about a TrueNAS takeover and either add some rsync to Synology with versioning, or Backblaze to have another backup method.

9

It’s a great question. Here are some of my thoughts:

  1. PUSH vs PULL - PULL can give you another level of security IF your PULL system is running less services than your primary. The idea here is that the more services you run on your TrueNAS the more potential attack vectors. If you only need your PULL service to act as a backup target then you may only need the SSH service enabled.
  2. SSH - Having said that SSH is probably the most likely way you are going to be hit so it makes sense to secure this as best as you can. Use key exchange and disable password authentication for the entire service. Keep your network secure and if possible use network segmentation to limit certain services to certain addresses/ranges for example VLANs and network ACLs.
  3. As mentioned use 2FA for access to the WebUI and also the above suggested network segmentation to limit which addresses can access it. Disable the default truenas_admin user and create your own bespoke admin user.
  4. Create a specific and dedicated replication user and limit their access to only what is required.
  5. Don’t forget IPMI (if applicable). If I had a pound every time I came across a server that had been locked down hard but the IPMI address was exposed to all and sundry with default login credentials.
  6. Disable default console access without a password as others have mentioned physical access to your system is just as likely if not more so regardless of where it’s kept.
  7. Explore the idea of creating periodic zpool checkpoints to protect against accidental deletion and malicious attacks.
  8. Consider the use of ZFS holds for snapshots ensuring they cant be easily deleted.
1 Like
10

My line of thinking was something else.
If I use push, I need to have rw permissions on both systems.
If I use pull, the remote system only has r permissions to the local system.

  • if my remote TrueNAS gets taken over, it can’t destroy the data on local
  • if my local TrueNAS gets taken over, it can’t initiate a connection to the remote TrueNAS from a firewall standpoint. So while it might pull the encrypted data, it won’t destroy itself.

Sure. Password auth I always disable for all SSH stuff. And also from a firewall perspective, the only device allowed to even connect over Port22 is the static IPv6 of local to the static IPv6 of remote.

Own VLAN.

7 and 8 sound very interesting, I will take a look into that.

11

Yep although I don’t believe this is currently supported in TN and would need extra manual configuration however i like your thinking.

1 Like
12

You can use zfs allow to assign a given user ‘rights’ to send and receive a given dataset. That would mean no sudo rights needed for that user and the given user would not have the rights to delete datasets on your pool.

zfs allow repuser send,receive pool/dataset

If you want the user to be able to create and destroy snapshots (if you are trying to keep the two systems in sync) then you can also add the snapshot element.

zfs allow repuser send,receive,snapshot pool/dataset

Might be worth a look if you really want to tighten security.

1 Like
13

Times have changed and you should look to have many layers, as is the updated 3-2-1-1-0 backup rule that has started floating around

The next evolution: 3-2-1-1-0 backup strategy

The 3-2-1 backup strategy remains valuable, but modern threats like ransomware require additional protection. One backup copy must be isolated, either physically or virtually, to safeguard against attacks that target all data, including backups. To further enhance security and reliability, many organizations have recently transitioned from the 3-2-1 strategy to the upgraded 3-2-1-1-0, which introduces the following updates to the key principles:

3 = Maintain Three Copies of Your Data

Always keep at least three copies of your data—one primary and two backups.

2 = Use Two Separate Storage Media

Store backups on two different types of media, such as a local server or hard drive and a cloud storage platform. This diversifies your protection and reduces risks.

1 = Keep One Copy Offsite
At least one backup should be stored in a separate location, such as a remote office or in the cloud, to protect against local disasters.

1 = Store One Copy Offline or Immutably
To defend against ransomware and other cyberattacks, one backup should be stored offline (air-gapped) or set to be immutable. An offline backup is disconnected from your network, while immutability ensures that data cannot be altered or deleted, adding an extra layer of protection.

0 = Ensure Zero Backup Errors
Regularly verify that backups are completed without errors. This includes daily monitoring and periodic restore tests to confirm that data can be fully recovered. Zero-error backups are essential for a reliable ransomware recovery plan.

The 3-2-1-1-0 backup strategy goes beyond traditional backup methods by adding crucial layers of protection against both physical and digital threats. By incorporating offsite, offline, or immutable backups and ensuring error-free recoverability, this approach equips businesses with the resilience needed to combat modern cyber threats like ransomware while safeguarding critical data.

1 Like