ZFS Snapshots and replication to backup nas

Getting my feet wet wrt. snapshots and replication to “Backup/Secondary NAS”

I made my first “manual” dataprotection → Replication task yesterday.
Replicating my linux server dataset on my primary nas to my secondary nas.
2.8TB taking approx 8hr. The snapshot was named manual-%…

Now i would like to “automate” my snapshot and replication.

Background:
Due to energy costs i’ll use the NAS’es as pure backup boxes, turning them on once a week (sunday around dinner time).
Primary NAS will rsync pull data from select sources, and save them in my Rust-Pool1/Backup/ dataset.
It’s a “Home setup” , so for now we’re talking about 2…5 servers, each have their own Backup/ dataset.

I did watch the STUX Tiered Snapshot YT video, but i have a different “setup”, where minutes/hour/daily will not make sense.

Goal:
To make weekly snapshots of my server datasets, and xfer “select” server DS snapshots to my Backup-NAS (Backup have approx half the space of primary).
Initial snapshot retention “two month / 4 weekly”

Plan:

1: Rsync pull linux (per) server data via “own CLI rsync job” - GUI seems too restrictive
2: Create per server dataset snapshots - named --%… - This naming would hopefully make pruning not clash with other snapshots ???
3: Replicate select server datasets to the backup nas.

Q1:
I expect step 2 & 3 is a per server (dataset) job.
Would it be smarter to make a “replicate to backup” sub-dataset in Backup, and put the servers to replicate to the backup nas there. That would mean less flexibility (ie. different retention) , but just two jobs. Remember it’s a home solution so i expect a limited amount of servers.

Q2:
When do i Scrub ?
Before or after the rsync pull from the servers ?

Q3:
Do i have to adjust the scrub cronjob times ?
I suppose 00 00 * * 7 - Mean “Sat–> Sun” Transition , the NAS isn’t powered on there.

Q4:
Would it be more feasible to have the NAS turned on ie. Sat @23:30 , in order to run the “internal default jobs”
Are there any “hidden internal jobs” that would be affected by a “just weekly schedule”

Q5:
The snapshots i (replication task) have taken for now, are apparently set to never expire.
If i make a new automated snapshot with retention and replicate it to the backup nas (same DS name). Can i then remove the “Initial no expire” snapshot, and the snapshot data will still be there (it was an 8h xfer) ?
I would expect that the snapshot data is bound to the dataset , and not to the snapshot name … ???

Q6:
By prefixing the snapshots with a “function oriented name”, i’m avoiding the “zettarepl” purge to match other prefixes … Correct ?

Q7:
On one of my replication tasks i now see a possibility to chose a periodic snapshot task , i suppose as “trigger”

I don’t think i saw it , when initially making the repl. task …
What’s the pre-req for that line showing up ??

TIA
Bingo

My entry in the advice competition… (its a lot of question in one post)

Q1: Depends how you restore
Q2: After
Q3: Scrub every 3 months?
Q4: probably… tell us if there’s an issue
Q5: yes, you can remove
Q6: makes sense, but test
Q7: Yes if its relevant… see text in docs below.

Select the snapshot schedule for this replication task from the dropdown list. Select from previously configured periodic snapshot tasks. This replication task must have the same Recursive and Exclude Child Dataset values as the selected periodic snapshot task. Selecting a periodic snapshot schedule removes the Schedule field.

2 Likes

My ZFS snapshot replication task is running now (primary to backup). Seems that my backup nas cpu is “busy” …

I followed “Chris video” here

And set the backup nas replication dataset to gzip-9 :thinking:
Like Chris did …

My cpu utilization went from 20…22% (lz4) to 85…100% (gzip-9).

The little EPYC 3151 (MJ11-EC1 board) is fighting hard too keep up.

And is actually doing a nice job, considering i have not enabled HT’ cores.
And my new $6 silent “China cpu fan” is too … Combined with the Node-304 coolers. CPU Max 53C.

I didn’t think that gzip-9 would have such an impact on the CPU utilization.

My rationale for trying gzip-9 was that i “just have” 6x4TB in Raidz2 on the backup, making disk space less than half of the primary.
And that i have a lot of “source files” like linux kernels etc, that prob. would pack well.

But i think i’ll switch back to lz4 again.

Fun learning experience :upside_down_face:

Edit:
Now my Backup replication pool has been set to both lz4 & gzip-9 , but that wouldn’t affect any access or restore … Correct ?
I mean i assume that files (or is it blocks ?) created will somehow be tagged with the compression used.

1 Like

You could consider zstd compression.

Perhaps level 3 or 4.

Better than Lz4, but only a bit slower to compress.

Sorry. Could not find the nice comparo i saw.

1 Like

This one ?

Or maybe

1 Like

For inline compression with ZFS (version 2.2+), the early abort feature for ZSTD obsoletes some of these benchmarks.

See also this thread from the legacy forums.

1 Like

Yes. This one :slight_smile:

1 Like

So basically it’s kind of a “can i compress at all” scheme.

If i set my desired compression to “LevelX” , it will do :


if LZ4 compress >= 12.5% 
    compress w. "LevelX".
else
    if ZSTD-1 compress >= 12.5% 
        compress w. "LevelX".
    else
       Save uncompressed

That would be nice for my huge “already compressed” .tar.gz archives that i xfer.

But could (in rare cases) end up testing/doing a triple compression … LZ4 → ZSTD-1 → “LevelX”.

Q1:
When/where is this compression test made ?
I suppose it’s on some kind of zfs “block level” … Correct ?
Answered here : Upon writing a new record

Q2:
If the Dataset is set to LZ4 would it still test for >= 12.5% , and save uncompressed if less ?

Q3:
Is the above algo’s, already “default active” in Scale (DF+) ?
I suppose so

Thank you for this. :+1:

Edit:
In the above “Thread” it clearly shows :
There’s no such thing as a free “lunch” …
As i read it even ZSTD-1 would reduce throughput quite a lot on both compress & decompress … Minimum a factor 3.

My reasoning:
If you’re “long term archiving” and can/will afford the reduced throughput, do use efficient compression. Else just get a bigger disk.

On my backup nas i might “get away unpunished” with a high compression rate. But that’s just because it currently has 1Gb netcards, that’s the limiting factor for the compressed throughput.
If the box had 2.5Gb cards i might have impacted throughput with a high compression rate. That’s assuming my spinning rust Z2 can do 2.5Gb sustained.

LZ4 and ZSTD-1 are so fast that the overhead is practically moot for decent CPUs.

The result is you end up using your desired ZSTD compression level on a block that is clearly compressible. This is a payoff that is worth it if you want to squeeze out more from your storage. You set your dataset to use ZSTD, after all!

This is why ZSTD should be avoided for datasets that will hold incompressible data, yet you should still leave it set to LZ4.


Correct. This is all done in RAM with your CPU. The storage drives are not factored into it.

LZ4 (and ZSTD-1) zip by so fast, that one (or both) make sense as a heuristic to see if it’s worth compressing the block with ZSTD-X.


LZ4 has its own early abort feature, but the end result is essentially the same. Incompressible blocks are saved uncompressed.


Factor of 3? ZSTD-1 is incredibly fast, and most levels are still quite fast for decompression.

When pulling from spinning drives, it’s possible that reading less data (to be decompressed in RAM/CPU) is faster than reading more uncompressed data from the drives themselves.

Even though the main purpose of using ZSTD compression is to save space, it can even theoretically improve performance for datasets on spinning HDDs if the underlying blocks are highly compressible.

I use ZSTD-9 myself for certain datasets, but level 7 or 5 might be more ideal. I only use LZ4 for everything else.

For “jail” and “container” filesystems, this is especially useful for keeping them lean.

New FreeBSD 13.4 jail with LZ4
Size: 963 MiB
Ratio: 1.98X

New FreeBSD 13.4 jail with ZSTD-9
Size: 786 MiB
Ratio: 2.62X


What does the network card have to do with compression performance?

Do you mean that smaller data takes less time to travel over the network? That would only apply to raw replications. Clients are still receiving the uncompressed data over the network, such as with SMB or NFS.


TL;DR:

Use LZ4 for almost everything, and reserve ZSTD-X where it clearly will save space.

I meant that xferring 1Gb/s from my primary (lz4) to the secondary (gzip-9) load’s the 4-core EPYC ~85% , but xfer rate is still “Max”/unaffected.
So “performance wise” I could keep gzip-9 on the DS, as it seems like the EPYC can do 1Gb/s gzip-9.

If both ends had 2.5Gb cards the EPYC would IMHO not have been able to do full 2.5Gb/s xfer rate, as the CPU was already loaded 85% with 1Gb/s.

Why use GZIP when ZSTD is faster, uses less CPU, and achieves just as good or better compression ratios?

1 Like

I will not use gzip-9, but am considering ZSTD-3…5
The reason i used gzip-9 in the first place was that Chris did that in the video, and i was just “following the leader” … Until i saw the CPU impact.