Isn't Referenced supposed to be identical in each replica of a given snapshot. It isn't!?!?

TrueNAS General
1

Here is the tl;dr of this post now that I figured it out:

  1. Referenced will be smaller on the clone replicas than the original. That’s normal. All the clones will likely have the same size.
  2. You can always safely delete any snapshot (that the system will allow you since some have clones), subject to #4.
  3. Full replication will find the newest snapshot in common between the systems.
  4. However, if the newest snapshot in COMMON is not COMPLETE, it will let you know and STOP rather than trying to find an earlier complete snapshot. It’s just trying to get your attention that that shouldn’t have happened (and was likely because you did some manual snapshot deletion).

----- END OF SUMMARY-------
Here’s the original post.
My full replica backups to a second truenas system were going along just fine until I decided to do a little housecleaning.

So I decided to move minio dataset to a new location. So I did a clone and promote and then i deleted the old dataset

But when my daily replication job started, I gave me this error which is super confusing to me because that particular snapshot was replicated identically on the two backup systems I have.

Here’s the output. See my additional comments below.


Critical
Replication "main -> backup" failed: Last full ZFS replication failed to 
transfer all the children of the snapshot main@auto-2024-09-24_21-10. 
The snapshot main/apps/minio@auto-2024-09-24_21-10 was not transferred. 

Please run `zfs destroy -r main@auto-2024-09-24_21-10` 
on the target system and run replication again..
2024-09-25 21:10:06 (America/Los_Angeles)
Dismiss

And here’s the screen shot showing the original is there and both the replicas of the snap are there (with a slightly smaller Referenced size):

image
image799×710 41.4 KB

The top image is my running system. main is the original, SSD is the backup.

I have an offsite clone of main, that’s the second image.

The issue is the error message… It’s baffling because nothing is missing for minio… all three snapshots on 9/24 are there just as expected.

So why did it think something was missing and not tranferred? there is no evidence of that so I’m baffled.

OK, i see the problem. I think I wasn’t careful when I hand deleted some snapshots to try to roll things back when I turned off zero snapshots.

I started these full replicas on 9/21, but there isn’t a minio replica on 9/21 so clearly something is missing.

The error was right that things were missing it’s just the stuff before that was mising I belive due to user error.

I’m going to start from scratch and monitor.

For now, I’m good.

2

My understanding is that Referenced is FIXED at the time of the snapshot. So all the replicas of a given snapshot have to have exactly the same Referenced value. The Used value can go up or down after the snapshot is taken and can be different on each system depending on the pruning of older snapshots for example.

So I’ve been taking regular snapshots and doing replication to my SSD and look, the Referenced values are NOT the same.

There seems to be something very wrong here. Either I’m not understanding this or there is a serious bug.

Check this out. Right after I snap each dataset, I immediately replicate the snap to my SSD. As you can see, the Referenced numbers for the original and the replica of the snapshot DO NOT MATCH. How can the replica be of a different size than the original?!?

image
image1127×538 16.6 KB

I asked ChatGPT and here’s the explanation:
The difference in the Referenced values between your snapshots on main/user and their replicas on SSD/user can happen due to several factors related to how ZFS replication works. Here are some potential reasons:

1. Compression Settings

  • If the compression settings differ between the main/user dataset and the SSD/user replica, the amount of space referenced on each might differ. Compression can cause the same data to take up more or less space on different datasets.
    • Check if compression is enabled on the SSD/user dataset and ensure it matches the settings on main/user.

2. Deduplication

  • If deduplication is enabled on either the source (main/user) or the replica (SSD/user), it can affect the Referenced size. Deduplication can make it so that certain blocks are not stored more than once, thus altering the amount of space a snapshot references on disk.

3. Block Size Differences

  • ZFS uses different block sizes for different datasets depending on the dataset’s settings and the workload. If the recordsize on main/user and SSD/user differs, this could also cause the Referenced values to differ for the same snapshot.

4. Sparse File Handling

  • If there are sparse files (files that do not actually occupy their full logical size on disk), the way ZFS handles sparse files on the source dataset might differ from the replica, leading to different Referenced values.

5. Metadata Overhead

  • Sometimes metadata overhead can be different between datasets, especially if one dataset has different properties (like quotas, reservation, or extra features) compared to the other. This overhead can cause slight differences in space usage between the source and the replica.

6. In-flight Changes or Scrubbing

  • If the replication occurred at different times or if a scrub was running on either pool, the amount of referenced data can vary slightly due to temporary changes in disk usage, like freeing of blocks.

7. Fragmentation

  • If one pool (such as SSD) is more fragmented than the other, ZFS might store data less efficiently, which could cause the Referenced size to differ.

What to Check:

  1. Compression: Run zfs get compression main/user SSD/user to see if compression settings differ.
  2. Deduplication: Run zfs get dedup main/user SSD/user to ensure deduplication is either enabled or disabled consistently across both datasets.
  3. Recordsize: Run zfs get recordsize main/user SSD/user to verify if the block sizes differ.
  4. Sparse Files: If you suspect sparse files, ensure the replica is handling them the same way as the original dataset.
  5. Scrub or Fragmentation: Check if there were ongoing scrubs or look at pool fragmentation levels.

Conclusion:

Differences in Referenced values for replicated snapshots typically arise from variations in dataset settings (like compression or block sizes) or ZFS features (like deduplication). Double-checking these properties should help you pinpoint the cause of the difference.

And as you can see, the properties were identical for the properties it suggested:

root@truenas[/mnt]# zfs get compression main/user SSD/user
NAME       PROPERTY     VALUE           SOURCE
SSD/user   compression  lz4             inherited from SSD
main/user  compression  lz4             inherited from main
root@truenas[/mnt]# zfs get dedup main/user SSD/user
NAME       PROPERTY  VALUE          SOURCE
SSD/user   dedup     off            default
main/user  dedup     off            default
root@truenas[/mnt]#  zfs get recordsize main/user SSD/user
NAME       PROPERTY    VALUE    SOURCE
SSD/user   recordsize  128K     default
main/user  recordsize  128K     default
root@truenas[/mnt]#

But I noticed that Referenced sizes were identical on the REPLICAs of the snapshot (i have an offsite backup). So SSD=offsite backup Referenced and the sizes were smaller than the original. So there must be something in the ZFS send/receive process that compresses the size.