Replication adding excess blocks?

TrueNAS General
, , , ,
1

I’ve spent some time reading through as many threads on the topic as I can, and this just does not seem like the appropriate behavior… Is this a bug??

Setup:

  • [system] tardinas (TrueNAS-SCALE-22.12.4.2)
    • [pool] np
      • [zvol] auth-swap
  • [system] entropic (Dragonfish-24.04.1.1)
    • [pool] sp
      • [dataset] replicas
        • [dataset] tardinas-np
          • [zvol] auth-swap

Replication:

  • tardinas/np replicates to entropic/sp/replicas/tardinas-np

Here’s how tardinas starts out…

root@tardinas[~]# zfs list -ro space -t all np/auth-swap
NAME                                                         AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
np/auth-swap                                                 1.96T  4.84G     48.6M    742M          4.06G         0B
np/auth-swap@auto-2024-04-29_19-51                               -     0B         -       -              -          -
np/auth-swap@auto-2024-04-30_00-00                               -     0B         -       -              -          -
np/auth-swap@auto-2024-05-01_00-00                               -   156K         -       -              -          -
np/auth-swap@auto-2024-05-02_00-00                               -     0B         -       -              -          -
np/auth-swap@auto-2024-05-03_00-00                               -     0B         -       -              -          -
np/auth-swap@auto-2024-05-04_00-00                               -     0B         -       -              -          -
np/auth-swap@np-manual-2024-05-04_09-57-XXXXXXXXXXXXXXXXXXX      -     0B         -       -              -          -
np/auth-swap@auto-2024-05-05_00-00                               -   144K         -       -              -          -
np/auth-swap@auto-2024-05-06_00-00                               -     0B         -       -              -          -
np/auth-swap@auto-2024-05-06_19-37                               -     0B         -       -              -          -
np/auth-swap@auto-2024-05-06_19-38                               -     0B         -       -              -          -
np/auth-swap@auto-2024-05-06_19-44                               -  3.60M         -       -              -          -
np/auth-swap@auto-2024-07-17_03-01                               -     0B         -       -              -          -
np/auth-swap@auto-2024-07-18_03-01                               -     0B         -       -              -          -
np/auth-swap@auto-2024-07-19_03-01                               -     0B         -       -              -          -
np/auth-swap@auto-2024-07-20_03-01                               -     0B         -       -              -          -
np/auth-swap@auto-2024-07-21_03-01                               -     0B         -       -              -          -
np/auth-swap@auto-2024-07-22_03-01                               -     0B         -       -              -          -
np/auth-swap@auto-2024-07-23_03-01                               -     0B         -       -              -          -
np/auth-swap@auto-2024-07-24_03-01                               -     0B         -       -              -          -
np/auth-swap@auto-2024-07-30_03-01                               -     0B         -       -              -          -
np/auth-swap@auto-2024-07-31_03-00                               -     0B         -       -              -          -
np/auth-swap@auto-2024-08-01_03-00                               -     0B         -       -              -          -

Ok, so just shy of 5G collectively over everything: zvol + its snapshots

After replication has completed, on entropic I have:

root@entropic[~]# zfs list -ro space -t all sp/replicas/tardinas-np/auth-swap
NAME                                                     AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
sp/replicas/tardinas-np/auth-swap                        2.14T  6.11G      135M   1.92G          4.06G         0B
sp/replicas/tardinas-np/auth-swap@auto-2024-04-29_19-51      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-04-30_00-00      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-01_00-00      -   418K         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-02_00-00      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-03_00-00      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-04_00-00      -   395K         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-05_00-00      -   395K         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-06_00-00      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-06_19-37      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-06_19-38      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-06_19-44      -   349K         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-07_03-00      -   360K         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-05-08_03-00      -  1.75M         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-07-17_03-01      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-07-18_03-01      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-07-19_03-01      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-07-20_03-01      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-07-21_03-01      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-07-22_03-01      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-07-23_03-01      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-07-24_03-01      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-07-30_03-01      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-07-31_03-00      -     0B         -       -              -          -
sp/replicas/tardinas-np/auth-swap@auto-2024-08-01_03-00      -     0B         -       -              -          -

Over 6G?? Where did this extra 1G come from?? Shouldn’t these be the same blocks coming over?

Here’s the properties on the two:

tardinas…

root@tardinas[~]# zfs get all np/auth-swap
NAME          PROPERTY                 VALUE                    SOURCE
np/auth-swap  type                     volume                   -
np/auth-swap  creation                 Thu Nov 23  9:13 2023    -
np/auth-swap  used                     4.84G                    -
np/auth-swap  available                1.96T                    -
np/auth-swap  referenced               742M                     -
np/auth-swap  compressratio            2.42x                    -
np/auth-swap  reservation              none                     default
np/auth-swap  volsize                  4.00G                    local
np/auth-swap  volblocksize             16K                      -
np/auth-swap  checksum                 on                       default
np/auth-swap  compression              lz4                      inherited from np
np/auth-swap  readonly                 off                      default
np/auth-swap  createtxg                34495880                 -
np/auth-swap  copies                   1                        default
np/auth-swap  refreservation           4.06G                    local
np/auth-swap  guid                     14003731452527524641     -
np/auth-swap  primarycache             all                      default
np/auth-swap  secondarycache           all                      default
np/auth-swap  usedbysnapshots          48.6M                    -
np/auth-swap  usedbydataset            742M                     -
np/auth-swap  usedbychildren           0B                       -
np/auth-swap  usedbyrefreservation     4.06G                    -
np/auth-swap  logbias                  latency                  default
np/auth-swap  objsetid                 90444                    -
np/auth-swap  dedup                    off                      default
np/auth-swap  mlslabel                 none                     default
np/auth-swap  sync                     standard                 default
np/auth-swap  refcompressratio         2.44x                    -
np/auth-swap  written                  532K                     -
np/auth-swap  logicalused              1.86G                    -
np/auth-swap  logicalreferenced        1.76G                    -
np/auth-swap  volmode                  default                  default
np/auth-swap  snapshot_limit           none                     default
np/auth-swap  snapshot_count           none                     default
np/auth-swap  snapdev                  hidden                   default
np/auth-swap  context                  none                     default
np/auth-swap  fscontext                none                     default
np/auth-swap  defcontext               none                     default
np/auth-swap  rootcontext              none                     default
np/auth-swap  redundant_metadata       all                      default
np/auth-swap  encryption               off                      default
np/auth-swap  keylocation              none                     default
np/auth-swap  keyformat                none                     default
np/auth-swap  pbkdf2iters              0                        default
np/auth-swap  snapshots_changed        Thu Aug  1  3:00:00 2024  -
np/auth-swap  org.freenas:description  auth vm swap             local
np/auth-swap  org.truenas:managedby    10.0.0.129               local

entropic…

root@entropic[~]# zfs get all sp/replicas/tardinas-np/auth-swap       
NAME                               PROPERTY                 VALUE                     SOURCE
sp/replicas/tardinas-np/auth-swap  type                     volume                    -
sp/replicas/tardinas-np/auth-swap  creation                 Thu Aug  1 21:25 2024     -
sp/replicas/tardinas-np/auth-swap  used                     6.11G                     -
sp/replicas/tardinas-np/auth-swap  available                2.13T                     -
sp/replicas/tardinas-np/auth-swap  referenced               1.92G                     -
sp/replicas/tardinas-np/auth-swap  compressratio            2.43x                     -
sp/replicas/tardinas-np/auth-swap  reservation              none                      default
sp/replicas/tardinas-np/auth-swap  volsize                  4.00G                     local
sp/replicas/tardinas-np/auth-swap  volblocksize             16K                       default
sp/replicas/tardinas-np/auth-swap  checksum                 on                        default
sp/replicas/tardinas-np/auth-swap  compression              lz4                       inherited from sp/replicas/tardinas-np
sp/replicas/tardinas-np/auth-swap  readonly                 on                        inherited from sp/replicas/tardinas-np
sp/replicas/tardinas-np/auth-swap  createtxg                1164948                   -
sp/replicas/tardinas-np/auth-swap  copies                   1                         inherited from sp/replicas
sp/replicas/tardinas-np/auth-swap  refreservation           4.06G                     received
sp/replicas/tardinas-np/auth-swap  guid                     15863898417419792455      -
sp/replicas/tardinas-np/auth-swap  primarycache             all                       default
sp/replicas/tardinas-np/auth-swap  secondarycache           all                       default
sp/replicas/tardinas-np/auth-swap  usedbysnapshots          131M                      -
sp/replicas/tardinas-np/auth-swap  usedbydataset            1.92G                     -
sp/replicas/tardinas-np/auth-swap  usedbychildren           0B                        -
sp/replicas/tardinas-np/auth-swap  usedbyrefreservation     4.06G                     -
sp/replicas/tardinas-np/auth-swap  logbias                  latency                   default
sp/replicas/tardinas-np/auth-swap  objsetid                 29387                     -
sp/replicas/tardinas-np/auth-swap  dedup                    off                       default
sp/replicas/tardinas-np/auth-swap  mlslabel                 none                      default
sp/replicas/tardinas-np/auth-swap  sync                     standard                  default
sp/replicas/tardinas-np/auth-swap  refcompressratio         2.44x                     -
sp/replicas/tardinas-np/auth-swap  written                  0                         -
sp/replicas/tardinas-np/auth-swap  logicalused              1.86G                     -
sp/replicas/tardinas-np/auth-swap  logicalreferenced        1.76G                     -
sp/replicas/tardinas-np/auth-swap  volmode                  default                   default
sp/replicas/tardinas-np/auth-swap  snapshot_limit           none                      default
sp/replicas/tardinas-np/auth-swap  snapshot_count           none                      default
sp/replicas/tardinas-np/auth-swap  snapdev                  hidden                    default
sp/replicas/tardinas-np/auth-swap  context                  none                      default
sp/replicas/tardinas-np/auth-swap  fscontext                none                      default
sp/replicas/tardinas-np/auth-swap  defcontext               none                      default
sp/replicas/tardinas-np/auth-swap  rootcontext              none                      default
sp/replicas/tardinas-np/auth-swap  redundant_metadata       all                       default
sp/replicas/tardinas-np/auth-swap  encryption               off                       default
sp/replicas/tardinas-np/auth-swap  keylocation              none                      default
sp/replicas/tardinas-np/auth-swap  keyformat                none                      default
sp/replicas/tardinas-np/auth-swap  pbkdf2iters              0                         default
sp/replicas/tardinas-np/auth-swap  snapshots_changed        Thu Aug  1 22:11:29 2024  -
sp/replicas/tardinas-np/auth-swap  prefetch                 all                       default
sp/replicas/tardinas-np/auth-swap  volthreading             on                        default
sp/replicas/tardinas-np/auth-swap  org.freenas:description  auth vm swap              received
sp/replicas/tardinas-np/auth-swap  org.truenas:managedby    10.0.0.129                received

(same, but thrown into meld)

image
image1743×874 207 KB

And then here’s as many oddball things I can think of that may potentially be at play in any capacity

  • tardinas had a manual snapshot that was omitted from the replication task
  • tardinas had some automatic snapshots marked with indefinite retention policies (no idea why I did this, will probably go back and remove that policy/remove these snapshots)
  • The replication task was initially set up to push from tardinas to entropic, but later was changed to pull from entropic (reflected in snapshots by 03-01 vs 03-00 (03:00 vs 03:01))
    • Note in switching this I got some warning along the lines that it would need to blow away the existing and redo them (I didn’t read this very carefully… figured, sure fine do what you need to do).
      • I did try a quick test with just the zvol in push vs pull, and both appear to yield similar results
      • I also tried blowing away the zvol & having it pull from scratch… same results
    • The original/initial setup likely was setup in a previous version of TrueNAS, but I’m not sure how far back

What really bothers me is that before I went messing with this, I had replication of pools np → sp, and then sp → dp (not shown here, and not hooked up to the newly replicated sp yet). I’m looking at dp, and that shows reasonable sizes – what I would expect from np. There’s other datasets/zvols not shown here as well, and this is consistent across the board. np & dp both sitting at 1.5 used, but sp is sitting at 1.68. Anyone have any idea what’s eating up this 180G??

2

Different pool geometries? For the very same data:
mirror < raidz1 < raidz2 < raidz3

3

Hmm… Interesting. That would certainly make a lot of sense. I did some quick tests replicating back to tardinas (to a different test zvol). Ok… So It’s at least consistent – the new and original amount used on tardinas match up. So I’m definitely out of bug territory then, but that leaves me in “I’m misunderstanding something” territory then. Ha… Ok, so now I’m just trying to figure out where my misunderstanding is…

I thought the redundancy was already accounted for in these metrics…

tardinas-np entropic-sp Comments
Min Disk Size (TiB) 1.82 0.45
Pool Geometry 2 x MIRROR (2 wide) 1 x RAIDZ3 (12 wide)
Raw Disk Space (TiB) 7.28 5.47 Disks added up in JBOD like fashion
Usable Capacity (TiB) 3.51 3.82 What TrueNAS advertises as “Usable Capacity” for the Pool (“Used” + “Available”)

Shouldn’t Used & Avail be incremented/decremented in terms of what the user sees as the file/vol size of these? This feels like doubly shorting the “Available” space if redundancy is factored into the max and then also factored in as files are put in. What am I missing??

4

Honestly, that’s not uncommon with ZFS…
With a CoW filsesystem, “space”, “free”, “available”, “used” become complicated issues.
To write a file, mirrors simply allocate the appropriate number of records. Raidz arrays allocate a stripe of records that is wide enough to meet the redundancy requirements; this generally requires extra padding and can waste a lot of space for small files, but the impact cannot be accounted for advance as it depends on what’s written.

5

Ok… I was going to set up a test but I’m so confused by the results I don’t even remember what the original test was going to be…

I just replicated auth-swap to a new zvol on entropic, and then blew away all of the snapshots to not have those muddy the waters with the reported sizes…
I then did the same on tardinas: replicate, blow away all snapshots…

BOTH of these now report 4.06G for USED. What?? Didn’t we just get done coming to the conclusion that these will show up differently because of underlying geometry??