Asynchronous dedup

Problem/Justification
This is more about OpenZFS than about TrueNAS.

In todays implementation of dedup, rather large amounts of RAM is required. When data is stored for archive purposes, that RAM sits mostly idle waiting for another write where the system needs to check for duplicate blocks.

I suspect that most users of dedup have little need for dedup to happen synchronously in real time. Most would find it okay for the system to do dedup at similar intervals to scrubs. This due to the fact that dedup is invisible to the user (except for the increase in storage capacity when data is deduplicated).

Assuming dedup would be an optionally asynchronous process it could run whenever the system was lightly loaded. It could write all the block hashes to disk and only load a few into memory at a time when doing dedup thus the reducing the RAM consumption.

Impact
Asynchronous dedup would make dedup usable for everyone because the cost of using it would be negligeable. This is contrary to todays dedup functionality which seems to have limited use due to the cost.

Assuming users where given the choice between asynchronous and synchronous dedup there would be zero negative effects except for the development and testing time spend.

User Story
Users would select asynchronous dedup and dedup frequency as a property of a pool. At the specified frequency the system and when the load on the system is light, it would examine all new blocks since last run and calculate dedup hashes. These would be stored in system files on the pool. Then system would then run run through all hashes to find duplicate blocks. This could be done intelligently only requiring a few of them to be loaded into memory at any time.

There is a new ZFS feature likely coming out next year, “Fast DeDup”. iX actually participated in it’s development if I remember correctly.

Your point about selecting times for de-duplicating is interesting. In theory, a new tool could be made, that lists all the files in a Dataset that have not been de-duplicated. This would work for both “Classic DeDup” and “Fast DeDup”.

Using that new tool, this could happen:

  1. Turn off De-Dup on the affected Dataset
  2. Normal, un-De-duplicated writes will occur to that Dataset
  3. At some future time, (possibly regularly scheduled), re-enable De-Dup on the Dataset
  4. Using the new tool, scan the Dataset for un-De-duplicated files, and using a script to copy / rename them. Thus, triggering De-dup on them
  5. When time window closes, stop additional “copy / renames”.
  6. Jump to step 1.

Not sure if this is easily doable. But, meets your requirement and has a much higher chance of being added to OpenZFS than a completely automated version.

Hmm, my theoretical case of semi-manual scheduled De-Dup just took off and landed. It should no longer be a theory;

0 - Set new Dataset attribute with date and time you are disabling De-Dup.

4 - Use find on Dataset root to get a list of files created since disabling De-Dup, and pipe them to a script that copies them, and then renames them.

6 - Jump to step 0.

Now this does not take into consideration that all the files might not be De-Dupped during the time window, (like evenings only). But, that can be overcome with scripting.


Will I bother to put together a script to implement this?
Not very likely, (unless their is $1,000s in it for me :slight_smile:

Should it work?
Absolutely.

Would a tool in OpenZFS to list the un-De-Dupped files be better?
Yes, because their is a window when a file could be created, but before the date & time stamp is saved.

Not to mention that dedup works at block level, not at file level.
I suppose you could retrive a list of changed blocks by browsing through transaction groups. But it’s going to be a complete rewrite of ZFS dedup.