What are fast dedup memory requirements?
Short answer: Less horrible.
Long answer: People are unlikely to be comfortable with committing to a clear answer before the feature is complete (half or so of the PRs are yet to be integrated), as changes are possible as the design and implementation get reviewed.
The next PR in line for Fast Dedup seems to have just been approved, so the code is moving closer to completion.
3 Likes
What he said ^^
The memory requirements will be a tad less, but more importantly, you can now cap the memory and dedup tables so your system performance doesn’t go off the cliff when it spills out of RAM.
One of the big benefits is that the DDTs won’t contain all blocks, but only recent blocks as the belief is that duplicated data tends to be temporally correlated
Ie, if you don’t get a dupe to a block in 60 days, you can assume you’ll never, and that block can be dumped from the DDT
Actual duped blocks are never dumped.
But if you think about it with traditional Dedupe the vast majority of blocks tend NOT to be dupes. So… it’d be good if they didn’t have to be tracked in the dedupe tables 