I pretty much have my build figured out by now. My bottleneck is most likely going to be my network. I might end up bonding two or three 100 GBpe ports eventually to relieve that bottleneck, and allow one machine to copy 1 TB within 3 minutes to my ZFS system. I’m guessing if I do that I’ll need to use a few special vdevs as a write cache, before propagating the data down to the much larger HDD pool.
Asking Grok about this it recommended I build the special vdevs and script according to these specifications.
Write Cache Strategy with Special Vdevs:
Write Cache Using Special Vdevs: You’re on the right track thinking about using SSDs or NVMe drives as a sort of write cache. ZFS does not natively support using special vdevs as a write cache in the way you describe where data is automatically moved to HDDs after being written.
ZFS Cache Devices: ZFS does support cache devices (L2ARC), but these are for read caching, not write caching.
Manual Write Cache: To achieve what you’re describing, you would indeed need to write a script or use some form of automation to move data from SSDs to HDDs. Here’s a conceptual approach:
Write data initially to SSDs configured as special vdevs.
Use a script (like a cron job or a custom daemon) to periodically move data from SSDs to HDDs based on certain conditions (e.g., when write operations are below a threshold or during off-peak times).
You would need to ensure data integrity and consistency, possibly using ZFS snapshots before and after moving data to ensure no data loss.
Is there already a project out there for this purpose? It sounds like writing my own script would not be too complicated. I’d just have multiple special vdevs configured to absorb all writes, right?* Then when the specials get past a threshold and/or no writes are coming in they move their data down to the zraid or draid beneath them.
So, before that script moves the data I take a ZFS snapshot, and after it’s done I take another snapshot, and compare the data to each other with a ZFS diff to make sure everything is the same. If there are any differences I just move the blocks the diff does not match on to the same location?
*I’ll most likely add the normal stuff for improving write performance first.