I’m using TrueNAS SCALE 24.10, the latest stable version of TrueNAS SCALE.
One month ago, 160TB of data was copied with the bclone feature enabled. This is a large dataset. During that time, a lot of data writing was observed, and it took a lot of CPU time.
Now, those copies of data are no longer needed, and I’m deleting them. This process is very slow. I invoked watch -n 1 "zpool get all my_pool | grep -i bclone" to monitor the bclone value. It took 20 minutes to decrease from 160 TB to 140 TB. During this process, a lot of disk writing was observed, too.
Why are there so many write operations, and why is this process so slow?
All the writes are for metadata pointing to the data. Since ZFS is a COW, (Copy On Write), any pointers to that lower level metadata also have to be updated as well. All the way up to the top, reaching the Uber blocks. For each transaction that does deletes.
If you were able to / had planned the data more carefully, you could have destroyed the dataset containing the block cloned data. Might have been faster. (Or perhaps you did ZFS destroy the dataset…)
ZFS was designed so that all operations are interruptible, including dataset / zVol destroys. The side affect is that after some data is deleted, full metadata updates are required to make the deletes effective AND still have the pool 100% on-disk consistent.
Now if you want to go into the past, Solaris 10 ZFS had a serious problem with larger ZFS dataset destroys. (It is solved now.) It was such a problem that some Sun Microsystem customers would try rebooting their Solaris 10 servers that hung / were slow destroying larger ZFS datasets. Unfortunately, it did not help as the ZFS destroy would have to restart. Stupid I know…
One of the first things the Open Source version of ZFS did, (independent of Solaris ZFS), was Async Dataset Destroy: feature@async_destroy
That was a game changer. Still not perfect, but worlds better.
A later OpenZFS feature was to have asynchronous deletes of larger files, run in the background. Not sure if that got implemented.
