Special VDEV (sVDEV) Planning, Sizing, and Considerations

Actually, I came to the conclusion that zfs-inplace-rebalancing was the better option for me, since zfs rewrite doesn’t apply any changes to the record size, and I wasn’t sure how much of my data was in an older, smaller block size from many migrations from system to system over the years.

Current setup is 2M record size and 1M small block size, working great! And in my case 50TB in the data vdev is using only 140GB of the special vdev (out of 2TB); so now I wish I’d chosen 4M recordsize and 2M small block size, but 2M wasn’t an option in the UI. Probably only minor gains to be had; just wish I was using more of my special vdev.

So,

  1. rewrite is OK as long as the recordsize is not changing, while
  2. rebalance is needed if recordsizes do change?

If I read the rewrite command manual page correctly, a rewrite can be done without updating snapshots, which is attractive. A rebalance would require snapshots to be turned off first followed by re-enabling them after the rebalance, correct?

Sorry for the pedantic questions, I’m contemplating changes to the resource to better reflect what the current state of the art is.

Yes, zfs-inplace-rebalancing will update recordsizes, but will require snapshots to be turned off. zfs rewrite can be done without updating snapshots with the -P flag, but TrueNAS 25.10.6 (the latest General release) still runs zfs 2.3.4 and doesn’t support rewrite with -P which was added in zfs 2.4.0 (it should work in TrueNAS 26 beta, but I haven’t tried that myself yet).