“Blocks” are units of storage for datasets as well. The nomenclature can get confusing.
Think of “recordsize” as the policy for the maximum size that a block can be.
Think of a “block” as the actual data itself as seen in RAM.[1]
Think of the “block-on-disk” as the form (and size) of the block as stored on the physical storage medium. (This is also the same form as held in ARC when applicable.)
Some examples:
If your dataset’s “recordsize” policy is 1M, and you save a non-compressible 8K file, then it will be comprised of a single block that is 8K in size. It will be 8K in RAM[1:1] when used by applications, and 8K stored on disk.
If your dataset’s “recordsize” policy is 1M, and you save a non-compressible 980K file, then it will be comprised of a single block that is 1M (next power-of-two) in size. It will be 1M in RAM[1:2] when used by applications, and 980K stored on disk (or 1M stored on disk if you’re not using any inline compression).
If your dataset’s “recordsize” policy is 1M, and you save a highly-compressible 980K file, then it will be comprised of a single block that is 1M (next power-of-two) in size. It will be 1M in RAM[1:3] when used by applications, and perhaps 160K stored on disk. (Again, assuming inline compression is enabled.)
If your dataset’s “recordsize” policy is 1M, and you save a non-compressible 4.5M file, then it will be comprised of a fives block that are 1M each in size. All blocks will be 1M in RAM[1:4] when used by applications, and 1M each stored on disk, except for the last block which will only be 512K if you’ve enabled any form of inline compression.
For any scenario above, whenever you do a true in-place modification, the entire block must be read, decrypted, decompressed, and rewritten as a new block, and then have the pointers updated to include this new (modified) block. Consider the last example: Modifying in-place just a few KB in the middle of the file will require reading and rewriting 1MB worth of data. (This doesn’t mean that an additional 1MB of data will be consumed on the disk, since compression can shrink the final size. If the 1MB of data in the block is incompressible, then yes, it will ineffeciently waste an extra 1MB on the disk, even if only a few kilobytes were changed.)
Enabling ZLE inline compression (at minimum) will removing any “padding” at the end of a file’s last block. (ZLE is insanely fast, and compresses a sequence of “null bytes” into nothing.) Therefore, the last block of a file might only be a few KB, while the remainder is “padding of null bytes” up to the 1M recordsize limit. On the storage disk itself, this block physically consumes only a few KB, since any form of inline compression makes this padding a non-issue.