Warning about 16k record size on a dataset

TrueNAS General
1 
WARNING: Based on the pool topology, 32K is the minimum recommended record size. Choosing a smaller size can reduce system performance.

Sure, the wrong record size can reduce performance. The right record size can increase it.

Also I’m more concerned with efficient storage use here than performance; backups happen seldom (I mean, not every hour or something!) and unattended, automated. A LOT of the files are smaller than 16k.

This dataset is going to be for rsync backups of web sites, where images are rare and those present are small files (decor rather than the big deal). 16k seemed like a good size for that. (And, in datasets for backing up a photo-heavy site, I’d pick a larger record size.)

Is “based on the pool topology” the key here? The pool this is on is a RAIDZ2 of 6 x 6TB drives (just that one vdev).

I also suspect that the difference in record size won’t matter much for the number of files and such I have here, which would argue for a conservative record size rather than spending time over-optimizing anything. This isn’t like a dataset intended for an active database, say (I believe that matching the database page size is what’s wanted there; but I’m not running any databases at home so I’m not making such datasets).

2

Set the dataset recordsize to 1M. Don’t overthink it.

You could have the recordsize set to 16M and those files will still be saved as single blocks on the disk at 16K or smaller in size. That doesn’t even take compression into account.

3

So the default (128k) isn’t a suitable default?

And – oh, does it not waste space when I save a 20k file to a dataset with a 1M recordsize? How does that even work? (But I’m not familiar with the insides of any modern filesystem; I don’t even know how compression works without really messing up seeks!).

4

Maybe in the year 1900 it was. :smiling_face_with_sunglasses:

My default is 1M and I couldn’t be happier. I’m almost inclined to switch to 4M. There’s a certain point where you lose out on parallel processing. The sweet spot seems to be 1-4M.

The reasons for 128K being the “default” is to play it safe for users who don’t really know what they’ll use their datasets for. It’s not too small to go crazy on the overhead and inefficiency, but it’s not too large that it would cause noticeable write amplification for in-place modifications and databases.

If you already know your use-case, you might as well set it to the optimal recordsize, such as 16K for certain DB software, or 1M for typical file usage, or something very small for VMs, in which you’d use blocksize with a zvol instead of a dataset.

I don’t buy that 128K is a “balance”. It’s not good for DBs and it’s not optimal for normal usage of just saving files for storage and reading.

Imagine a country that sets the universal speed limit to 35 MPH. “People drive up to 80 MPH on the highway and 20 MPH in their neighborhoods. We think that 35 MPH is a good balance. Everyone should drive at that speed.”

“Can the entire file fit into a single block that is equal to or less than the recordsize?”

“Yes”
→ Store the data inside a block that is the smallest binary iteration possible. (4K, 8K, 16K, 32K, 64K, … 128K, 256K, 512K, 1M)

“No”
→ Store the data inside 1M blocks. Use however many 1M blocks are needed to store all of the file’s data.

You might think that this makes sense for the first scenario, since it doesn’t use up a full 1M to save a 20K file. What about the second scenario? That’s a lot of wasted space on the last block of every file. That’s up to 1M of wasted space per file.

Are you using any inline compression at all? ZLE? LZ4? ZSTD? Then it’s moot. Null bytes at the end of the last block compress into nothing, take up no space, and it happens extremely fast.

This is why even for datasets that will hold media or uncompressible files should still have ZLE or LZ4 compression enabled, at minimum.

5

Just starting to experiment with zstd. I’ve got lz4 on by default, I don’t think I’ve turned it off anywhere. (I just updated to the default zpool level for TN 25, so that I can hold off in TN 26 for a while, if that makes sense. It was something ancient before that.)

So the “recordsize” isn’t the only division of the disk space? There are sub-blocks or something, managed somehow? I probably need to experiment with some code that creates large sets of files with various different size distributions and see how much space the datasets containing them really take up.

This looks to be an area in which the user documentation desperately needs to be updated.

6

When it comes to storage on physical disks (and the ARC in RAM), a file is divided into blocks if the file is larger than the recordsize value.

Each block is the size of the recordsize value (1M) before compression. While still in RAM, each block of data is compressed → encrypted → checksummed. Now it is written to physical storage.

The last block is padded with null bytes to fill the block, but it’s moot and a non-issue. Compression immediately truncates it to nothing. As a matter of fact, this may be outdated information, when in fact these null bytes are truncated regardless, even without compression.[1]

  1. I’m not sure about this, but I would have LZ4 compression enabled at minimum anyways. ↩︎

7

In honesty, and fairness to the engineers, most people who “think they know their use-case” are, let’s say “optimists”. When I start thinking about “what’s really there” with any care it gets interesting.

But I do have, right now, 2 different backup datasets which contain nothing but 1.2GB files (compressed tar files). And I have others which contain a more random range of files, they’re directly rsynced from the web host. It seems like there might be some benefit to be had optimizing for those rather big differences, but quite possibly not as much benefit as it would take effort to figure out what was really right.

Then in other areas, I’ve got datasets that contain tens of thousands of 10MB and 20MB image files, plus larger numbers of smaller files (like, a tiny xmp file next to each raw file, and often smaller rendered copies as well). I think of that as “lots of big files”, but it has even larger numbers of small files still.

8

Those small files won’t cause “inefficiency” with a 1M recordsize, unless you have software that does true in-place modifications. (Unlikely.)

A 20K file will consume 20K[1] on the disk.

A 31K file will consume 32K[1:1] on the disk.

A 61K file will consume 64K[1:2] on the disk.

A 1.25MB file will consume 1.25MB on the disk, split between 2 blocks. (1M and 252K)

Another 61K file might only consume 32K[1:3] on the disk if it is highly compressible.

  1. 4K is the smallest unit possible. A 1K file will still take up an entire sector of 4K anyways. ↩︎ ↩︎ ↩︎ ↩︎

9

I don’t think there’s in-place modification for the xmp files or the rendered jpegs. There might be for the bigger files, but might well not (ExifTool, Photo Mechanic, and Lightroom are sophisticated in lots of ways, they might be in that way also, perhaps). Going to some effort to not re-write all of 20MB files (or in some cases 10GB files) might be worth it, if it’s in fact possible.

There isn’t for the general backup files. And as I said, not running databases at home (currently).

Hmmm; fascinating. I’ve been kind of curious about “recordsize” for some time, so getting filled in on things now make me happy!

10

Many programs were designed to do their own “safe copies” under the hood before CoW filesystems like ZFS and Btrfs became mainstream.

They don’t make it obvious. Oftentimes they make a copy of the original as a temporary hidden file while the “modified” file is being written to disk. This isn’t a big deal for non-CoW fileystems. It’s actually the safe approach. It’s a problem for CoW filesystems like ZFS and Btrfs, since as @HoneyBadger put it: “The software is doing it’s own copy-on-write”, even though ZFS already uses copy-on-write.

You mentioned you’re using rsync. Make sure to use the --inplace option when writing to a ZFS destination, since --inplace tells it to do a true in-place modification.

11

Yeah, mediocrity has become standardized, is perhaps the nasty way to look at it. As you say, not trashing the user’s files was a big deal, and they weren’t getting as much help then as now.

(I had my mind kind of blown by copy-on-write memory management and a filesystem mostly used by mapping portions of files into memory back in the late 1970s. Different issues for memory management than for filesystems, but I’m reminded of former glories or at least former head-melting and coming back together. TOPS-20.)

12

Well, I’m trying 1M recordsize in my new datasets. Local copies within the server are fast enough that, if I have to create new ones and copy the data over next month, it won’t hurt that much, and this sounds like a good choice.

13

I use smaller recordsizes for my apps storage to force app data like my pihole blackhole files into my sVDEV. That way, all that data is accessible at SSD speeds.

Smaller recordsizes also make sense for lots of small files getting changed all the time, like database contents.

I also significantly reduced my file related ZFS overhead by storing archival file systems in Apple Sparsebundles vs. natively. There are a few thousand largely dormant sparsebundle files compared to hundreds of thousands of small files making up a standard file system (lproj files inside a system folder on a Mac, for example)

An rsync on a sparsebundle will thus run circles around a native file system in a WORM use case. Each band will only require one checksum for hundreds of files. All that IO really adds up.

There is a performance penalty associated with mounting a sparsebundle and navigating through it but given how rarely I make such journeys vs. how often the sparsebundle is being backed up, I gladly make that tradeoff.

14

@dd-b - One concept that has not been covered, is that ZFS RAID-Zx can do partial stripe writes. With 6 columns, 2 of which are parity, (RAID-Z2), then 4 columns are data. If a file is small enough to fit into less than 4 data columns, that is what ZFS will do. Even down to 1 data column, (even though their would still be 2 parity columns).

Thus, ZFS is not as efficient writing smaller files that don’t take up a full width of the RAID-Zx data stripe. Generally not too much of a problem when having a mixture of files larger than data stripe size, and files smaller than data stripe size. But, horribly inefficient for things like VM storage where the VM wants to write 4K - 16K records.

Note that dRAID vDevs are always full width stripe writes. This was part of the reason why Special Metadata vDevs are helpful for dRAID vDev pools.

Actually, I vaguely recall tail blocks where tiny amount of data can be stored in the directory entries’ Metadata block(s).

15

Finally getting back to this. It’s been a week. Month. Duration. :face_with_spiral_eyes:

I see we already covered a lot of the bases w.r.t recordsize being a ceiling, not a floor, so the smaller files will get written and compressed down inside it, as well as RAIDZ being capable of partial stripe writes.

This is the part doing the heavy lifting - it has to do with padding records. Pulling up from the ancient tomes: