L2ARC: Support separate metadata and data cache regions

Feature Requests
1

Problem/Justification
When the L2ARC SSD is smaller than the hot data set, L2ARC’s cache-all mode will eventually evict less frequently used metadata, resulting in expensive random reads later.

It would be useful if L2ARC could be split into two circular buffers: one dedicated to metadata and the other dedicated to everything else. The configuration could specify that X% of the L2ARC capacity is reserved for metadata, with the remaining capacity used for all other data.

The X is entered while adding L2ARC device. So admin can decide the best number for their load.

Impact
This would make L2ARC more useful by allowing users to cache both metadata and general data without having to choose between metadata and cache-all modes. There is no significant disadvantage to this approach.

User Story
An administrator does not want to risk the entire pool by relying on a small number of special-vdev SSDs, so they use L2ARC to accelerate random I/O.

Suppose a user needs to traverse a large directory tree every few days. With a cache-all L2ARC, the relevant metadata is cached after the initial, painfully expensive series of random reads. However, after one or two days, that hard-earned metadata may be evicted by cached file data. The next time the user traverses the directory tree, they have to suffer the expensive random-read I/O again.

With this feature, a dedicated portion of L2ARC would retain metadata, preventing it from being evicted by file data and improving performance for subsequent directory-tree traversals.

2

This is not a TrueNAS request but an OpenZFS one.

3

I think there is already an setting for what you are trying to achieve

4

AFAIK that only applies to the ARC in RAM. The OP would see improvements in directory traversal, crawling, and random seeks by favoring metadata in the ARC. The more RAM, the better.

The L2ARC will be populated with evicted metadata and data. It’s an “interesting” design decision. Rather than reads from storage being “cached” in the L2ARC, they reside in the ARC (RAM). From here they can be relocated to the L2ARC… if they are evicted.

To this day I think ZFS would have benefited from a more traditional “cache” on a fast device, like the OP is suggesting. This would not only liberate a lot of RAM for other uses, but would be more intuitive for home users. Not as a replacement, but as another optional feature.

5

Yeah. Maybe it helps you store all metadata in ARC? And L2ARC you only need for some evicted metadata and files. Or maybe OP notices that he/she no longer needs L2ARC because of it?

I have a pretty hard time imagening in which scenario that would help.
Especially considering that we have special vdev. Only downside there is that it is critical for the pool. I don’t think that this is a huge issue, but many users seem to dislike that about svdev.

Maybe some kind of svdev that also stores the metadata on the pool? Sure the write gains fly out of the window, but you could have a metadata read cache with a single, non ciritical SSD.

7

That would actually be really useful for someone like me who cares more about metadata read performance over write speeds.

The fast device would be non-critical, but serve metadata to RAM/ARC before attempting to pull it from the spinning HDDs? If this fast device fails or is removed, your pool is still intact because the same metadata being written to it was also being written to the data vdevs (HDDs)?

It’s kind of like a “+1 metadata mirror” made of a fast device that has “read priority”, which isn’t critical to the pool’s integrity? :open_mouth:

8

How is that different to a metadata only L2ARC?

Would the ‘l2arc_mfuonly =2’ flag solve the original issue?

9

L2ARC is populated from ARC eviction.

The idea by @Sara would bypass ARC hit/miss logic but still benefit from NVMe speeds and random seeks without putting the pool at risk with a critical vdev.

10

I feel like MRU churn causes that to happen anyway. i.e the cache fills readily in practice despite needing imminent evictions.

Certainly my L2 metadata only disk seems to fill with new metadata easily. I need to have a play with this, and the setting I posted above if it is supported.

11

L2ARC needs RAM, so I am not sure about how great a ARC & L2ARC metadata only would work. Doesn’t all metadata fit in ARC anyway and the only real advantage you get over just ARC, is that it is reboot persistent?

IMHO the “essential for your pool” part of special vdev gets a bad rep. In reality, it is nowhere near as bad as you think. Get two trashy 128GB system drives you have have lying around. Does not matter if SATA or NVME. Special vdev is the easiest task there is in most cases. It is only tiny amount of data. Even the trashiest SATA SSD will seriously outperform you HDD pool in terms of IOPS. If one fails, simply replace it and resilver it within what, maybe half an hour?
If you want to be on the cautious side, go with a 3way mirror. Whatever you do, I would use two different SSD brands, with different controllers.

12

I am running 3 pools. 1 pool has spinners, it is backed by a metadata only L2ARC of 13.41 GiB. I have 64GB RAM, with typical system load I have a 32GB ARC. Just the HDD pool alone has a metadata size of 11.5Gib (and Psize 3.5G), this is with a 1M record size. If I run multiple queries including complex directory traversal on the SSDs some of the metadata in ARC gets evicted and reloaded from L2ARC. My metadata only L2ARC is as below:

L2ARC size (adaptive): 31.5 GiB
Compressed: 26.8 % 8.4 GiB
Header size: 0.4 % 135.9 MiB
MFU allocated size: 11.0 % 945.8 MiB
MRU allocated size: 125.4 % 10.6 GiB
Prefetch allocated size: 0.3 % 29.4 MiB
Data (buffer content) allocated size: 0.0 % 0 Bytes
Metadata (buffer content) allocated size: 136.7 % 11.5 GiB

This results in fairly rare hits, but improves folder browsing performance a little. 6.5 % hit ratio, but a paltry 4.1Gib read.

Then I changed L2ARC to store all metadata, and also only data MFU.
In GUI System > sysctl I set type:ZFS, Variable:l2arc_mfuonly, Value 2

sudo zfs set secondarycache=all main

This resulted in:

  • A huge increase in write to the drive. Peaks around 200Mb/s (ongoing after hydration), mean 2 MiB/s, vs 3 KiB/s with metadata only.
  • Read speed averaging about 2% the backed array reads when running normal operations.
  • Read speed averaging about 6-10% the backed array reads when running operations ennumerating a lot of files. I think this is more meaningful than total hits as folder navigation was A1155’s problem.
  • An impression of quieter drives, suggesting in practice this is doing a good job of absorbing additional random reads beyond ARC.
  • Folder navigation not appreciably slower, possibly faster.

My vague thoughts after having a play:
MFU data + metadata L2ARC has been supported since about EE 24.10. It offers an alternative to MFU only or metadata only L2ARC. Even on a very small drive, it outperforms a metadata only L2ARC, at the expense of much heavier writes. It is possible the write volume would be reduced with a larger disk, as rewriting evicted data is presumably occurring. For WORM datasets this may allow larger lower endurance SSDs to be viable for L2ARC, and the data MFU is a fraction of MRU (and by extension a fraction of total evicted data) - Not shown, but my overall ARC is a 20/80 (MFU/MRU) ratio for L2 eligible evictions.

Of course this doesn’t address all cases, nor some of the other points made in this thread. But it is an existing feature which may address A1155’s problem.

After the new settings have had time to settle I will edit this post to update. I had to reboot for unrelated reasons so my current data reflects restarting services rather than usual running.

Update: I let it run at MFU + metadata only cache for a few days (below). It is backing 18TB, of which about 6TB is likely to be hit. Writes have just become smaller than reads, and L2 reads are about 40% of the array’s total reads. Navigation and overall responsiveness is a little better than with a 16GB drive, but the directory traversal and retrieval of thumbnails not seen for a while is noticeably more consistent (and the array quieter during these actions). The total metadata loaded in is surprisingly small. Not shown is the increased writes to small MFU+metadata L2ARC vs metadata only, which is nontrival for small drives. At 1.5TB I have started getting low memory aborts. So in conclusion: It depends, but a MFU+metadata L2ARC should be considered as an alternative to metadata only, but you need to tolerate increased writes.

L2ARC size (adaptive): 1.5 TiB
Compressed: 99.8 % 1.5 TiB
Header size: < 0.1 % 146.7 MiB
MFU allocated size: 80.5 % 1.2 TiB
MRU allocated size: 8.8 % 138.8 GiB
Prefetch allocated size: 10.7 % 169.1 GiB
Data (buffer content) allocated size: 100.0 % 1.5 TiB
Metadata (buffer content) allocated size: < 0.1 % 703.2 MiB

Update2: The TN GUI gets a Massive Sad if you disconnect 1.5TB of cache from an array under moderate load.

13

I ran a persistent, metadata-only L2ARC for years here and I’ll simply caution that use case really matters.

ZFS kept insisting that my ARC hit rate was 99% or whatever without the benefit of a L2ARC. Per the usual guidance, that was high enough, there would be zero benefit re: L2ARC.

In my use case (and that distinction is important), a persistent metadata-only L2ARC sped up my rsync backups by a factor of 12x. Regular browsing was also noticeably faster.

Why? Well, rsync exhaustively traverses the source and destination disks as it looks for differences. Those traversals really add up, especially if the data being compared is full of small files (like operating system files these days).

I’m not going to put on a conspiracy hat, just reiterate that the stats are nice but real life testing with your use case is better.

Being able to allocate how the L2ARC splits metadata and data files in it would be a neat upgrade and similar to that functionality already built into sVDEV.

For funsies, I did keep a data-only L2ARC around after upgrading to a sVDEV. For my use case (mostly WORM), the impact was negligible, so I removed it.