L2ARC tuning guide and common misconceptions

Resources
,
59

I am absolutely not an expert on ARC or L2ARC, but I have a couple of comments:

  1. My ARC on my small NAS (media / backup server) behind c. 10TiB of data is only c. 3.5GB in size, but achieves a cache hit rate of > 99% (so >>>>> 90%).

  2. With that level of hit rate, I can only assume that what falls off the bottom of the MFU cache and would be moved to L2ARC if I had it would be very rarely read.

Steady state

NAS systems are supposed to stay up for months on end, so let’s consider the situation of a NAS that has been up and running for long enough for ARC to be dropping stuff off the bottom of the MRU / MFU lists.

My gut reaction based only on my unique sample of one system is that if you have a reasonable amount of ARC but are getting < (say) 95% cache hit rate, then either

  1. Your workload is sufficiently random that ARC doesn’t help - in which case neither adding more memory nor adding L2ARC will help.

  2. Your workload is not random, in which case adding more memory to increase ARC is likely to improve cache hits and that might well be a better investment than L2ARC.

  3. But if memory is maxed out, then L2ARC might help serve metadata / data that would have stayed in ARC had it been bigger.

This is still pretty much what the old guidelines used to say. Yes - reducing the L2ARC overhead from 180ish bytes to 90ish bytes is a help, but L2ARC still take memory away from normal ARC functions.

Persistent L2ARC

Again, naively perhaps, it seems to me that the older non-persistent L2ARC didn’t have that much to offer before reaching steady state (e.g. as an edge case during boot) because it was empty.

But now it is persistent then I can see that it might be beneficial to populating ARC before it reaches stability - kind of in the same way that Windows has boot optimisation using a pre-fetch cache except on Windows it is the same boot drive whereas L2ARC is on faster technology.

L2ARC vDev vs. special allocation (metadata) vDev

I do appreciate that L2ARC is a helper cache which doesn’t lose data if it fails, whilst Metadata vDevs are critical for the pool and needs to be redundant (so significantly more hardware & $$$), but if I am designing a new system / pool which should I choose (assuming that it is one or the other and whichever I choose will be on the same NVMe Optane technology)?

L2ARC / Special vDev allocation block sizes

Can you specify an asize or similar for these specialised vDevs, and if so what relationship should they have to the asize of your pool/data vDevs and how should they relate to your dataset recordsizes (which have no equivalent in either L2ARC or special vDevs)?

I genuinely have no idea what the answers are or whether my guesses or right or wrong. So if anyone can respond to these points that would be great.

1 Like