Questions about L2ARC rule of thumb

[swc-phil]

I saw statements like “if the ARC hit ratio is above 90%, there is no need for L2ARC” in multiple sources (1, 2).

My average ratio is very close to the 90-100 range. However, looking at the report page, it can drop to something like 50-% during the load (backups, but I’m asking for the general case).

My questions are:

1. Does this rule of thumb apply to the general hit ratio or to the hit ratio during a particular load?
2. Could the performance (during the high load) be improved with L2ARC?
3. Does a hit ratio during the nonpresent or low activity even mean that much after all? I mean, it makes sense in the enterprise environment with a nearly 24/7 load. But for home use? Not relevant.

[Constantin]

1. I’d argue it has more impact on your experience for the general use case. So I’d focus on ARC stats that are representative of your user experience, ie what do you care about? For example, I care about how long it takes to backup to my DAS, so a L2ARC or sVDEV made a huge difference for me while using rsync. My other users don’t notice the difference.
2. Depends really on what your use case is. For example, your NAS already has a sVDEV, so the use of a L2ARC should be close to zero unless said L2ARC configured to ignore metadata AND you have a bunch of read-only files that are not stored in the sVDEV. That use case may exist (Plex catalog?) but I have no experience with same.
3. Really depends on what your system is trying to do, no? If the system is idle but a task takes a long time, who cares? At the same time, you may want certain tasks executed in idle times and be done by the time high use hours start, so even idle time may be bounded by other use considerations.

[swc-phil]

I gave it another thought. Question 3 is not relevant, as the average ratio is calculated from all hits and misses, and there are not many hits or misses when idle. So average hit ratio is a useful characteristic despite the lower ratio during the loads.

I’ll edit my post.

[Arwen]

We have a Resource about L2ARC. Probably needs some updating, especially when the new improvements on ARC / L2ARC are more common.

Here it is:

L2ARC tuning guide and common misconceptions

We have had lively discussions of ZFS’ L2ARC, as L2ARC has both changed over the years, and how to tune the L2ARC has improved. However, this discussion was taking up much of the responses in this thread: Here are common misconceptions: Size should be about 5 times RAM and no more than 10 times RAM. This is no longer true. L2ARC should not be added until RAM is maxed out. While more RAM is better / faster, using L2ARC on even 16GB RAM configurations is practical. L2ARC takes up lots of RAM,…

[Johnny_Fartpants]

Max out the RAM on your mobo first and then consider adding L2.

[swc-phil]

Yeah, already read it. Mb even twice.

[Arwen]

@Johnny_Fartpants

Max out the RAM on your mobo first and then consider adding L2.

While this is definitely better, it turns out with the L2ARC improvements that is no longer a hard and fast rule.

That said, not every non-maxed out RAM server and the applications run on it, can benefit from a L2ARC.

[Constantin]

The devil is in the details and I always suggest testing under the use case that matters to the end user.

For example, my NAS’ arc stats looked perfectly fine but the addition of a 512GB metadata-only, persistent L2ARC to a 32GB RAM C2xxx system sped up rsync operations by 12x once I ran three rsync operations in a row and got the cache “hot”.

The key is to figure out how to test the system under “use case conditions”. That’s not always easy. But I would try to test objectively rather than rely on stats that may not be reflective of impacts on work flows that matter to the end user.