Given that the metadata hits are not in the 99%+ range, I’d like to think that it will be noticeable to get the metadata from a SSD (using a metadata-only L2ARC) vs. reading it from a HDD. HDDs struggle with small file access in general, so directory traversals really speed up when the data has lower latency. See here for an example.
Granted, this is a worst-case scenario where the entire pool is traversed exhaustively. This is unlikely to be the case for you in your day to day as you work on a specific set of folders, etc. But it does give you an idea about how pool-wide operations can be impacted.
But this is driven in part by work flow / use cases. My system benefitted a lot from metadata being readily accessible because it made rsync operations so much faster. If you don’t regularly use rsync for backups, a similar program to traverse a lot of directories (i.e. potentially lightroom), then the use case isn’t there.
Also, even if you have a persistent, metadata-only L2ARC, it has to “get hot”. That will take multiple “misses” as you can see from my data. As of SCALE, all L2ARCs are persistent by default, but the metadata-only setting has to be set manually.
I have a question. Maybe it’s even worthy of a separate topic. I saw in multiple sources statements like “if the ARC hit ratio is above 90%, there is no need for L2ARC”. Well, my average ratio is very close to the 90-100 range. However, looking at the report page, it can drop to something like 50-% during the load (backups).
So, the question is – does this rule of thumb apply to the general hit ratio or to the hit ratio during a particular load? Would the performance (during the load) be improved with L2ARC? Well, does a 90+% hit ratio during the nonpresent activity even make sense (to be taken into account) after all?
I’d say it really comes down to the use case. For my mostly WORM NAS, having a L2ARC made a big difference because of rsync. It also made directory traversals for the user via SMB much more consistent re: NAS responsiveness.
One of my major use cases (offsite backup) was running a lot of rsync jobs that took ages without the benefit of a metadata-only L2ARC and that could finish in an afternoon with it.
Nah, my responsiveness is good (perhaps thanks to the sVDEV). Thanks for the reply anyway. I will create a separate topic later.
There is likely zero reason for a metadata L2arc if you have a decent, properly-configured sVDEV. I briefly played with keeping the L2ARC around for frequently accessed files after installing my sVDEV and found it hopelessly un-needed. So I repurposed the drive.
Metadata hits are 99.8% - the 88.3% is the % of data requests that were metadata rather than data, with 11.4% being data access, and the rest being pre-fetch accesses.
@thoresson - with an overall ARC hit rate of 99.4%, that means that out of every 1,000 i/os, 994 are being satisfied immediately from memory and only 6 are going out to disk.
This means that if you implement the L2ARC, you will *at best only see a 0.6% performance improvement if all I/Os currently to disk are sourced from memory, but the reality is that sometimes you do need to get actual data from HDD which you haven’t accessed for a long time, and these will not be in ARC or L2ARC and so you won’t even achieve this 0.6% improvement. Against this, because the ARC size will be reduced, some I/Os currently sourced from ARC will end up being sourced from L2ARC (which is significantly faster than HDD but significantly slower than memory) and this will further bite into the performance improvements. Personally I doubt you could notice 0.6% performance with the naked eye, nor notice the performance degradation from reduced ARC - in other words, from a practical perspective, IMO L2ARC will be a waste of technology for your system.
This is why use case is so important. I’ll add though that my ARC stats were super high also but that complete traversals of the NAS took hours instead of minutes when the L2ARC was fitted.
I imagine something along the lines of averaging, how many minutes the ARC stats go back, etc. have something to do with that. So ARC stats can claim what it wants to, the real question is the results when the machine is doing its job.
For my use case, that includes backing up and those backups were 12x faster for a mostly WORM NAS with L2ARC enabled. I collected the data and posted it here for that reason.
Given that L2ARC can be added and removed without penalty, my suggestion would be to explore adding a L2ARC on a metadata only basis. Naturally, there should be adequate RAM and no sVDEV also. Then, test. If there is no big benefit re workflow, the L2ARC can be removed again.
