Even though L2ARC probably doesn’t use the exact same eviction algorithms as ARC, there’s certainly some intelligence behind which blocks get evicted, it’s not just a simple “oldest first” policy.
I also agree that the entire NVMe space used for L2ARC is essentially always full. Cached blocks aren’t discarded just because the algorithm says so; they’re only overwritten when space is needed for new data. So the “ring buffer” argument basically still stands.
I doubt that TRIM would make any real difference here. TRIM’s usual advantage (allowing the SSD to perform garbage collection ahead of time) doesn’t really apply in this case, because the L2ARC is always fully utilized. A decent NVMe drive constantly manages its own read/erase/write cycles internally as part of normal NAND operation, but handling this on the fly, may slightly reduce performance.
In other words, the L2ARC device will always be close to 100 % utilization, and TRIM won’t really help. I’d also avoid cheap QLC-based NVMe drives for this purpose, since they can wear out quickly under such a workload, and performance-wise they’d probably be a real PITA.
Here it is as attachment, it’s really too long and barely understandable to me!
ah: keep in mind I got the file as soon as the machine booted!
So the TRIM command works at pool’s level and not at disk’s level?
As soon as I will get the RAM sticks, I will tell you… if you all tell me what to do to test them! LOL!
oh well…. a Crucial P3 probably is exactly that cheap QLC-based drive you talk of But it is what I have at the moment. If you can suggest me some better model I’ll consider it for Xmas
Well, if you already have one, you can definitely try it out and see how it performs under your specific workload (or if it makes a difference at all…)
Since it’s only being used as an L2ARC device, there’s no risk of data loss if it fails — it’s just a cache after all.
QLC isn’t really ideal for this kind of use, though, for several reasons:
The internal read/erase/write cycles (garbage collection) are more complex and therefore slower on QLC.
Once any internal DRAM or SLC cache is exhausted, QLC drives become much slower compared to TLC.
And of course, there’s the endurance factor (lower TBW).
That said, go ahead and give it a try, you can always monitor the drive’s stats and see how it holds up over time.
Under the hood it is on the drive level, of course. Perhaps it can be tuned in this way. I just didn’t bother.
Not a specialist myself. The most interesting one imo is ARC demand data accesses. And ARC demand metadata accesses if you have slow directory listings. For L2ARC it is probably L2ARC breakdown.
You should use your system for some time before these stats start to make sense.
Thanks, Phil, for the interesting link.
L2ARC really is just a ring buffer, but since it’s fed with data that’s about to be evicted from the ARC, it’s still much smarter than the simple LRU-type caches used by other filesystems.
The part about L2ARC feed rate and its impact on SSD endurance was also very insightful. Learned something new again!
That’s not good, as you really want to have used the NAS for quite some time before obtaining statistics. ARC needs to fill up first!
The relevant part is here:
ARC total accesses: 39.7k
Total hits: 96.9 % 38.4k
Total I/O hits: 0.5 % 179
Total misses: 2.7 % 1.1k
ARC demand data accesses: 0.0 % 0
Demand data hits: n/a 0
Demand data I/O hits: n/a 0
Demand data misses: n/a 0
ARC demand metadata accesses: 98.4 % 39.0k
Demand metadata hits: 98.0 % 38.2k
Demand metadata I/O hits: 0.1 % 53
Demand metadata misses: 1.9 % 738
It is not representative so far, but if it were you would not need a L2ARC.
after one day of moving, copying, fixing, deleting, renaming, indexing an huge number of files I’ve measured that copying a total of ~3TB of data from fast storage has needed ~4 hours, for an average of 170-180MB/s . Not bad but not exactly a Speedy Gonzales’ thing!
SO if I would increase the whole general speed of the system what should I do?
I was reading somewhere about the possibility of setting: atime=off xattr=sa
as compression is already set to LZ4, and record size to 1M. What do you think? Any drawback?
Just to put this into perspective: you copied 3 TB in 4 hours (which is around 210–220 MB/s) from from “fast storage”…
over 10 GbE ?…
to fast storage? slow storage?
If so, even with 10 GbE, lots of small files can severely limit SMB throughput.
So, this could already be reasonable for this workload… (I’m not sure if any further fine-tuning would help much….)
But… is this workload really representative? It seems to me like a one-time / initial transfer…
just a bit less of 3TB, actually it was 2,77TB … and yes from an external NVME’s to the ZFS raidz-1 over a 10GbE . I know it’s not bad … and it’s an initial transfer … but just to understand and learn better on how ZFS works.
