I have a pool consisting of a single HDD and an L2ARC (metadata only) SSD. This is deliberate
The SSD is ancient and I would like to take the opportunity to replace it. As its metadata only there should be very little space used on the disk.
The disk however is 280GB where the designated replacement is 100GB. This means that I cannot just replace the disk ( I am assuming, I haven’t actually tried).
I could just remove the L2ARC vdev and then attach a new one - but I would lose the metadata thats been cached already - I do not want to lose that data on anything other than a very temporary basis
Given that there are several TB of data on the HDD, in lots of folders - what is the best way of regenerating the cache if I replace the drive. Is a simple ls -R enough or would more be required?
Also (idle question) would the same process (if there is one) also work for a proper metadata vdev?
Some options that I have found are:
find . > /dev/null
ls -R > /dev/null
Of course its entirely possible that due to 256GB RAM that the metadata isn’t even reaching the L2ARC which would make it pointless, in which case a medata svdev would work, but make the pool more vulnerable than it already is
For the record I am trying to reduce the IO Load on the single HDD by “caching” metadata on an SSD. Looking at zpool iostat “Pool” -v -n 1 the cache rarely gets touched (R or W) so its probably doing nothing
I’d remove and replace the SSD then crawl the entirety of the HDD three times with rsync or some similar application. If the new L2ARC is persistent and metadata=only, that should do it.
My own advice - now redundant - would have been that IF you are replacing the L2ARC then to remove the existing L2ARC and add the new and take the hit on performance for a bit. (I am assuming that you have a VERY unusual app where you would actually notice the performance hit.)
But equally, if your actual data is residing on a single HDD, then why not replace that with SSD(s) and get even faster performance without an L2ARC.
All that said, as a TrueNAS MVP I am surprised that you are asking this here - but I also applaud you for doing so before attempting the swap because you were unsure about it despite your experience.
The App is Storj. Replacing the data storage with SSD would be nice, but fundamentally unaffordable.
I think the major reason for asking is my wondering if the metadata cache is actually doing anything significant due to the aforementioned 256GB ARC. If the metadata cache is populated the same way as L2ARC normally is (and I cannot believe that there are two different mechanisms) - then I have a suspicion that this SSD is achieving very little indeed
Looking at the disk IO over the last few days - there are bursts of some activity, but mostly it hovers around zero. Average reads over the last week is 22.4KiB/s - which isn’t exactly a lot
Unless you are on latest ZFS (EE?) L2ARC is repopulated on reboot anyway.
Not sure what Storj does or how it works, but unless response times are critical and unless you are doing a LOT of access to small files, IMO you are not likely to notice the metadata accesses from HDD (because those that are not in ARC will be dwarfed by the data reads).
In other words, personally I doubt that L2ARC will be providing you with any useful performance benefit.
Response times are important
And there are a lot of small files / any of which can be accessed at any time (at least in theory). There is continuous activity
Oh, my earlier comment (I just re-read it) the looking at DiskIO - that was for the metadata cache SSD, not the data on the HDD
I’d suggest replacing the HDD with a SSD then? FWIW, I have decided to recycle my L2ARC SSD (1TB mSATA) to be the host for my apps, if I ever get around to installing some.
If your main pool has a sVDEV, migrating the Storj dataset there. A sVDEV really makes small files fly and the small file cutoff gives you a lot of flexibility re: where to set that limit.
I think that is backwards. You have a system, you start with no l2arc or anything else special. You have no performance issues, you do nothing. You have issues, you look into it and then decide do I need a special device of some sort. Starting with it is not the way to do it IMHO.
OK, so you get paid for storing other people’s stuff… why is fast access a priority then? is there a lot of activity going in and out of this drive? Or is it mostly WORM?
If redundancy is not a requirement, why not have some fun and make the HDD a VDEV and a reasonably OK SSD of your choice as it’s sVDEV in a StorJ pool? Now get the benefit of fast small file AND fast metadata storage.
Kind of related to this, is a metadata only special vdev a workable solution to long term capture of ‘arc’ if the machine is shut down regularly? (thereby losing the arc content but not having enough uptime each boot to usefully populate arc again).
A sVDEV does not capture ARC. It stores all metadata on the sVDEV disks and if they are quick, it will help repopulate the ARC quickly as well. However, even SSDs are typically much slower than DRAM/ARC.
If you want the ARC to keep tracks of lots of files, I suggest making the ARC as big as possible (thereby limiting evictions) and limiting the instances of the machine being shut down. In practice, I doubt ARC being ‘preserved’ has all that much impact except in specific use cases.
The other way that the sVDEV speeds transactions is taking on small files, which HDDs struggle with. By default, the sVDEV is 75% allocated to small files, 25% to metadata. But you will want to tune your VDEVs and small-file cutoffs by use case and application. For example, you could force your sVDEV to host all VM / app datasets by aligning dataset recordsize and sVDEV small file cutoff. See the sVDEV resource for more info.
I think it’s worth pointing out that a “metadata only L2ARC” is something very different from a “metadata only special vdev.” Probably the most significant difference is that latter is critical to the pool, such that if/when it fails, the whole pool fails; while L2ARC isn’t critical.
My case would be the addition of a single SSD as a nice to have performance improvement in walking the tree after a restart, rather than introduce a proper svdev which would need at least 3 drives and a whole new mounting mechanism and expander. I’m asking really because i’ve never heard the expression “metadata only l2arc”
Not to derail this thread, but you would be better served if you increase your RAM and/or apply a tunable (depending on your version of TrueNAS) to improve metadata performance (crawls, browsing, directory listings, rsync etc).
At least try it first before you introduce an L2ARC, which might not even improve performance (and could even interfere with ideal ARC usage).
There’s no true “metadata-only L2ARC”. When people say that, they’re referring to setting a dataset’s property to secondarycache=metadata with the presence of an L2ARC in your pool.
For what it’s worth, the property primarycache is often overlooked, but can be quite useful in certain circumstances. For my torrent “seeding” dataset, I set it to primarycache=metadata to prevent it from pointlessly attempting hold any blocks in the ARC, which theoretically lessens the overall pressure on the ARC, in favor of much more important data and purposes.
