Hello,
I was getting an alert from TrueNAS that I had more than 10k snapshots, so I went to investigate.
Using the command line (because the UI for snapshot management is really slow, more on that later), I deleted some unneeded snapshots, as well as I found some datasets that were left over from a migration of my Apps pool to another drive so destroying those datasets also destroyed the related snapshots. I am now under 7k. Good, alert addressed.
But I still have a few questions.
I have read some documentation, as well as a few threads about snapshots for example this one: Too many snapshots being detected, how to reduce?
I think my snapshot schedule is sensible, and I understand a large number of snapshots is not a problem in itself. I donāt think Iām using too much space with my snapshots (though I havenāt checked, but my TrueNAS device is a home NAS and there is not much churn).
My questions are:
- If 10k+ snapshots are not really a problem, should I increase the warning threshold? Where is that configured?
- Whatās the calculation of how many snapshots will be āliveā at any one time given a schedule like the above? It would be really useful if the UI showed that information, and Iām sure itās pretty easy to calculate given the datasets the system knows exist and the configured schedule.
- I understand datasets are created for every appās mounts. I have 21 apps running on TrueNAS at the moment, and Iām sure others have a lot more. Some of these have multiple mounts (config, data, pg_data, ā¦). That leads to a lot of snapshots, doesnāt it? In that kind of situation, the 10k threshold for the alert seems small. And I donāt want to make my snapshot tasks non-recursive because I want the data protection and donāt want to micro-manage my app snapshots.
- Even at less than 10k snapshots (now 7k) the snapshots list UI in TrueNAS takes forever to show the snapshots in the list. It shows the first 50 on screen after like a minute of spinner time, but then clicking the next page / last page buttons is near instant. This indicates to me that itās actually listing ALL snapshots and gathering data for them, even though it only shows the first 50. This seems to have been reported a lot on these forums and the answer is generally āyou have too many snapshotsā or āwait longerā or āuse the command lineā, but IMHO since the UI is only showing 50 there is not that much data to load and it should be near instantaneous, no matter how many you have total. This seems like an area that should be improved to make managing snapshots less frustrating.
Thanks in advance.
Some of this is dictated by the internal design of libzfs itself (potentially walking entire list of snapshots prior to yielding to consumer in order to generate a sorted list of them), some of it by python library design in older versions of TrueNAS (due to thread-safety problems we were having to throw requests into a worker process pool that could get overloaded by UI ops in some edge cases). In 26 weāre transitioning to a library thatās inherently thread-safe (which resolves the worker process pool issue). You can see if itās better there for your use-case.
There are maybe some additional changes we could do to improve the situation, but I donāt think weāll remove the alert because itās legitimately pointing out that youāre approach UX rough edges.
Thanks for the reply! Iām a programmer myself so I can definitely appreciate when there are limitations based on the third-party dependencies chosen or other historical reasons. I look forward to trying out the new version to see for myself.
Even if the alert cannot be removed, can the threshold be configured? I was surprised to get to 10k snapshots in just a few months of home use, I bet Iām not the only one (unless my snapshot schedule is very uncommon or 21 apps is way more than most people use).
The alert for snapshot counts can be disabled. libzfs iterators are callback-driven which ends up being a not-great fit for the traffic patterns API consumers can drive (especially if you have deep nesting of datasets). 10K+ snapshots is unusual and maybe indicates a too-aggressive snapshot policy IMO, but I focus more in conventional NAS space.
1 Like
Itās worth mentioning that if you can live without the āShow extra columnsā fields then its much snappier.
My busiest production server has just over 7000 snapshots and it is slow to display them (if Iām showing extra columns but near instant if not). Thats based on about 130 datasets using the below snapshot schedule.
Every hour for a day
Every day for 2 weeks
Every week for a month
Every month for a year
What I was mentioning is without āShow extra columnsā enabled.
I made a quick program to calculate (approximately) how many snapshots will exist maximum with a given schedule. (my calculations are approximate because I used integers and donāt really care if Iām one or two off because things donāt divide evenly)
Here are the results with my own schedule with my 76 datasets:
Every10minutes (every 10 Minute / retain for 2 Hour): 12 snapshots
Hourly (every 1 Hour / retain for 2 Day): 48 snapshots
Daily (every 1 Day / retain for 2 Week): 14 snapshots
Weekly (every 1 Week / retain for 2 Month): 8 snapshots
Monthly (every 1 Month / retain for 2 Year): 24 snapshots
Maximum number of snapshots per dataset: 106
Maximum number of snapshots for 76 datasets: 8056
So thatās why I say, the threshold of 10k datasets for the warning is not really that hard to reach! Those 76 datasets are mostly from apps, I only have 2 datasets that are my own NAS storage datasets.
BTW hereās the result of my program for your schedule and 130 datasets:
Hourly (every 1 Hour / retain for 1 Day): 24 snapshots
Daily (every 1 Day / retain for 2 Week): 14 snapshots
Weekly (every 1 Week / retain for 1 Month): 4 snapshots
Monthly (every 1 Month / retain for 1 Year): 12 snapshots
Maximum number of snapshots per dataset: 54
Maximum number of snapshots for 130 datasets: 7020
Yep. In my backup of the backups server, backing up 2 other TN backup servers, plus 2 TN app servers, just saw the warning as it has around 17,000 snapshots from all those systems.
If the possible performance issues are only when listing snapshots, I donāt care.
If it creates access/transfer issues, I want to know.
A more detailed warning would be nice (like: too many snapshots can result in long snapshot listing times etc.).
