On my main pool, zpdb, zfs list -t snapshot zpdb gives me 318 lines of output; not, I feel, an excessive number of snapshots. (There are probably more down at the dataset level though.)
In the GUI, local URl /ui/datasets/snapshots (or reached by Data Protection → Snapshots), though, it’s just spinning for ages. (Also twice, now, it’s shut down power to the server before completing; the first time I assumed it was just random coincidence, but a second time worries me. So, I’ll bring the system up and see if it shuts down without digging into the snapshots, I guess.) “Ages” exceeds 5 minutes, but I think the previous time exceeded 15 minutes. And if I wasn’t doing anything else on the server, no ongoing disk activity showing in the drive lights.
Regardless of what’s going on with the shutdown, it shouldn’t take that long to display a few hundred snapshots in the GUI!
(Oh, and no notifications about anything after bringing it back up after one of the weird shutdowns.)
few hundred at the top dataset can still balloon once you recurse. check the real count:
zfs list -t snapshot -r zpdb | wc -l
the Data Protection → Snapshots page has been slow for a lot of people once you get into the thousands. middleware walks them for the table and the browser chokes.
for day to day i just use the shell list / destroy. if the page keeps spinning past a few minutes, close that tab rather than leave it grinding.
the random power offs while that page is open are worth treating separate. check UPS / System → Advanced power settings and dmesg around the shutdown times. two in a row is sketchy.
I never used to have problems with this level of snapshots, back in the Solaris days, but I’ve noticed TrueNAS tends to complain. Being able to do an hourly snapshot and keep it for a couple of weeks is one of the core benefits of running a NAS!
(Actually, rather more; I’ve cut a lot of things way back trying to get it down, including taking days off the retention time.)
I guess I should just give up on using the GUI for examining or pruning snapshots – and perhaps find a way to reduce it.
It’s highly recommended to have separate datasets rather than sharing them. Taking periodic automated snapshots is very useful, and I think common. Between these two, the number of snapshots does grow. I only have about 35 datasets, according to zfs list -r zpdb. And almost half of them are under iocage, not really mine at all.
Am I taking an entirely wrong approach here? I’m not really happy with the idea of just giving up on snapshots as something I count on. Is doing one automated snapshot at the top, recursively, somehow the wrong approach?
No such page? The UI search in the GUI finds nothing for POWER other than the stuff in the toolbar? UPS is there, and active, but no log or anything, just the config for UPS monitoring.
And I found dmesg -T to make more sense of the timeline there, which I hadn’t found before, so that’s useful in general!
Haven’t so far found any log entries saved from the shutdowns that look interesting, but that’s a lot of log to examine, I’m still looking.
Looking at those numbers again – recursive snapshots shouldn’t multiply the numbers by more than the number of datasets! So 318 top-level snapshots times 35 datasets is only 11k. So I’ve got 20k I don’t know where they came from. That might be a productive avenue of investigation too.
31k snapshots is enough to make that page awful by itself. Since there are no apps or VMs, I’d test the API and ZFS directly:
zfs list -t snapshot -o name | wc -l
zpool iostat -v 1
Run the first once, then watch pool I/O while opening the page. If the page is still slow with no disk activity, that points at the middleware/API query rather than the container.
For cleanup, use the retention on the periodic snapshot tasks and prune old snapshots from the shell after checking the dataset names. Don’t bulk-destroy with a guessed pattern. Keep a recent rollback point, then see if the page speed changes.
you’re cheating and passing -o name . Libzfs does different fetch using a “simple” zfs handle when you do that. IIRC 26 should be faster because we’re eliminating the process pool in favor of multithreading with our new libzfs python library. It won’t be hugely faster if you’re retrieving full set of 31 K snapshots recursively. That’s a lot of info that has to be parsed through. Using API is faster if you don’t request full data on snapshots (limit to name and createtxg for sorting for instance). People usually want lots of info in the UI, but that also massively pessimizes performance. You can’t have your cake and eat it (snapshot properties and speed).
Even getting all the default fields doesn’t slow it down that much; not into territory that seems problematic. Just adding one random field hardly changed it at all.
ddb@fsfs:~$ time zfs list -t snapshot -r zpdb -o name,createtxg | wc
31545 63090 4037760
real 0m4.147s
user 0m0.576s
sys 0m3.606s
ddb@fsfs:~$ time zfs list -t snapshot -r zpdb | wc
31545 157725 4447854
real 0m16.827s
user 0m2.134s
sys 0m14.726s
Knowing the performance problem is in the GUI, I’ll just avoid using it for snapshot stuff for now. This is kind of a side issue, but I was reminded of it again when the “obvious” thing took apparently forever.
Found one batch of auto snapshots that didn’t get deleted at their expiration, but that only accounted for 5000 total (recursive zfs list). Still, a little bit will help a little.
I’ve looked through the other snapshots and none of them are “mysterious”. So they somehow add up, still, to 25k somehow. I’ll shelve this and get back to it – and remember not to use the GUI to look at snapshots. There aren’t performance problems with ZFS itself even with 30k snaps.
Ah, I’ve been wondering what happened in that situation; a bit afraid to try it, and I see I was right.
I did dig up the rest of my extra snapshots, and they seem to be the same thing; leftovers from the old snapshotting practice before I got to SCALE. It’s deleting 16k or so now (at least it doesn’t seem to use a global lock; I’ve got 4 processes doing it and get better throughput). (Some of the system datasets have more layers of dataset under them then I had thought, which takes the multiplier up.)
. Libzfs does different fetch using a “simple” zfs handle when you do that. IIRC 26 should be faster because we’re eliminating the process pool in favor of multithreading with our new libzfs python library. It won’t be hugely faster if you’re retrieving full set of 31 K snapshots recursively. That’s a lot of info that has to be parsed through. Using API is faster if you don’t request full data on snapshots (limit to name and createtxg for sorting for instance). People usually want lots of info in the UI, but that also massively pessimizes performance. You can’t have your cake and eat it (snapshot properties and speed).