I caused apps to fail by manually deleting old snapshots from an app that I removed and reinstalled. Turns out k3s is not very robust.
To restore apps to a working state, I stopped all apps:
Then rolled back these datasets from ix-applications: catalogs, default_volumes, kubelet, and k3s datasets.
Then I rebooted and magic, everything works.
Also, you may find you’ll need to delete some pods in TaintToleration mode. So if you rebooted 30 minutes ago, you’ll see these pods are older than your reboot time.
Here’s an example:
sudo k3s kubectl get pods -A
sudo k3s kubectl delete pod tailscale-6f65f65d6b-br4s6 -n ix-tailscale
If you don’t do this, when you look at the logs or try to access the shell of the pods for an app, it will prompt you with the TaintToleration pods instead of the pod(s) you just started.
You may be able to get away with rolling back less.
Moral: Before doing any manual cleanup of k3s files, do a recursive snapshot of ix-applications.