You mentioned that it ran out of space. Is that still the case?
I would make sure there’s actually space first before attempting any other troubleshooting.
In short no. Jail drive is now approx 20% usage.
So everything was lagging up and freezing and running really bad.
I logged into truenas gui and that took forever to load and login.
Then I had a few error alerts noticed the jail drive was at 100% use.
I then couldn’t stop any jails or do anything basically.
I rebooted the system, deleted a few media cache files from the plex jail freeing up a small amount of space.
With the few media files deleted it then let everything run normal sort of because none of the jails would load now even though they were started and up/running.
I then discovered the jail drive is full because of snapshots going back to like 2019 when trying to restore the jails from a snapshot to get them to work which didn’t.
I then deleted all snapshots prior to 2024 which put the jail drive usage down to like 20% from 100%…
I’ll have to make a note to monitor this each year to clear out the previous year.
Shout out to winnielinnie for the discovery of the rc.conf files being wiped for each of the jails that stopped them from loading. I was able to get 5 of 6 of my jails back online. The 6th i think I am going to have to remake and set up. I plan to setup some sort of backup system of the rc.conf file for each jail in case something like this ever happens again. I also purchased a 2tb drive I plan to migrate the jail drive to from the 1tb. Both these things I’ll have to research how to do on my next off day.
I am also heavily thinking about finally making the jump over to scale.
Might be a good time now that they’re finally migrating away from k3s… though generally, I usually try not to be an early adopter and wait for a couple of iterations for other people to work out the bugs first.