A timeline:
Saturday I decided to update my TrueNAS to 24.10 and it went very smoothly. All files and services were accessible. Everything was working as before.
Sunday morning I was able to access all files and services- everything was fine.
Sunday night my system takes a snapshot.
Monday afternoon ALL files gone and deleted. Very odd… decided to Rollback with the latest snapshot from Sunday. An error message blinks and the snapshot disappears. Attempted rollback from before the update- no luck.
Checked most of the logs- no obvious errors, no indication of a process kicking in and deleting.
I have TrueNAS running in a vm on Proxmox. I have made no hardware changes, and the monitoring services show no issues.
I am now in deep deleted file recovery mode and wondering a few things:
- What kind of incident in the logs should I have looked for and maybe I did not look at the correct logs.
- I am now concerned that I just did something wrong after updating and the consequences are not felt until almost 24 hours later.
- I have learned a lesson about system backups (RAID is not a backup) but there has to be a more elegant way of recovering from this…
System stats:
TrueNAS 24.10
4 8TB discs in a single RAID1 pool one left for redundancy for a total of 21TB