All data in a dataset just disappeared

TrueNAS General
1

I’m running
TrueNAS Core 13.3-U1. I have a dataset where all the data on it just disappeared. Like 12 TB of data. The GUI still shows that there is data in it:

image
image1641×448 44.1 KB

But when I check the directory size in a shell it clearly shows it empty.

root@Eru[/mnt/Pool_01/Media]#
root@Eru[/mnt/Pool_01/Media]# ll
total 1572
d---------+   10 root  wheel  uarch                  11 Dec 31 18:31 ./
drwxr-xr-x     8 root  wheel  uarch                   8 Dec 27 22:29 ../
----------+    1 root  wheel  uarch,hidden,system 18436 Dec 26 19:38 .DS_Store
d---------+    2 root  wheel  uarch                   2 Jan  1 22:14 Documentaries/
d---------+    2 root  wheel  uarch                   2 Dec 31 18:31 images/
d---------+ 5651 root  wheel  uarch                5652 Jan  2 08:41 Movies/
d---------+  489 root  wheel  uarch                 491 Jan  1 22:14 Music/
d---------+   15 root  wheel  uarch                  15 Dec 26 09:55 photos/
d---------+    3 root  wheel  uarch                   4 Dec 25 18:03 Streams/
d---------+   77 root  wheel  uarch                  78 Dec 31 18:54 Television/
d---------+    3 root  wheel  uarch                   3 Dec 22 16:02 template/
root@Eru[/mnt/Pool_01/Media]# du -sh Documentaries
 39K    Documentaries
root@Eru[/mnt/Pool_01/Media]# du -sh Movies
 15T    Movies

So obviously I’m in freakout mode right now. I would love some assistance in figuring out what the hell happened. What logs should I check? I have plenty of snapshots. Can I restore my data from that? I’m not expert in dealing with disaster recovery since I rarely deal with this situation.

image
image1640×786 61.1 KB

2

Which dataset? “Documentaries”?

zfs list -t filesystem -o space Pool_01/Media/Documentaries

Not trying to jump the gun, but did an application “use” this path and decide to “clear” it? I doubt you ran rm -rf yourself.

3

Yeah, I deleted a file from a share on my Macbook and something weird happened. I kind of dismissed it because the Mac shares are always acting weird. But then I looked at the dataset and everything was gone.

But other tests showed that there was actually data still in the dataset. Fortunately I take snapshots each night, so I was able to revert to a snapshot from last night and all the data became visible again. :grimacing:

It’s spooked the hell out of me for sure. I do have a backup server. But it is completely full and out of date. I might have to spend some money to upgrade it.

2 Likes
4

I demonstrate some ways to recover data using snapshots in this video, may be worth a watch :wink:

2 Likes
5

It’s not normal for Mac mounted shares to behave badly. Get to the bottom of that, ideally before upgrading.

For my NAS, it was a matter of ensuring that the SMB share presets were set to none (other than Time Machine, which use Time Machine presets) as well as enabling Apple character compatibility (but only because I also had that previously enabled in CORE before switching to SCALE).

Now, the SMB NAS experience is no different from AFP, except that SMB allows encryption, isn’t deprecated, etc.

1 Like
6

Thanks! Very informative.

7

It’s never been anything concerning honestly. It just has issues with the mounts sticking around. I always have to reconnect, so I assumed something like that was happening…until I thought about it more. But that’s all for convenience sake so I can manage my server while my ass is sitting on my couch. :rofl: My PC is my main access to my servers.