How to recover from rm -rf /* ? (lost everything)

TrueNAS General
,
1

Hi There,

First of all, I had no snapshots, no backups.

I was trying go get Immich installed using the help of ChatGPT. While following the some command line I was asked to remove an environment variable :

root@Borealis[/mnt/TankPrincipal/Immich]# rm -rf ${DB_DATA_LOCATION}/*

Bash ignored the variable and started to delete everything :

root@Borealis[/mnt/TankPrincipal/Immich]# rm -rf ${DB_DATA_LOCATION}/*

zsh: sure you want to delete all 27 files in / [yn]? y
rm: cannot remove '/audit': Read-only file system
rm: cannot remove '/bin': Read-only file system
rm: cannot remove '/boot/initramfs_config.json': Read-only file system
rm: cannot remove '/boot/efi': Read-only file system
rm: cannot remove '/boot/vmlinuz-6.6.44-debug+truenas': Read-only file system
rm: cannot remove '/boot/config-6.6.44-production+truenas': Read-only file system
rm: cannot remove '/boot/initrd.img-6.6.44-production+truenas': Read-only file system
rm: cannot remove '/boot/System.map-6.6.44-debug+truenas': Read-only file system
rm: cannot remove '/boot/grub': Read-only file system
rm: cannot remove '/boot/initrd.img-6.6.44-debug+truenas': Read-only file system
rm: cannot remove '/boot/config-6.6.44-debug+truenas': Read-only file system
rm: cannot remove '/boot/System.map-6.6.44-production+truenas': Read-only file system
rm: cannot remove '/boot/vmlinuz-6.6.44-production+truenas': Read-only file system
rm: cannot remove '/conf/truenas_root_ds.json': Read-only file system
rm: cannot remove '/conf/base/etc/syslog-ng/syslog-ng.conf': Read-only file system
rm: cannot remove '/conf/base/etc/aliases': Read-only file system
rm: cannot remove '/conf/base/etc/ssl/openssl.cnf': Read-only file system
rm: cannot remove '/conf/base/etc/group': Read-only file system
rm: cannot remove '/conf/base/etc/passwd': Read-only file system
rm: cannot remove '/conf/rootfs.mtree': Read-only file system
rm: cannot remove '/data': Read-only file system
rm: cannot remove '/dev/pts/1': Operation not permitted
rm: cannot remove '/dev/pts/0': Operation not permitted
rm: cannot remove '/dev/pts/ptmx': Operation not permitted
rm: cannot remove '/dev/shm': Device or resource busy
rm: cannot remove '/dev/hugepages': Device or resource busy
rm: cannot remove '/dev/mqueue': Device or resource busy
rm: cannot remove '/etc/exports.d': Operation not permitted

rm: cannot remove '/home': Read-only file system
rm: cannot remove '/initrd.img': Read-only file system
rm: cannot remove '/initrd.img.old': Read-only file system
rm: cannot remove '/lib': Read-only file system
rm: cannot remove '/lib64': Read-only file system
rm: cannot remove '/media': Read-only file system
rm: cannot remove '/mnt/MainTank/Documents/Documents_Pro': Device or resource busy
rm: cannot remove '/mnt/MainTank/Documents/Documents_Familly': Device or resource busy
rm: cannot remove '/mnt/MainTank/Documents/Documents_Perso': Device or resource busy
rm: cannot remove '/mnt/MainTank/VMs': Device or resource busy
^C
root@Borealis[/mnt/MainTank/Immich]# echo $DB_DATA_LOCATION


root@Borealis[/mnt/MainTank/Immich]# cd /mnt/MainTank
root@Borealis[/mnt/MainTank]# ls
Bilalzz  Documents  ISOs  Immich  Media  Photos  Sauvegardes  Telechargements  VMs  Videos
root@Borealis[/mnt/MainTank]# cd Documents
root@Borealis[/mnt/MainTank/Documents]# ls
Documents_Familly  Documents_Perso  Documents_Pro
root@Borealis[/mnt/MainTank/Documents]# ls Documents_Pro
root@Borealis[/mnt/MainTant/Documents]# ls Documents_Perso
root@Borealis[/mnt/MainTank/Documents]# ls Downloads
ls: cannot access 'Downloads': No such file or directory
root@Borealis[/mnt/MainTank/Documents]# cd Media
cd: no such file or directory: Media
root@Borealis[/mnt/MainTank/Documents]# cd Berboo
cd: no such file or directory: Berboo
root@Borealis[/mnt/MainTank/Documents]#

Then I shut down the server.

As you can see, I had years of documents and pictures.

I know, I should have been more carefull backuping my files, I know I should have been more carefull stupidly.

If I loose everything, I will not die, but imagine having some precious videos of your parents (that are getting old) lost ? I have tears in my eyes as I’m writing this.

Please don’t moralize me but give some solutions, if there are any, to get back my files.

I had 3 Hard drive of 4 TB each, in a RAIDZ1 Pool, the system is (was ?) In a SSD hard drive

I have some basic command line knowledge, I would pay someone if someting could be done

Thanks a lot

2

Sounds about right…

First question, did you create any snapshots or snapshot tasks? Did you ever create a pool checkpoint?

EDIT: I missed that part in your post.

3

I have used this in the past which have worked flawlessly: Klennet ZFS Recovery

6 Likes
4

Thanks for the suggestion

I guess I will have to do a fresh start of TrueNAS.

I’m scared that if I do a fresh install of TrueNAS, it may write some data on the Pool and write over the deleted data.

How to do this the safest way ?

5

Do not reinstall TrueNAS, do not do anything further to the drives. Power off the system and disconnect them immediately.

Use Klennet ZFS Recovery from a separate system.

6 Likes
6

Nop sorry I did not.

In fact I thought that as long as I don’t have any damaged clusters or some S.M.A.R.T problems I will be good…

7

If you feel comfortable, can you at least rule this out?

There’s a possibility you (or a script, or another trigger) created a snapshot.

zpool get checkpoint MainTank

zfs list -r -t snap MainTank

As stated by @HoneyBadger, you want to minimize any “use” of this pool. Ideally: zero use at all, until you can recover the data.

1 Like
8

Since I don’t have any other system, I could use that 60 GB SSD to install an appropriate distro (thus removing TrueNAS for now) and then plug back the drives to try the Klennet solution ?

9

Klennet only works on Windows, jfyi. So yes, you can do that.
Be sure to physically disconnect the data disks before installing another OS.

2 Likes
10

Ok thank you very much guys. I’ll try this in the next few days and come back to you with the outcome

11

Bash did not ignore anything.
You never assigned a value to the variable, right?

12

It’s in the posted log of the mishap, echoing the variable showed it wasn’t assigned any value.

13

Right, it looks more like chatgpt hallucinated its way into telling OP to basically delete everything.

3 Likes
14

I’m just glad the OP didn’t post this on April 1.

15

To give you guys more context :

  • I was trying to install immich-via docker-compose.yml, the official one.
  • I set a separate .env file with the entries.

Logs showed that the installed version of postgres was newer that the one it was wanting to install (version 15 for the installed one 14 for the one one the docker-compose.yml file.

  • So I set the new link in the docker-compose file but it didn’t do any good. Same error :
2024-11-18 17:12:33.625 UTC [1] FATAL:  database files are incompatible with server
2024-11-18 17:12:33.625 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 15, which is not compatible with this version 14.10 (Debian 14.10-1.pgdg120+1).

PostgreSQL Database directory appears to contain a database; Skipping initialization

2024-11-18 17:13:34.197 UTC [1] FATAL:  database files are incompatible with server
2024-11-18 17:13:34.197 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 15, which is not compatible with this version 14.10 (Debian 14.10-1.pgdg120+1).
root@Borealis[~]#

Then ChatGPT said wrote :

Your docker-compose.yml file seems correctly configured to use PostgreSQL 15. However, the issue likely persists due to a conflict with existing data in the PostgreSQL data directory (${DB_DATA_LOCATION}). This data was initialized with PostgreSQL 14 or another incompatible version.

Steps to Resolve the Issue

  1. Check Database Compatibility If you want to use the existing data, ensure it was created with PostgreSQL 15. Data created with another version will not be compatible.
  2. Reset the PostgreSQL Data Directory If you don’t need to keep the existing data:
  • Stop the containers:
docker compose down
  • Remove the PostgreSQL data directory:
rm -rf ${DB_DATA_LOCATION}/*

Replace ${DB_DATA_LOCATION} with the actual path specified in your .env file.

  • Restart the containers:
docker compose up -d

It may sounds funny for some of you but ChatGPT help me setting other things in a pretty good way.

I know that I should avec took some precautions but it the way it is. I should have at leat tried in a VM on TrueNAS before installing it on the host.

By the way I read that Klennet in the free version doesn’t copy the files but shows you what you can recover.

I assume that the files it will recover needs to be copied on a another drive ? those 60GB will not be enough. Am I assuming right ?

16

Now I see the line I missed…

Replace ${DB_DATA_LOCATION} with the actual path specified in your .env file.

17

I’ve never done it, but it is possible to import a pool manually at a specified TXG

https://openzfs.github.io/openzfs-docs/man/master/8/zpool-import.8.html#T

I’m guessing the process would be to list the candidate TXGs, then repeatedly mount the pool read-only at progressively further back TxGs until you predate your ChatGPT experience.

And this recipe seems to be it

3 Likes
18

Both correct. Klennet will tell you for free what it can recover if you pay the $399 licence. Fair enough, I’d say. And you’ll need enough space for everything.

If you’re into the forensics, there were three steps into this disaster.

One (no snapshots is a very bad mistake: These are basically free!)

Two (at the very least not vetting the information; but I’d say “trusting AI” was the mistake in and by itself)

And… three

1 Like
19

And remember, if you set up snapshots then you can recover from mistakes like this

3 Likes
20

“Hey, I know that guy.”

Attempting to fish out the previous txg and importing the pool directly at that point with -T may be a potential fix here if there’s still a valid uberblock that it can enter at, but if they’ve all rotated through (IIRC there are only 32 of them) then there may not be a consistent entry point where ZFS can checksum.

3 Likes