How to recover from rm -rf /* ? (lost everything)

Okay, now we can spam the :confetti_ball: emotes.

6 Likes

You guys did an amazing job. I can’t thank you enough, especially you @HoneyBadger. I’m sooo grateful !

Let’s celebrate by watching the end credits of this video. :wink:

1 Like

@winnielinnie You also deserve an award ! Thanks a lot for your support :pray:

I still have a lot to do :

  • It seems that the SSD boot-drive needs a firmware update
  • I will need to replace the drive with bad sectors
  • Run the script to test the new drive I bought

BUT first of all set some Snapshots… :sweat_smile:

Consider a dedicated backup (besides the emergency one you made) as a crucial safeguard, just in case.

If you’re not up for automation, make a checkpoint for your pool prior to doing anything that involves scripts and commands that could even maybe introduce risk.

Discarding a pool checkpoint is just as easy as creating one. It’s almost zero cost and minimal inconvenience, but imagine the payoff of being able to rewind your pool without resorting to the study of ZFS partitions and emergency operations.

EDIT: To be clear, snapshots are way more versatile and useful than pool checkpoints in everyday use. But a checkpoint serves a vital purpose for those times where you never want to find yourself in a situation that requires rewinding a pool… but where you have no choice.

2 Likes

I think that this is lesson I will not forget.

I wanted to make a checkpoint right away since it easy and may be very helpful.

root@truenas[~]# zpool checkpoint TankPrincipal
cannot checkpoint 'TankPrincipal': operation not supported on this type of pool
root@truenas[~]#

I don’t know if will continue posting here or creating new posts regarding the issues I may encounter during the steps I’ll need to make.

How old is your pool? :flushed:


You can enable single features at a time (such as this one), or if you don’t believe you’ll ever need to import your pool into a system with an older version of ZFS, you can do a full pool upgrade.

5 years maybe…and it has always been set as it is now. I have never messed with these settings…

It’s up to you.

If you only want to enable a single feature you can do this:

zpool set feature@zpool_checkpoint=enabled TankPrincipal

If you want to go ahead and upgrade your pool to all the latest features, you can do this:

zpool upgrade TankPrincipal

:warning: Upgrading a pool is a one-way operation.


I’m not too sure if enabling only a single feature that is newer than older (yet still disabled) features has any unforeseen implications.

I think I’ll set this one only for now. I’ll take time later to check and understand all the consequences for a zpool upgrade…

So ? Do I need to wait or go for it ?

…so you should make a checkpoint beforehand.

Oh wait… :stuck_out_tongue:

4 Likes

@etorix has discovered the first ZFS paradox. :infinity:

2 Likes

If it was my pool, I would try to enable it. It’s a critical feature that’s been around since 2018, I believe.

For the record, I’m on TrueNAS Core 13.3, which uses OpenZFS 2.2.4. I’ve recently upgraded all of my pools (except the boot-pools, of course), since every server/PC I use is up-to-date and has the latest ZFS.

I highly doubt I’ll ever find myself in a situation where I cannot import one of my pools because the software/system is too old for it.

1 Like

I trusted you and went for it :

root@truenas[~]# zpool set feature@zpool_checkpoint=enabled TankPrincipal
root@truenas[~]# zpool checkpoint TankPrincipal
root@truenas[~]#

To cofirm:

zpool status TankPrincipal | grep checkpoint
root@truenas[~]# zpool status TankPrincipal | grep checkpoint
checkpoint: created Sun Dec  1 12:22:37 2024, consumes 1.95M
root@truenas[~]#

You’re going to hate me for saying this, but you might want to discard the checkpoint, and then only create one before you ā€œdo somethingā€.[1]

Later on, you can explore creating a script that runs maybe once every few days, which discards and creates a new checkpoint. In a sense, it’s like a checkpoint follows closely behind you at all times.

If you trust your intuition, you can simply rely on doing this manually (ā€œcreate then discardā€) right before and shortly after doing something you’re nervous about.


  1. The catch is that you might neglect to create a checkpoint manually if you don’t believe what you’re about to do has some notable risk. Consider what got you here: Did you really think that ā€œpreparing Immichā€ seemed at all dangerous? If you had known about checkpoints and snapshots, would you have paused to create one first, and then proceed in the command-line? ā†©ļøŽ

1 Like

Checkpoints may be useful not only against human mistakes but also hardware fails or something right ?

So why wait ? Maybe I could set that safe net now ?

This is where a checkpoint differs from snapshots and backups. They serve a very narrow purpose.

I try to explain it in this thread.

You really don’t want to sit on a checkpoint for more than a few days or so. Otherwise, it loses its usefulness.

On the other hand, you don’t want to neglect creating one temporarily, so that you have a ā€œrecentā€ safety net in case things go sour shortly after.


Think of it like this. Let’s say you created a checkpoint back in July. That’s about 6 months ago. How useful is this safety net from 6 months ago?

If you rewind your pool back to July, you will lose everything after July.

If you had created a checkpoint on November 17, then you’d only be rewinding back for 1 day. A safety net from 1 day ago is much preferable to one from 6 months ago.

The reason (theoretically) why ā€œautomatingā€ this to follow closely behind you by about a few days, is to provide an ā€œalways recentā€ safety net at all times, for those of us who are prone to neglecting or forgetting to use this feature when it matters the most.