Deleted files from an almost full pool and now can no longer mount

TrueNAS General
1

Hi,

I’ve had my TrueNAS system running for years and it’s been solid - I’m running a R510 with various HDDs in a zpool with multiple mirrored VDEVs (unfortunately I can’t get access to the GUI to give you the full stats)

I think the pool is around 24TB big, with 16GB of RAM (which might be part of the issue).

So, I foolishly had a veeam backup that was running and over time slowly started to fill my drive and I noticed it was 97% full. I of course, wide eyed, quickly started deleting backups, as I decided they one of my backups was unnecessary as I had a duplicate backup anyway - long story short I started to delete the backups. It went okay at first, getting down to about 94% full, but then a bit of time into it the NAS froze (became unresponsive to both SSH and GUI) - thinking it was due to a du command I was running which got stuck I force rebooted the server.

It came back and all was fine. A few hours later, I (now realise foolishly) re-ran the delete from veeam. Again about 20-30 mins in the server freezes again, and I realise the delete is the issue. I do another hard reset as that worked last time, but this time the server doesn’t come back up.

I connect a monitor to the server, and I get a bunch of errors “failed to reclaim memory” when it tries to mount the pool.

So presumably it’s trying to do something with the pool and it’s running out of memory doing it. I do not have dedup on, it’s a fairly standard pool I think.

I did try rebooting into single user mode and importing the pool readonly, but I was getting issues around it complaining that I’m trying to mount something into a readonly file system, and I didn’t want to mess around with it too much and risk running something wrong and losing my zpool.

So, the question is, how do I proceed from here?

I can add more HDDs to expand the pool and increase the storage, but I don’t know how I can do that if I can’t even mount the pool. I can buy more RAM, but will this fix the issue?

Ideally I can only do one of those two at a time due to money constraints.

I do have backups of the most important data, but I really would like to save the pool if I can.

Yes I know i’m stupid for letting the pool get too used, but how can I proceed from here?

Thanks in advance, I’m desperate for some advice and help

2

Here is a picture of the screen when it hangs

20240905_010911
20240905_0109111920×1493 345 KB

3

Some update - been following this guide (ZFS Has Failed you | TrueNAS Community) and I’ve managed to finally mount the pool as readonly in single user mode - which is great.

So where do I go from here?

zpool import -R /mnt -FX poolname

sounds like a risky command, so I’m reluctant to run it at the moment until I know more

4

I’m thinking the best thing to do is to buy more RAM (as i’m on 16gb which is a bit low for the amount of TB I have) - I can max out at 64gb RAM for about £70 - and then see if that gets the pool mounting again. If so, I can get a couple of HDDs and add a vdev and expand the pool to get it healthy again - does this seem like a good plan?

5

Anyone? Would love some feedback to make sure I’m doing the logical thing here

6

The plan sounds logical.
First more RAM—it never hurts.
Then an extra vdev, which you can still remove later, if lack of free space is an issue. (CoW needs space to reclaim space…)
As a last resort, try the possibly destructive manual import commands. But do a dry run with -n before issuing the real thing.

7

This can only work with mirror vdevs. So, @brettjenkins, stick with mirrors if this is something you might consider in the future.

8

Hi both,

Thanks a lot, weight off my mind! Got the RAM coming tomorrow, i’ll be maxing it out to 64GB (from 16GB), so hopefully that will be enough RAM to get it to mount - and then I’ve got another hard drive on order (I actually already have a spare HDD from a couple of months ago due to an accidental order, long story) so I can add a new Mirrored VDEV assuming it all mounts successfully. Fingers crossed!

9

For anyone following this thread and for future folks, adding the new RAM (64GB), booting into single user mode, and doing a normal mount, took about 10minutes or so and the hard drives were churning, but it eventually mounted.

Rebooted back into normal TrueNAS, and everything is back up and running! It did manage to delete some stuff, as it’s down to 89% used, I’ll limp along until Monday when the new HDD comes then I can form another mirrored VDEV, and make it much more stable. Won’t be doing any more deleting or storing of anything big until then!

Just want to thank @etorix and @winnielinnie for helping, really helped calm down the blind panic you get when dealing with mounting issues!

1 Like