My stable TrueNAS Scale just died

[Phil242]

Hi,

While I was moving file from a share to another inside an app, my trueNAS died.

The disk where the apps are installed is at 100% but I’m unable to locate where the data filling the disk are! Funny situation … as I’m a n00b in TrueNAS Scale, I can’t imagine what’s happening. “df -h” doesn’t help me.

Now, Docker doesn’t start, Samba share are stopped, only NFS is sharing.
Web interface is a little bit weird, but usable.

I’m in version ElectricEel-24.10.1

I think the dataset where the system stock the apps, now hidden, made the mess, but impossible to find where those data filling the dataset are …

Any idea are welcome, please.

Phil

[neofusion]

Others know how to do this better than me, but maybe showing the output of
zfs list -o space would be a start.

Please paste the output using the </> button in the forum toolbar.

[Krill]

@Phil242

The disk where the apps are installed is at 100%

Is it really at 100%? That breaks things. Sorry, but I can’t help, except to note that if you were moving a file and the ZIL couldn’t be written you might be in the “Here be dragons” territory.

Special VDEV (sVDEV) Planning, Sizing, and Considerations

I believe the ZIL resides on SLOG first and data VDEVs second, so Very Bad Things would happen at 100% data VDEV capacity even if there is technically free space on the sVDEV. The SPA would throw the error trying to write into the ZIL first before it got a chance to write to sVDEV. Asynchronous writes that fit entirely into sVDEV might succeed in that scenario but the updates to meta/uberblock would probably still make it go “nope” on the final txg sync though.

[Phil242]

Hmm, ok, zfs is the start point.

So, here is the dataset where the apps are stored (I have removed the boot dataset and another with no space problem):

VM                                                          0B  1.93T        0B    140K             0B      1.93T
VM/.system                                                  0B  2.07G        0B   1.47G             0B       617M
VM/.system/configs-ae32c386e13840b2bf9c0083275e7941         0B  24.3M        0B   24.3M             0B         0B
VM/.system/cores                                            0B    96K        0B     96K             0B         0B
VM/.system/netdata-ae32c386e13840b2bf9c0083275e7941         0B   591M        0B    591M             0B         0B
VM/.system/nfs                                              0B   128K        0B    128K             0B         0B
VM/.system/rrd-ae32c386e13840b2bf9c0083275e7941             0B    96K        0B     96K             0B         0B
VM/.system/samba4                                           0B  1.46M     1.06M    404K             0B         0B
VM/.system/services                                         0B    96K        0B     96K             0B         0B
VM/.system/webui                                            0B    96K        0B     96K             0B         0B
VM/backup-tatm                                              0B   104K        0B    104K             0B         0B
VM/debian24-f3ii4m                                       72.0G  81.3G        0B   9.27G          72.0G         0B
VM/ix-apps                                                  0B  1.61T        0B    116K             0B      1.61T
VM/ix-apps/app_configs                                      0B  6.36M        0B   6.36M             0B         0B
VM/ix-apps/app_mounts                                       0B  1.61T        0B    120K             0B      1.61T
VM/ix-apps/app_mounts/adguard-home                          0B   406M        0B    104K             0B       406M
VM/ix-apps/app_mounts/adguard-home/conf                     0B   812K      704K    108K             0B         0B
VM/ix-apps/app_mounts/adguard-home/work                     0B   405M      377M   27.9M             0B         0B
VM/ix-apps/app_mounts/deluge                                0B   878G        0B    104K             0B       878G
VM/ix-apps/app_mounts/deluge/config                         0B  38.1M     14.4M   23.7M             0B         0B
VM/ix-apps/app_mounts/deluge/downloads                      0B   878G      191G    687G             0B         0B
VM/ix-apps/app_mounts/transmission                          0B   769G        0B    104K             0B       769G
VM/ix-apps/app_mounts/transmission/config                   0B  12.2M     12.1M    204K             0B         0B
VM/ix-apps/app_mounts/transmission/downloads-complete       0B   769G      769G    120K             0B         0B
VM/ix-apps/app_mounts/transmission/downloads-incomplete     0B   384K      264K    120K             0B         0B
VM/ix-apps/app_mounts/wg-easy                               0B   904K        0B     96K             0B       808K
VM/ix-apps/app_mounts/wg-easy/config                        0B   808K      704K    104K             0B         0B
VM/ix-apps/docker                                           0B   920M        0B    920M             0B         0B
VM/ix-apps/truenas_catalog                                  0B  12.6M        0B   12.6M             0B         0B
VM/online_torrent                                           0B   241G        0B    241G             0B         0B
VM/phil                                                     0B   667M        0B    667M             0B         0B

As I have migrated from 24.04, it seems I have always the old disk structure.

Something really weird is that “transmission” isn’t used at all :

root@maximaxi[/mnt/.ix-apps/app_mounts/transmission]# du -hs .
122K    .

But on the ZFS report there is a lot of room used :

VM/ix-apps/app_mounts/transmission/downloads-complete       0B   769G      769G    120K             0B         0B

As I can’t start Docker, I can’t remove this old unused “transmission” app from the web interface.
Is their a command line to remove an app ?

Thanks you guys for your suggestion !
Phil

[neofusion]

Looks like it’s your Linux ISOs taking up the space.
Specifically Deluge and Transmission download folders contain around 800 - 900GB each.

(I added back the headers back for ease of readability)

NAME                                                     AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
VM/ix-apps/app_mounts/deluge/downloads                      0B   878G      191G    687G             0B         0B
VM/ix-apps/app_mounts/transmission/downloads-complete       0B   769G      769G    120K             0B         0B

I expect that even if you were to use the shell and cd into those folders, you will get an error trying to rm a file manually. Reaching 0 bytes remaining is terribad.

Still, try it and see.
If it fails, I am sorry but I don’t know how to recover from that kind of event.

[Phil242]

@neofusion

VM/ix-apps/

My problem is this folder isn’t exist anymore :-).

I’m just about to move some datas from “VM/online_torrent” and I hope I’ll be able to start again the apps management, and remove Transmission app, to see if I get back the 769G

[essinghigh]

Any snapshots present taking up a load of space?
zfs list -t snapshot -o name,used | sort -k2 -h

[neofusion]

Okay, try a normal zfs list to see if TrueNAS has mounted the transmission downloads somewhere else, or not at all.

[Krill]

I have absolutely no idea if this is a viable option: do you have a spare drive, even a usb stick, you could add as a new vdev to get some space? If you could unlock it you might be able to get back into a working position but you will likely have to destroy and recreate the pool in the medium term.

[neofusion]

Just a reminder, you can only remove a vdev from a pool made out of mirrors, not raidzX.

[Phil242]

Ok, @neofusion show me how to find file usage with “zfs list -o space”. It allow me to move enough files to run again the apps daemon and delete Transmission, who are not useful at all.

Then … @essinghigh give me THE solution, and the winner is:

VM/ix-apps/app_mounts/deluge/downloads@1.1.7          878G

I have actually version 1.1.8 of Deluge. Then, I discover in the webapp a feature to rollback apps to version 1.1.7 !!!

So, when your ISO files are stored in the deluge app scope, all of them are in a snapshot because of the rollback feature. This means I need to store files in a separate dataset, to not snapshot them, so it’ll keep the app at a few megabytes size.

The bad luck is I CAN’T DELETE the rollback files for Deluge app from the web interface . I’ll try to find a solution in CLI.

Thanks you so much guys for your support! I was a little upset of loosing the NAS this evening, all is up and running now

Cheers,

Phil

[essinghigh]

zfs destroy VM/ix-apps/app_mounts/deluge/downloads@1.1.7

EDIT: Or do it from the WebUI, Datasets → Manage Snapshots

[Phil242]

Yeah, with the snapshot clearing, VM pool went back to 44.5% !

Again, many thanks for the support ! I’ll feel a little less n00b this evening.

And I’ll read on ZFS usage, it seams to be very powerful fs !!!

[neofusion]

@Phil242

Ok, @neofusion show me how to find file usage with “zfs list -o space”. It allow me to move enough files to run again the apps daemon and delete Transmission, who are not useful at all

Not sure what you mean here, you already posted that output and it showed the two offending places were data was lingering, thusly limiting your further investigation to those two spots.

[Phil242]

From my understanding, the 878G of Deluge app was twice on disk: one time in the usual filesystem and another time in the ZFS snapshot (because of the “rollback” feature of each app installed). The reason is certainly because I’ve move some datas to another dataset.

Really not sure of those hypotheses, but it’s 100% sure the snapshot removing of old Deluge version solved the space issue at the second.