TrueNAS crashed during SMB transfer, causing boot freeze and crash on pool import

TrueNAS General
, ,
1

Hello everyone,

Yesterday I was moving a large archive (400gb) over SMB to my NAS and during this process the system crashed.
When it rebooted, it would freeze during part 2 of ix-zfs.service.
I tried booting older versions of both Dragonfish and Corbia, but they would only freeze in a different part like swap or during some /etc/ service.

I reinstalled TrueNAS, went with 24.4.2.1 in case this issue would be resolved.
The system boots fine, but any attempt to import the pool would crash the system and the pool to not be imported on reboot.

I mounted the pool as readonly via zpool import -f -o readonly=on Storage hoping to figure out whats wrong but it reports no errors.
While mounted as read-only the system threw [EFAULT] Failed retreiving USER quotas for Storage and [ENOENT] Path /Storage not found when in dataset GUI, also /mnt/ directory was empty.

zpool status -v Storage output

root@truenas[/home/admin]# zpool status -v Storage
  pool: Storage
 state: ONLINE
  scan: scrub in progress since Thu Sep 12 22:40:05 2024
        0B / 4.17T scanned, 0B / 4.17T issued
        0B repaired, 0.00% done, no estimated completion time
config:

        NAME                                      STATE     READ WRITE CKSUM
        Storage                                   ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            b9634d3d-084f-461d-b141-c30a5653b76e  ONLINE       0     0     0
            8000f7b0-edda-4fe1-a504-9eaa04fab760  ONLINE       0     0     0

errors: No known data errors
root@truenas[/home/admin]#

I’m currently running long SMART tests on the disks to see if anything comes up.
Would really appreciate any help with this, I’m very new to running a NAS.

— System Information —

TRUENAS Scale: 24.4.2 (24.4.2.1 on reinstall) Dragonfish

Server:
MOBO: B450 Asrock Fatality - ITX
CPU: 4650g Pro
RAM: 2x16GB Kingston KSM26ED8/16HD ECC 2666MHz
DISKS:
1x Kingston NVME 512GB → System drive
2x Mirrored WD Red Plus 8TB → The broken pool
1x Striped WD Green 3TB
1x Striped SATA SSD Kioxia

2

Resolved the issue by myself.

As I suspected this was a TrueNAS issue, there was no hardware damage on the drives themselves, something caused the pool to become unstable post crash.

I mounted the pool without issue in a Ubuntu live cd and slowly moved out all the data to other drives I had available.

In case anyone else is attempting to do this in the future:

  • Import the pool with:
    sudo zpool import <pool name>
  • Mount a root dataset with:
    sudo zfs set mountpoint=<destination> <pool name>/<dataset name>
  • I had trouble getting permission into the pool, so I just created a user in the live cd with the username of the TrueNAS user and switched to it.
  • Move the data out to other pools or drives.
  • Create a new pool inside TrueNAS using the drives in your old pool, which will wipe them.
  • Move the data back onto your new pool.

Not a fun way to spend a weekend.

3

usernames are irrelevant for filesystem access. The OS does not store usernames anywhere on filesystem. Access is determined by user id (uid) and group id (gid). These are numerical values. Just because you create “bob” on a live CD, does not mean that it will have the same access as “bob” on your original NAS. They often will have different UIDs. If you need to bypass permissions on an imported pool you can circumvent by using the root account (uid 0).