TrueNAS SCALE System hard reboots when importing “apps” pool (NVMe mirror), pool otherwise healthy

TrueNAS General
21

@HoneyBadger I have cleared the device with these commands:

zpool labelclear -f /dev/sdh || true
zpool labelclear -f /dev/sdh1 || true
zpool labelclear -f /dev/sdh2 || true
zpool labelclear -f /dev/sdh3 || true

sgdisk --zap-all /dev/sdh

wipefs -a /dev/sdh

zpool labelclear only did something on sdh and sdh3.
Then I installed 25.10.1 from the iso.

Here are the first 40 lines. I think you found the problem:

root@truenas-installer:~# zdb -ul /dev/sdh3 | head -40
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'boot-pool'
    state: 0
    txg: 366
    pool_guid: 14651499613250692636
    errata: 0
    compatibility: 'grub2'
    hostname: '(none)'
    top_guid: 8221514309517628231
    guid: 8221514309517628231
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 8221514309517628231
        path: '/dev/sde3'
        whole_disk: 0
        metaslab_array: 256
        metaslab_shift: 32
        ashift: 12
        asize: 499563102208
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3
    Uberblock[0]
<etc...>

root@truenas-installer:~# zdb -ul /dev/sdh | head -40
failed to unpack label 0
failed to unpack label 1
------------------------------------
LABEL 2 (Bad label cksum)
------------------------------------
    version: 5000
    name: 'boot-pool'
    state: 0
    txg: 366
    pool_guid: 14651499613250692636
    errata: 0
    compatibility: 'grub2'
    hostname: '(none)'
    top_guid: 8221514309517628231
    guid: 8221514309517628231
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 8221514309517628231
        path: '/dev/sde3'
        whole_disk: 0
        metaslab_array: 256
        metaslab_shift: 32
        ashift: 12
        asize: 499563102208
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 2 3
    Uberblock[0]
<etc...>
22

So this does look sort of like the expected result, in that it doesn’t quite understand that the “fake” labels on /dev/sdh are actually part of /dev/sdh3 - but it should be able to identify the correct partition.

Is there possibly a UEFI boot issue with the motherboard? :thinking:

1 Like
23

@HoneyBadger Thank you for your answer!
How do I know if there is a UEFI boot issue with the motherboard?

This board ran truenas fine since 22.02 with 2 clean installs in between and I haven’t messed with the BIOS since. But I would love to troubleshoot if I can.
The board is an ASUS WS-X299 PRO/SE.
CSM is on. Should I try with it off?

root@truenas-installer:~# blkid /dev/sdh2
/dev/sdh2: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="ADCF-37D3" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="feb<etc...>
24

Just a quick update:

I have tried booting with CSM disabled and got the exact same behavior: boot gets stuck at this line:

Begin: Sleeping for ... done.
Begin: Importing ZFS root pool 'boot-pool' ... Begin: Importing pool 'boot-pool' using defaults ... Failure: 1

Failure: 1

Command: /sbin/zpool import -N -f 'boot-pool'
Message: cannot import 'boot-pool': no such pool available
Error: 1

Failed to import pool 'boot-pool'.
Manually import the pool and exit.

BusyBox v1.35.0 (Debian 1:1.35.0-4+b4) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)_

until i manually import -d /dev boot-pool

Then I tried messing with the cachefile.
First, I installed the older 25.10.0.1 from the ISO. It got stuck like the 25.10.1 before.
From the shell on the running 25.10.0.1 system, I ran:

root@truenas[~]# zpool set cachefile=/etc/zfs/zpool.cache boot-pool
root@truenas[~]# sync
root@truenas[~]# nano /etc/initramfs-tools/conf.d/zfs

where I put:

ZPOOL_IMPORT_PATH="/dev"

I cant update-initramfs, because its read-only, so I updated the system to 25.10.1 to hopefully rebuild initramfs.
But it did not work. It still gets stuck because it cant find the boot-pool.

Maybe interesting: I have connected my old vault pool and apps pool and booted the system. It still gets stuck, but when I run

(initramfs)_ /sbin/zpool import

I can see my apps pool and my vault pool. Just the boot-pool is missing until i import with -d /dev

I would be grateful for any advice as I really dont know what to do now.