TrueNAS SCALE System hard reboots when importing “apps” pool (NVMe mirror), pool otherwise healthy

TrueNAS General
1

Hi all,
after a clean TrueNAS 25.10.1 reinstall I’ve run into a reproducible crash related specifically to importing my apps pool. This is a completely fresh install. I have not yet restored my saved system configuration. So the panic occurs on a “clean” system.
The only actions so far were:

  • Boot new 25.10.1 install
  • Set correct time zone
  • Import “vault” pool fom within GUI (6 × 8 TB HDD RAIDZ2) → went OK
  • Attempt to import “apps” pool (2 × 1 TB NVMe mirror) → causes immediate reboot

This is what I tried so far:

root@truenas[~]# zpool import -F apps

causes immediate reboot

root@truenas[~]# zpool import -F -n apps

causes immediate reboot

root@truenas[~]# zpool import
pool: apps
id: <ID>
state: ONLINE
action: The pool can be imported using its name or numeric identifier
config:

    apps                                    ONLINE
      mirror-0                              ONLINE
        <nvme-guid-1>                       ONLINE
        <nvme-guid-2>                       ONLINE

root@truenas[~]# zpool import -o readonly=on -f apps
cannot mount '/apps': failed to create mountpoint: Read-only file system
Import was successful, but unable to mount some datasets

I can see the apps pool and all datasets within it in the GUI under datasets now. Permission field shows “dataset not mounted”.

root@truenas[~]# zpool status apps
pool: apps
state: ONLINE
scan: scrub repaired 0B in 00:12:39 with 0 errors on Sun Jan 18 03:12:43 2026
config:

    NAME                                    STATE  READ WRITE CKSUM
    apps                                    ONLINE 0    0     0
      mirror-0                              ONLINE 0    0     0
        <nvme-guid-1>                       ONLINE 0    0     0
        <nvme-guid-2>                       ONLINE 0    0     0

errors: No known data errors

root@truenas[~]# zfs list -r apps
NAME                                    USED  AVAIL REFER MOUNTPOINT
apps                                    758G  142G  96k   /apps
apps/app-data                           138G  142G  136k  /apps/app-data
apps/app-data/filebrowser               656G  142G  116k  /apps/app-data/filebrowser
...

it lists all my datasets as far as I can tell.

Given that:

  • zpool import shows apps ONLINE,
  • zpool import -o readonly=on -f apps works (with the mountpoint warning), and
  • only a writable import (zpool import -F apps or GUI import) hard‑reboots the host,

What is going on? What would be the recommended next diagnostic step?
Any help is appreciated.

Related info:
I re-installed truenas because the system hit a panic when booting and was stuck in a boot loop. Well, I guess the panic happened while trying to import the apps pool. I was also on 25.10.1 before.

2

It might help if you tell us what system this is, Lucy or Abby, and if there are any changes from your sig info as there is currently problems getting it to update the sigs.

Usual advice is start by making sure hardware is okay. memtest86 and cpu stress. Running smart long tests and drives and checking results.

3 Likes
3

I may not be able to help you here however I know there are a few questions needed to be answered.

  1. Your system listed on your dropdowns are not complete, they both are TrueNAS CE and both neglect to state how much RAM you have. Nore do you specify the system in your posting.
  2. What prompted a clean reinstall?
  3. When was the last time and system configuration when your ‘apps’ pool was working.

Reproducible is a great term however if we have no idea how or why you got to this state, it doesn’t help much. If you said that you have been running 25.10.1 for the past month without issue and absolutely nothing changed, nothing at all, then that helps. But if something changed, then we need to know what changed as well. Even the little things matter.

1 Like
4

Its lucy.
these are the specs:

TrueNAS 25.10.1
ASUS WS-X299 PRO/SE
Core i9-7900X
Nvidia RTX 2080ti
Mellanox ConnectX-3 10G NIC
LSI SAS2308-8I (9217-8I)
96GB RAM DDR4 non-ECC (3 channels, 2x16GB per channel. 4th channel is dead.)
Pools:
  Vault:
    Data: 6x 8TB WD Wite label HDD RAIDZ2
    Metadata: 3x 1TB Samsung 980 SSD Mirror
  Apps:
    2x 1TB Samsung 970 Evo Plus SSD Mirror
  Boot:
    2x 500GB Samsung 870 EVO SSD Mirror

History:
I have updated the system to 25.10.1 from 25.10.0.1 without issues about 2 weeks ago. The system was running completely fine up until the 29th. I was logged into the GUI at this date and everything was fine, apps pool was healthy. I made no changes to the system, only updated some apps. I remember now that the updates were not working for some apps. They stopped, but the update timed out. I could manually start them again. I tried a few times, some apps eventually updated, but not all. I left the system in that state.
Then, the system had an unexpected reboot at night and I found it stuck in the boot loop in the morning. On the old boot pool, the panic accured on line

[**  ] (1 of 2) Job ix-zfs.service/start running (49s / 15min 39s)

Because I had no idea what this means and thought the boot pool was damaged, I re-installed TrueNAS fresh onto the mirrored boot devices, and attempted to import the pools. When importing the “apps” pool, it crashes.
Additional note: The boot hangs with the message

Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/nfs-top ... done.
Begin: Running /scripts/local-premount ... done.
Begin: Running /scripts/local-top ... done.
[ 9.179019] spl: loading out-of-tree module taints kernel.
[ 9.182074] spl: module verification failed: signature and/or required key missing - tainting kernel
[ 9.221341] zfs: module license 'CDDL' taints kernel.
[ 9.224453] Disabling lock debugging due to kernel taint
[ 9.227634] zfs: module license taints kernel.
[ 10.141222] ZFS: Loaded module v2.3.4-1, ZFS pool version 5000, ZFS filesystem version 5
Begin: Sleeping for ... done.
Begin: Importing ZFS root pool 'boot-pool' ... Begin: Importing pool 'boot-pool' using defaults ... Failure: 1

Failure: 1

Command: /sbin/zpool import -N -f 'boot-pool'
Message: cannot import 'boot-pool': no such pool available
Error: 1

Failed to import pool 'boot-pool'.
Manually import the pool and exit.

BusyBox v1.35.0 (Debian 1:1.35.0-4+b4) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)_

until I manually type:

/sbin/zpool import -d /dev -N -f boot-pool
exit

Then it boots fine.
But I am not sure whether this is even related.
This is where I am at right now.

5

I forgot to say: Thank you for your fast replies @joeschmuck @SmallBarky . I am very much thankful for your help!

6

I would try booting up a live linux version and run memtest86 (at least 5 passes), SMART Long tests, at least on your boot and apps devices and a CPU stress test. We should rule out hardware issues.

You could also try removing one of your boot devices and powering up. One of the two may be off a bit. I would remove the one that is listed to boot first in the bios and have the other come up. If your problem disappers, then it is the removed boot device or that ‘slot’

You should mention how everything is connected for your drives as it helps with sorting out some problems. Not sure if you have M.2 NVMe devices or all SATA / SCSI devices

1 Like
7

First, I did say I may not be able to help you but after reading your reply, here is what I am thinking, please read it a few times and do only what you feel comfortable with. This is not my area of expertise with APPS issues. You might have more than one problem.

With that said, it bothers me that you are apparently starting to bootstrap the computer and then you have a failure of the boot-pool cannot be loaded, “no such pool available”. This is very odd.

And just to clarify something:

Is this after running a fresh install of 25.10.1 without the loading of the previous configuration data? Regardless, you could have a failed/failing boot-pool drive. Just because they are redundant, does not mean they are equal and completely fault tolerant.

If you feel the boot-pool is an issue, power off, remove one of your boot-pool drives. Power on and ensure the BIOS is setup to boot from the installed drive.
Does the problem still exist? If yes, reinstall 25.10.1 and check again. If the problem still exists, swap out the boot drive for the other drive you removed. Repeat this all over again.

The goal is to rule out a possible drive failure/corruption.

Something addition if you have not already done so… Backup the data you need to rebuild your APPS from the ground up. You might have a corrupted pool.

If you have multiple apps running, disable them all. Power Down, wait a few seconds then power back on. This is different from Reboot, just pointing that out.

If you suspect a drive failure, as @SmallBarky said, run a SMART Long/Extended test on the drive. Or just do them all if you haven’t tested them since the problem started. Better safe than sorry.

Like I said, read this, use your best judgement. These are the steps I’d take knowing what you have told us. Right now I suspect your Apps pool is corrupt in some way. It may just be one of the apps and not the pool itself.

Good luck.

2 Likes
8

@SmallBarky Thank for your help!

This is the drive layout:

vault data: (RAIDZ2)
CPU → PCIEx8 → LSI SAS2308-8I (9217-8I) → 6x SATA 8TB WD Wite label HDD
vault metadata: (3x mirror)
CPU → PCIEx16_2 (bifurcated x4x4x4x4) → 4xM.2 NVME to PCIE adapter → 2x NVMe 1TB Samsung 980 SSD Mirror
CPU → PCIEx4 → 1xM.2 NVMe to PCIE adapter → 1x NVMe 1TB Samsung 980 SSD

apps: (2x mirror)
PCH → PCIEx4_M.2_1 → 1x NVMe 1TB Samsung 970 Evo Plus SSD
PCH → PCIEx4_M.2_2 → 1x NVMe 1TB Samsung 970 Evo Plus SSD

boot: (2x mirror)
PCH → 2x SATA III ports → 2x SATA 500GB Samsung 870 EVO SSD

So the apps and data pool are connected through the PCH.
I will try the stress test, but first I would like to know how I could back up the data from the apps pool. As ist is mounted read-only, I cant use the zfs-snapshot method to send it somewhere on my vault pool for backup. I also cant use rsync, it fails and states the pool is not mounted.

9

@joeschmuck Thank you for sharing your thoughts on this.
Yes, the problem with the boot pool needing the -d /dev to mount is after the fresh install. It happened at the first boot already and ever since. I have not loaded any config. This was not an issue on the old boot pool before all this happened.
Another note:
When I first booted the new install, in the BIOS, the boot drives appeared as

TrueNAS-1 (P4: Samsung SSD 870)
TrueNAS-2 (P3: Samsung SSD 870)

Now, in the BIOS again after a few reboots, the drives show up as:

TrueNAS-1 (P4: Samsung SSD 870)
debian (P3: Samsung SSD 870)

I tried booting one drive at a time (other drive is unplugged, remaining drive is selected in Boot menu)
The “debian” drive will not boot on its own. It reboots at the moment grub should appear.
The “TrueNAS” drive will boot on its own, but it also hangs during boot until it gets the -d /dev to mount the boot-pool. Then it boots fine.

Before I continue, I would like to backup the apps pool. Any way of doing it safely? As I mentioned, zfs snapshot and rsync do not work on the read-only pool.

10

You can try [SOLVED] Backup readonly pool

Wild guessing. Maybe create a new dataset and you mount the RO pool inside that. That may give you the ability to make a snapshot on that new dataset and do the backup with ZFS Replication as long as you include child datasets?

You might want to wait for experienced users to post if that is valid or not

1 Like
11

Thanks for the clarifying information. While I can’t tell you how to backup your apps, and having those have the ability to be restored as full apps, hopefully what @SmallBarky provided you will help.

It sounds like something is definitely wrong with the computer and/or boot drives.

Some more troubleshooting advice:

  1. Remove the app pool NVMe drives so they are not affected if you cannot back them up.
  2. Remove all your other drives so you cannot damage them by accident. While I’m certain you would not cause a problem on purpose, they call it an “accident” for a reason.
  3. On your boot drives, wipe them slick, or at least initialize them where there are no partitions at all.
  4. You can try it with both drives installed and then install a fresh copy of TrueNAS 25.10.1.
  5. Ensure your system boots up without issue. If you have problems still, then you need to solve this problem first. You might need to use only a single drive. The goal is to get TrueNAS to boot without any issues.
  6. If you got it running using a single drive, stick with that for now. The goal again is to just get TrueNAS up and running.

If you get to this point without too much trouble, restore your configuration file. Yes, without your pools connected yet. It should not be a problem other than a few warning messages that your drives are not around. Then you can power down and reconnect your main pool of drives, power on, did they mount?

I will assume they mounted without issue. The next one is the apps, if you are comfortable with it, hopefully you were able to get a backup, you will need to power down again and install your NVMe drives, then power back up. Hopefully they will just mount and start running.

Okay, it might take a miracle but stranger things have happened.

Good luck and keep up in the loop.

1 Like
12

Many thanks for the helpful replies @SmallBarky and @joeschmuck .

This rsync method worked and I could copy my data to the vault pool. Thank you @SmallBarky

I wiped both boot drives (sdc and sdh in my case)

zpool labelclear -f /dev/sdc || true
zpool labelclear -f /dev/sdc1 || true
zpool labelclear -f /dev/sdc2 || true
zpool labelclear -f /dev/sdc3 || true

zpool labelclear -f /dev/sdh || true
zpool labelclear -f /dev/sdh1 || true
zpool labelclear -f /dev/sdh2 || true
zpool labelclear -f /dev/sdh3 || true

sgdisk --zap-all /dev/sdc
sgdisk --zap-all /dev/sdh

downloaded the ISO from the website again, attached it from a pikvm as a flash disk and booted it. It booted fine, i installed it onto both clean drives.
The install finishes with no errors.
Booting truenas for the first time, got stuck at the exact same error:

Begin: Sleeping for ... done.
Begin: Importing ZFS root pool 'boot-pool' ... Begin: Importing pool 'boot-pool' using defaults ... Failure: 1

Failure: 1

Command: /sbin/zpool import -N -f 'boot-pool'
Message: cannot import 'boot-pool': no such pool available
Error: 1

Failed to import pool 'boot-pool'.
Manually import the pool and exit.

BusyBox v1.35.0 (Debian 1:1.35.0-4+b4) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)_

I am out of ideas. Why is it failing to find the boot pool automatically?
Is my install method wrong?

13

I would go with one boot drive physically in the machine. Does the new install work on that one? If it doesn’t, power down, switch to other boot drive and install. Can you get that to boot?
You are trying to systematically rule out your hardware and any failures of the sockets etc. One bad boot device in a mirror can be causing the problem.

1 Like
14

Try a single boot drive as I had stated before. Also you did not state if you removed/disconnected your other drives or not. It is difficult to help with the lack of information.

1 Like
15

@joeschmuck @SmallBarky Many thanks for your answers. I really appreciate your help!

I have removed all devices from the system exept the HBA and physically removed all drives from the system. I attached a manjaro drive and ran prime95 for a few hours and then memtest86 ten times or so. The system seems completely stable.

Next, I removed the Manjaro SSD and attached only one of the boot drives to the HBA.
I trust the HBA more than the onboard SATA controller. Up until now, the boot drives were attached to the onboard sata controller.
I wiped the boot drive clean with zpool labelclear, sgdisk --zap-all and wipefs and installed TrueNAS 25.10.1 from the ISO on just the one drive that is attached.
Result: Got stuck on the cannot import boot-pool line.
Shudown, physically removed the boot drive, attached the other drive. Same procedure. Got stuck at the exact same point.
I attached a known good Intel 530 SSD that has worked fine before as a TrueNAS boot device in a different system and did the exact same thing: Same result. I dont think its the drives.

What would be the next troubleshooting step?
It feels like I am running into some edgecase bug here with my hardware.

  • Maybe try 25.04 or an even earlier version?
  • File a bug report? What would I even state there?

Any ideas are very welcome.

16

have you checked the HBA firmware version and mode? sudo sas2flash -list

@HoneyBadger fresh install on three different boot devices still giving cannot import boot-pool errors. Any troubleshooting ideas?

1 Like
17

@SmallBarky Quick answer:

root@truenas[~]# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2308_2(D1)

    Controller Number            : 0
    Controller                   : SAS2308_2(D1)
    PCI Address                  : 00:09:00:0
    SAS Address                  : 500605b-0-068a-0080
    NVDATA Version (Default)     : 14.01.00.06
    NVDATA Version (Persistent)  : 14.01.00.06
    Firmware Product ID          : 0x2214 (IT)
    Firmware Version             : 20.00.07.00 
    NVDATA Vendor                : LSI  
    NVDATA Product ID            : SAS9207-8i 
    BIOS Version                 : 07.39.02.00
    UEFI BSD Version             : N/A
    FCODE Version                : N/A
    Board Name                   : SAS9217-8i
    Board Assembly               : H3-25566-00C
    Board Tracer Number          : SV31415409

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

It is in a server chassis with lots of airflow. Heatsink is barely warm to the touch.
The tests from today were done with the drives connected to the HBA. The tests before were all done with the drives connected to the Mainboard SATA controller. Both shows the same behavior.

18

Fail to boot sounds like a label challenge - not likely a duplicate, because that would show as an inability to identify which boot-pool is being requested - but it can’t find the pool at all. This feels like a job for a live boot and a sudo zdb -ul against the boot device partitions to make sure it has valid labels.

1 Like
19

@HoneyBadger Thank you for your involvement and help!

root@truenas-installer:~# zdb -ul /dev/sdh3
<bunch of similar output...>
    Uberblock[31]
        magic = 0000000000bab10c
        version = 5000
        txg = 19775
        guid_sum = 4426269849058796251
        timestamp = 1770164587 UTC = Wed Feb 4 00:23:07 2026
        bp = DVA=<0:172d6b1000:1000> DVA=<0:1823b76000:1000> DVA=<0:1d5e822000:1000> [L0 DMU object] fletcher4 uncompre​ssed unencrypted LE contiguous unique triple size=1000L/1000P birth=19775L/19775P fill=250 cksum=0000000449e67b9d:00001077afb3d97a:001fab6486515a80e:2898781e29a5f805
        mmp_magic = 00000000a11cea11
        mmp_delay = 0
        mmp_valid = 0
        checkpoint_txg = 0
        raidz_reflow state=0 off=0
        labels = 0 1 2 3

root@truenas-installer:~# zdb -ul /dev/sdh
<bunch of similar output again...>
    Uberblock[31]
        magic = 0000000000bab10c
        version = 5000
        txg = 19775
        guid_sum = 4426269849058796251
        timestamp = 1770164587 UTC = Wed Feb 4 00:23:07 2026
        bp = DVA=<0:172d6b1000:1000> DVA=<0:1823b76000:1000> DVA=<0:1d5e822000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=19775L/19775P fill=250 cksum=0000000449e67b9d:00001077afb3d97a:001fab6486515a80e:2898781e29a5f805
        mmp_magic = 00000000a11cea11
        mmp_delay = 0
        mmp_valid = 0
        checkpoint_txg = 0
        raidz_reflow state=0 off=0
        labels = 2 3

@HoneyBadger Maybe its clear already to you, but I dont know for sure: After running zpool import -d /dev boot-pool it boots fine. It just does not find the boot-pool automatically on boot.

20

@flammen Can you give me the <bunch of similar output> lines, specifically up to the first Uberblock [0] section?

It should start with

------------------------------------
LABEL 0
------------------------------------

and end with something like

    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3

You’ve already piqued my curiousity with the fact that both the whole block device /dev/sdh and the partition /dev/sdh3 have valid labels.

This was a reinstall over an existing 25.10.1 system, correct?

1 Like