TrueNAS SCALE 25.10.5 - zpool import deadlock after server relocation with RAIDZ expansion - OpenZFS issue #18129

TrueNAS General
1

Hello,

I am experiencing a critical issue where my TrueNAS SCALE 25.10.5 installation cannot import a RAIDZ1 pool after a normal shutdown and server relocation. The pool import process hangs indefinitely, blocking all ZFS operations and preventing the system from completing its boot sequence.

System Configuration:

  • TrueNAS SCALE 25.10.3 (now 25.10.5 after clean reinstall)

  • Boot disk: NVMe 476.9 GB (nvme0n1)

  • Data pool: 4x Seagate IronWolf 8TB in RAIDZ1 (/dev/sda1, /dev/sdb1, /dev/sdc1, /dev/sdd1)

  • Pool name: “pool 1” (with space)

  • Pool GUID: 7835538760692273829

Pool Metadata (from zdb -l):

  • raidz_expand_txgs[0]: 1415393 (pool underwent RAIDZ expansion)

  • txg: 8349471 (identical on all four disks — data is consistent)

  • Features enabled: com.delphix:hole_birth, com.delphix:embedded_data, com.klarasystems:vdev_zaps_v2, org.openzfs:raidz_expansion

  • ZFS labels 0, 1, 2, 3: all present and coherent on all disks

The Problem:
After normal shutdown for server relocation, the pool refuses to import. The zpool import process hangs indefinitely:

sudo zpool import -f -d /dev -o cachefile=none "pool 1"

This command produces no output and never returns. Meanwhile, the boot logs show:

  • task IoThread:1350 blocked for more than 120 seconds

  • task zpool:11319 blocked for more than 120 seconds

  • ix-zfs.service - Import ZFS pools stuck in “starting…” state (progresses from 39s → 3m57s → 5m5s but never completes)

  • Kernel Tainted flag set due to prolonged task stalls

Diagnostics Performed:

  • SMART checks on all four disks: PASSED, zero reallocated sectors, zero uncorrectable errors

  • Sequential read speed: ~260 MB/s on all disks (hardware functioning normally)

  • zdb -l /dev/sda1/sdb1/sdc1/sdd1: all ZFS labels intact and coherent

  • /dev/disk/by-partuuid/ symlinks: all present and pointing correctly

  • zpool.cache file present (1536 bytes, dated Jul 28)

  • iostat after first snapshot: zero I/O on sda/sdb/sdc/sdd — process is truly hung in kernel space

Root Cause (Known Bug):
This matches the symptoms of OpenZFS issue #18129: “Reboot after RAIDZ expand causes zpool import to hang”
https://github.com/openzfs/zfs/issues/18129

The combination of RAIDZ expansion (raidz_expand_txgs present in labels) and server interruption causes an indefinite deadlock during zpool import. The zfs kernel process enters uninterruptible sleep (state D) in spa_lookup, holding a lock that blocks all subsequent ZFS commands.

Question for the Community:
Is there a recovery procedure for this deadlock? I have read references to:

  • zpool import flags like -T or -X for transaction rollback

  • Manual ZFS pool recovery procedures

  • Possible kernel-level debugging steps

The pool data appears intact (all SMART checks passed, labels coherent, txg consistent), but I cannot access it due to this deadlock. Are there any known workarounds or recovery techniques for this specific bug?

Additional Context:

  • Configuration backup downloaded and restored, same issue persists

  • Clean TrueNAS SCALE 25.10.5 installation on new NVMe, same deadlock reproduced

  • All four disks physically present and detected by system

  • No recent hardware changes prior to relocation

Thank you for any guidance. I am willing to provide additional diagnostic output or logs if needed.

2

Did it complete? The bug report you linked is for someone who did not allow the expansion to complete before power cycling.

Before you try to forcefully import anything, what does zpool import -d /dev/disk/by-partuuid show?

Go slowly when there’s risk of data loss.

3

Hi winnielinnie,

Thank you very much for responding so quickly — I really appreciate your guidance. Your questions helped me isolate the real issue.

To answer your questions:

1. Did the RAIDZ expansion complete?
Yes, the expansion completed successfully. The pool was fully operational with all 4 disks in the raidz1 before the server relocation and shutdown.

2. What does zpool import -d /dev/disk/by-partuuid show?
Initially, the import process would hang indefinitely. I discovered the root cause: ix-zfs.service was launching an automatic import at boot and getting stuck in an uninterruptible kernel state (D), blocking all subsequent attempts. I disabled this service with sudo systemctl mask ix-zfs.service.

Current Status — Still Blocked:

After disabling the automatic import service and rebooting, I can now run sudo zpool import without the automatic service interfering. The pool IS detected:

  pool: pool no 1
    id: 7835538760692273829
 state: ONLINE
status: The pool was last accessed by another system.
config:
        pool no 1                                 ONLINE
          raidz1-0                                ONLINE
            0e7078fd-a523-4bf9-9cc5-bf94a79fdc7e  ONLINE
            0f50b683-d67e-4627-922b-e47ad750c550  ONLINE
            b237d905-b9e0-4e1e-a6e4-f2866248b8be  ONLINE
            sda1                                  ONLINE

However, when I attempt to import it with sudo zpool import -f "pool no 1", the process hangs again in uninterruptible sleep (state D). This time it’s a clean environment with no service conflicts.

Diagnostic Evidence:

Using iostat -x 2 5 while the import hangs, I see zero I/O activity on all four data disks (sda, sdb, sdc, sdd) — the process is truly stuck in the kernel, not waiting for disk I/O.

ZFS version: 2.3.4-1

Key Observation:

The pool has raidz_expand_txgs[0]: 1415393 in its metadata, indicating it underwent RAIDZ expansion. Even though the expansion completed before shutdown, the combination of RAIDZ expansion history + import attempt seems to trigger this deadlock consistently.

This matches the symptoms of OpenZFS issue #18129, though I expected that disabling the automatic service would allow a manual import to bypass the issue.

Question:
Is there a known workaround for this specific scenario? Should I attempt recovery from a Linux live USB with a different ZFS version, or are there kernel-level debugging steps I can try?

All SMART checks pass, ZFS labels are coherent on all disks, and pool data appears intact — I’m hopeful there’s a recovery path.

Thank you again for your patience and guidance.

4

You didn’t show this.

5

Apologies, here’s that output from earlier (before I discovered the stuck ix-zfs.service was interfering):

truenas_admin@truenas[~]$ sudo zpool import -d /dev/disk/by-partuuid
no pools available to import

At the time, this was because a leftover zpool import process (launched automatically by ix-zfs.service at boot) was stuck in uninterruptible sleep (state D) in the kernel, holding a lock. After masking that service and rebooting, sudo zpool import (without specifying -d) now detects the pool as ONLINE as I showed above — but attempting to actually import it (-f) still hangs the same way, even in this clean state.

6

From a clean state, what does sudo zpool import -d /dev/disk/by-partuuid show?

You have to keep things simple and not mix steps together. Don’t run the command during a separate import process.

7

Here’s the clean output, run right after reboot with no other zpool commands executed beforehand:

truenas_admin@truenas[~]$ ps aux | grep zpool
(empty - no zpool processes running)

truenas_admin@truenas[~]$ sudo zpool import -d /dev/disk/by-partuuid
  pool: pool no 1
    id: 7835538760692273829
 state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:
        pool no 1                                 ONLINE
          raidz1-0                                ONLINE
            0e7078fd-a523-4bf9-9cc5-bf94a79fdc7e  ONLINE
            0f50b683-d67e-4627-922b-e47ad750c550  ONLINE
            b237d905-b9e0-4e1e-a6e4-f2866248b8be  ONLINE
            d32b3a0c-c89d-4fc8-a404-f513589c30dd  ONLINE

The pool is detected cleanly with all four disks ONLINE. What would you like me to try next?

8

Without using -f, what does it say?

sudo zpool import -d /dev/disk/by-partuuid -R /mnt 7835538760692273829 pool1

I wrote pool1 to force a rename without white spaces.

If it immediately returns because of an error, you can try again with -f. Do not try again if the process hangs.

sudo zpool import -f -d /dev/disk/by-partuuid -R /mnt 7835538760692273829 pool1
9

Ran exactly as instructed, step by step.

**Step 1 — Without `-f`:**

Got an immediate error:

```
cannot import ‘pool no 1’: pool was previously in use from another system.
Last accessed by truenas (hostid=578a5557) at Mon Jul 27 20:54:45 2026
The pool can be imported, use ‘zpool import -f’ to import the pool.
```

**Step 2 — With `-f` as you suggested:**

Ran the command:
```bash
sudo zpool import -f -d /dev/disk/by-partuuid -R /mnt 7835538760692273829 pool1
```

It’s now hung — process is in state D, still running after 10+ minutes:

```
root 7808 S 13:36 sudo zpool import -f -d /dev/disk/by-partuuid -R /mnt 7835538760692273829 pool1
root 7809 D 13:36 zpool import -f -d /dev/disk/by-partuuid -R /mnt 7835538760692273829 pool1
```

`iostat -x 2 3` (run from a separate session) shows zero I/O activity on all four data disks after the initial moment — consistent with the same deadlock as before, not active work. I have not run any further commands. Waiting for your guidance before doing anything else.

10

We can try to rule out TrueNAS.

Are you able to boot into a live Ubuntu USB on this server? It should have support for the latest ZFS. Don’t use a different computer, since it might be unclear if the problem is hardware related if it succeeds.

Try to import the pool in the live Ubuntu session. If it succeeds, then at least your pool is safe and we know the problem might be TrueNAS or the version of ZFS that ships with it.

sudo zpool import -f -d /dev/disk/by-partuuid -R /mnt 7835538760692273829 pool1

If it fails or hangs, then it’s probably a deeper issue.

11

Wow. I had my own reasons to not prefer raid anything.
Basically, beyond the build times, you have more complicated code and more moving parts that must all work perfectly, before you get your data.

12

Tried on a fresh Ubuntu 24.04 live session (live USB, made with balenaEtcher, ZFS tools installed via apt install zfsutils-linux), same server, same disks. Same command:

sudo zpool import -f -d /dev/disk/by-partuuid -R /mnt 7835538760692273829 pool1

Result: same deadlock. ps aux confirms the actual zpool import process is in state D:

root  8595  D+  19:40  zpool import -f -d /dev/disk/by-partuuid -R /mnt 7835538760692273829 pool1

So this appears to rule out TrueNAS’s ZFS version as the cause — it’s reproducible on a clean Ubuntu 24.04 live environment on the same hardware. Screenshot attached.

13

24.04 or 26.04? The latest Ubuntu and OpenZFS is newer by 2 years.

14

Yes, I used Ubuntu 26.04 (live USB). The deadlock reproduced identically with the latest OpenZFS version included in that release.

15

This might seem redundant, but there’s one last thing to try that doesn’t involve any low level tinkering or complexity.

Try in the latest FreeBSD ISO. Choose to enter a live shell, not install. You won’t have access to a web browser or desktop GUI. You’ll need another way to read this thread if you want to remember the command to use.

For FreeBSD, it is different. As the root user:

zpool import -d /dev/gptid

zpool import -f -d /dev/gptid -R /mnt 7835538760692273829 pool1

If that doesn’t work, then leave out the -d flags. I’m not sure if a live FreeBSD session enables GPTID labels.

zpool import 

zpool import -f -R /mnt 7835538760692273829 pool1

You might want to install tmux first with pkg install tmux, if it isn’t already included. This will allow you to launch tmux new as a detachable session to run the import command. You can detach with CTRL + B then release and press D. To enter it again, use tmux a.

If you’re still unable to import the pool, then it means this could be a ZFS bug or corruption in the pool itself, or possibly a hardware issue, since you haven’t tried it on any other hardware. I’m worried it might be the former.

16

Thank you again for guiding me through this methodically. I’ve now tested on FreeBSD 15.1 live, and got a very different result than on Linux/Ubuntu.

FreeBSD behavior:

zpool import detected the pool immediately (all disks ONLINE, same as before). However, when I ran:

zpool import -f -R /mnt 7835538760692273829 pool1

It returned a warning message:

zfs: WARNING: Pool 'pool1' has encountered an uncorrectable I/O failure and has been suspended.

After this warning, the command hung — the prompt never returned, just like it did on Ubuntu. I’m attaching screenshots of both the successful detection and the warning message.

Key difference from Linux: FreeBSD at least reported an actual error (“uncorrectable I/O failure”) before hanging, whereas OpenZFS on Linux just hung silently with no message at all.

This suggests there might be a real I/O error or data corruption at a specific location that FreeBSD detects but OpenZFS/Linux doesn’t handle gracefully.

What would you recommend as the next step?

Thank you again for your patience and expertise — I really appreciate your help!

image
image1007×575 142 KB

image
image891×568 125 KB

17

Can you try again, but import it as readonly?

zpool import -f -o readonly=on -R /mnt 7835538760692273829 pool1

SMART “checks”?

Did you run SMART tests? I would run long selftests on all drives.

How are these drives connected? Are you using ECC RAM? What are the complete hardware specs?

18

Good news! The readonly import worked on FreeBSD:

zpool import -f -o readonly=on -R /mnt 7835538760692273829 pool1

Result: import successful — no deadlock this time. The command completed immediately.

There were some mount errors (expected with readonly and missing mount points), but the core import operation succeeded without hanging.

What should I try next?

19

What does the status show?

zpool status -v pool1
20

You might need to accept that a backup and restore could be a possible way forward. I’ll wait until someone with more knowledge on what steps are possible, which do not involve creating a new pool.

@HoneyBadger might have an idea.