Offline Z2 pool after drive replacement

Kenny_Duval · June 22, 2026, 3:13pm

I had a faulted drive in my Z2 pool that I replaced via the GUI with a new drive. During the replacement the process began throwing errors so I shut the machine down. This is a TrueNAS-13.0-U6.8 installation. A zpool import shows the following.

   The pool may be active on another system, but can be imported using
    the '-f' flag.

see:
config:

    Spinners                                          FAULTED  corrupted data
      raidz2-0                                        DEGRADED
        gptid/c6e19795-7228-11ef-ba7f-00074317efd0    ONLINE
        gptid/04e69d29-7330-11ef-a9f6-00074317efd0    ONLINE
        gptid/5d9b589e-72a0-11ef-a36b-00074317efd0    ONLINE
        gptid/2c0b1b10-72e7-11ef-917c-00074317efd0    ONLINE

It appears to now be holding on to both gtpids for the label da2. I do not want to do a forced import at this point as I feel that is a bad idea. Any guidance would be appreciated.

Kenny_Duval · June 22, 2026, 3:15pm

pool: Spinners
id: 15449366848463911438
state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the ‘-f’ flag.
see:
config:

    Spinners                                          FAULTED  corrupted data
      raidz2-0                                        DEGRADED
        gptid/c6e19795-7228-11ef-ba7f-00074317efd0    ONLINE
        gptid/04e69d29-7330-11ef-a9f6-00074317efd0    ONLINE
        gptid/5d9b589e-72a0-11ef-a36b-00074317efd0    ONLINE
        gptid/2c0b1b10-72e7-11ef-917c-00074317efd0    ONLINE
        replacing-4                                   DEGRADED
          gptid/78242ea7-72c2-11ef-a5f8-00074317efd0  ONLINE
          gptid/092e1995-6e43-11f1-881a-00074317efd0  UNAVAIL  cannot open
        gptid/87f22765-7360-11ef-88e2-00074317efd0    ONLINE
        gptid/06e7c7f0-7206-11ef-9a1b-00074317efd0    ONLINE
        gptid/fee51789-7308-11ef-8146-00074317efd0    ONLINE

SmallBarky · June 22, 2026, 4:50pm

A detailed listing of your hardware may help. How are the disks attached to the system. How did you do your replacement? What disk is showing UNAVAIL? Is that the old disk or the new replacement?

Kenny_Duval · June 22, 2026, 5:52pm

It’s an older Intel(R) Xeon(R) CPU E3-1230 @ 3.20GHz on an MSI server board. 32 gigs of ECC.

FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-RELEASE-p9 n245433-9dc2dc9b081 TRUENAS amd64

CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3192.90-MHz K8-class CPU)
Origin=“GenuineIntel” Id=0x206a7 Family=0x6 Model=0x2a Stepping=7

mpr0: <Avago Technologies (LSI) SAS3008> port 0xb000-0xb0ff mem 0xdf840000-0xdf84ffff,0xdf800000-0xdf83ffff irq 16 at device 0.0 on pci3
mpr0: Firmware: 16.00.12.00, Driver: 23.00.00.00-fbsd

t4nex0: mem 0xdf280000-0xdf2bffff,0xde800000-0xdeffffff,0xdf3c4000-0xdf3c5fff irq 16 at device 0.4 on pci7
cxgbe0: <port 0> on t4nex0
<6>cxgbe0: Ethernet address: 00:07:43:17:ef:d0

8 x Seagate EXOS 7E8 4TB drives connected to the SAS3008. Raid Z2 Pool Name - Spinners

4 x Toshiba P300 2TB drives connected to the SAS2008. Raid Z1 Pool Name - SAFE

The POOL drives are in 3 separate 4 bay hot swapcages.

Boot Mirror pool is directly connected to the onboard SATA controller of the motherboard.

All was working fine until drive replacement attempt this morning. The SAFE pool is online and happy as is the boot pool.

Given the RAID Z2 hasn’t failed multiple drives I am a bit stumped why the POOL has been marked bad and taken offline.

Kenny_Duval · June 22, 2026, 6:06pm

I do see this in the logging when booting.

<118>spa.c:6254:spa_tryimport(): spa_tryimport: importing Spinners
<118>spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): LOADING
<118>vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/gptid/092e1995-6e43-11f1-881a-00074317efd0’: vdev_geom_open: failed to open [error=2]
<118>vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/gptid/04e69d29-7330-11ef-a9f6-00074317efd0’: best uberblock found for spa $import. txg 27882043
<118>spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=27882043
<118>spa_misc.c:404:spa_load_failed(): spa_load($import, config untrusted): FAILED: couldn’t get ‘config’ value in MOS directory [error=5]
<118>spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): UNLOADING
<118>spa.c:6110:spa_import(): spa_import: importing Spinners
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): LOADING
<118>vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/gptid/092e1995-6e43-11f1-881a-00074317efd0’: vdev_geom_open: failed to open [error=2]
<118>vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/gptid/87f22765-7360-11ef-88e2-00074317efd0’: best uberblock found for spa Spinners. txg 27882043
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): using uberblock with txg=27882043
<118>spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config untrusted): FAILED: couldn’t get ‘config’ value in MOS directory [error=5]
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): UNLOADING
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): spa_load_retry: rewind, max txg: 27882042
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): LOADING
<118>vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/gptid/092e1995-6e43-11f1-881a-00074317efd0’: vdev_geom_open: failed to open [error=2]
<118>vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/gptid/c6e19795-7228-11ef-ba7f-00074317efd0’: best uberblock found for spa Spinners. txg 27882042
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): using uberblock with txg=27882042
<118>spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config untrusted): FAILED: couldn’t get ‘config’ value in MOS directory [error=5]
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): UNLOADING
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): spa_load_retry: rewind, max txg: 27882041
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): LOADING
<118>vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/gptid/092e1995-6e43-11f1-881a-00074317efd0’: vdev_geom_open: failed to open [error=2]
<118>vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/gptid/04e69d29-7330-11ef-a9f6-00074317efd0’: best uberblock found for spa Spinners. txg 27882041
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): using uberblock with txg=27882041
<118>spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config untrusted): FAILED: couldn’t get ‘config’ value in MOS directory [error=5]
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): UNLOADING
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): spa_load_retry: rewind, max txg: 27882040
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): LOADING
<118>vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/gptid/092e1995-6e43-11f1-881a-00074317efd0’: vdev_geom_open: failed to open [error=2]
<118>vdev.c:161:vdev_dbgmsg(): disk vdev ‘/dev/gptid/87f22765-7360-11ef-88e2-00074317efd0’: best uberblock found for spa Spinners. txg 27882040
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): using uberblock with txg=27882040
<118>spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config untrusted): FAILED: couldn’t get ‘config’ value in MOS directory [error=5]
<118>spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): UNLOADING

Kenny_Duval · June 22, 2026, 6:07pm

That points to it being unhappy about the previous drive and perhaps the replacement drive.

SmallBarky · June 22, 2026, 6:28pm

@HoneyBadger Core system with drive replacement issues. Looking for assistance since I haven’t been on Core for a while.

@Kenny_Duval If you have CLI info to post back, please use (</>) Preformatted text mode as it makes it a bit more readable. Ctrl+e also brings it up.

Your firmware may be a bit behind. You can check with the commands sas2flash -list and sas3flash -list You can then look to see what is the latest listed from the LSI / Broadcom website. I don’t think that is the current issue and is just maintenance.

Do you have a good, current backup of your SPINNERS pool?

Kenny_Duval · June 22, 2026, 6:42pm

I’ve got most everything on other sources of data retrieval. My Nextcloud install is the only real pain in the but thing of concern. I just don’t see a reason for this thing to not be recoverable unless more than 2 drives failed.

In the end I may lose the Nextcloud install. Not the end of the world. Thes rest is rebuildable from backups. I just never came up with a good solution for backing up the Nextcloud installation.

SmallBarky · June 22, 2026, 6:53pm

Your current situation looked a bit complicated to trouble shoot so I am hoping to get more experienced users to give feedback. It may be a pretty simple fix but I don’t want to risk breaking the pool. Guessing it will be comparing TXGs and, maybe removing a drive and booting. I would wait on doing anything for now.
You may want to make up a spreadsheet or documentation to keep the serial numbers and gptid straight. We don’t want any problems with removing an incorrect drive.

Kenny_Duval · June 22, 2026, 6:58pm

Yeah, I’ll wait for someone smarter than me about ZFS. The ISCSI shares I have mapped to other servers are backed up outside of the array to other drives. It is really just Nextcloud hanging me up. I’ve been planning a move to Scale for a while but with the current prices of even used junk I was trying to wait for some sanity to come back around.

Kenny_Duval · June 22, 2026, 7:00pm

I’m hopeful that this may boil down to off lining both the faulted drive and its replacement since it was in the middle of a replacement process but I can’t be sure that will work.

HoneyBadger · June 23, 2026, 3:10pm

Meta Object Set (MOS) corruption is ugly.

It does seem to keep barfing on that one drive:

/dev/gptid/092e1995-6e43-11f1-881a-00074317efd0

so ID that one with

lsblk -o NAME,SERIAL,PARTUUID

find the parent drive serial number for the matching partition, and try to physically disconnect it.

SmallBarky · June 23, 2026, 3:17pm

This is a Core system. (TrueNAS-13.0-U6.8) What command would do the same?

HoneyBadger · June 23, 2026, 3:19pm

whoops.

I believe geom part list should be the one we want first, and then geom disk list if needed to get the serial number.

Kenny_Duval · June 23, 2026, 3:25pm

Thanks for the suggestions. I have resolved myself to rebuilding from scratch if needed. Which seems like it is likely. I have read and absorbed a good bit on ZFS in the last 12 hours. I am currently trying to import it as read-only with a previous TXG and looking through the zdb -l info for each device. I have all the data but the data folder for the NextCloud instance. If I can get it to mount as a read-only that gives me a moment to suck that data off and start moving to a Scale install for everything.

So far it looks like the pool is being read as I have drive activity across the drive. No return at the command prompt yet but I don’t see the CAM errors tossed across the screen I saw earlier after pulling cleaning and re-seating the SATA connections.

Given the age of the hardware I am ordering some stuff and will rebuild from scratch as I no longer trust the antique by age gear it is running on. I’ll move to a new piece of hardware and add another layer to the backup strategy by using the old gear as a replication point for the pools.

I’ll update this if the read-only import works.

Kenny_Duval · June 23, 2026, 4:27pm

No go on that one. Just going rebuild in fresh hardware. I can’t trust this box with the data again.

HoneyBadger · June 23, 2026, 5:58pm

Same error? If you haven’t wiped it out, you should be able to pull logs with sysctl kstat.zfs.misc.dbgmsg to see if it errored out in read-only.

Kenny_Duval · June 23, 2026, 6:49pm

I’ll take look. I’ve already ordered hardware. I much newer Dell R540 is on its way.

It’s got multiple issues.

1782229245 spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_verify found 39 metadata errors and 896 data errors

I’ll move it into the new enclosure once it is here and take another run at it.

Best I can deduce at this point is a failing back plane on one of the 4 bay cages in it. At one point I can see it dropping another physical device out. That would put it beyond the limit of the RAID Z2 config.

Given what I see in the logs it may still be possible to recover it, but it will take someone with far more knowledge than I do about ZFS. It would be useless to continue trouble shooting or trying in the current enclosure.

1782227628   spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): LOADING
1782227628   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/092e1995-6e43-11f1-881a-00074317efd0': vdev_geom_open: failed to open [error=2]
1782227628   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/c6e19795-7228-11ef-ba7f-00074317efd0': best uberblock found for spa $import. txg 27882043
1782227628   spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=27882043
1782227628   spa_misc.c:404:spa_load_failed(): spa_load($import, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=5]
1782227628   spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): UNLOADING
1782227639   spa.c:6252:spa_tryimport(): spa_tryimport: importing Spinners, max_txg=27881836
1782227639   spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): LOADING
1782227639   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/092e1995-6e43-11f1-881a-00074317efd0': vdev_geom_open: failed to open [error=2]
1782227639   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0': best uberblock found for spa $import. txg 27857560
1782227639   spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=27857560
1782227639   vdev.c:166:vdev_dbgmsg(): replacing-4 vdev (guid 15442831169735986313): vdev_copy_path: vdev type mismatch: replacing != disk
1782227639   spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): provided vdev tree:
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 15449366848463911438, path: N/A, degraded
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 3044575188808327488, path: N/A, degraded
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 12608461263144079861, path: /dev/gptid/c6e19795-7228-11ef-ba7f-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 8494155210588754851, path: /dev/gptid/04e69d29-7330-11ef-a9f6-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 8477440304910353809, path: /dev/gptid/5d9b589e-72a0-11ef-a36b-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 305253118871903556, path: /dev/gptid/2c0b1b10-72e7-11ef-917c-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 4: replacing, guid: 15442831169735986313, path: N/A, degraded
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():         vdev 0: disk, guid: 8675270927048567072, path: /dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():         vdev 1: disk, guid: 10304588760689494607, path: /dev/gptid/092e1995-6e43-11f1-881a-00074317efd0, can't open
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 7922966623353351330, path: /dev/gptid/87f22765-7360-11ef-88e2-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 6: disk, guid: 9190466072889394239, path: /dev/gptid/06e7c7f0-7206-11ef-9a1b-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 7: disk, guid: 5433800611810275809, path: /dev/gptid/fee51789-7308-11ef-8146-00074317efd0, healthy
1782227639   spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): MOS vdev tree:
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 15449366848463911438, path: N/A, closed
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 3044575188808327488, path: N/A, closed
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 12608461263144079861, path: /dev/gptid/c6e19795-7228-11ef-ba7f-00074317efd0, closed
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 8494155210588754851, path: /dev/gptid/04e69d29-7330-11ef-a9f6-00074317efd0, closed
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 8477440304910353809, path: /dev/gptid/5d9b589e-72a0-11ef-a36b-00074317efd0, closed
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 305253118871903556, path: /dev/gptid/2c0b1b10-72e7-11ef-917c-00074317efd0, closed
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 8675270927048567072, path: /dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0, closed
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 7922966623353351330, path: /dev/gptid/87f22765-7360-11ef-88e2-00074317efd0, closed
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 6: disk, guid: 9190466072889394239, path: /dev/gptid/06e7c7f0-7206-11ef-9a1b-00074317efd0, closed
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 7: disk, guid: 5433800611810275809, path: /dev/gptid/fee51789-7308-11ef-8146-00074317efd0, closed
1782227639   spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): vdev_copy_path_strict failed, falling back to vdev_copy_path_relaxed
1782227639   spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): final vdev tree:
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 15449366848463911438, path: N/A, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 3044575188808327488, path: N/A, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 12608461263144079861, path: /dev/gptid/c6e19795-7228-11ef-ba7f-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 8494155210588754851, path: /dev/gptid/04e69d29-7330-11ef-a9f6-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 8477440304910353809, path: /dev/gptid/5d9b589e-72a0-11ef-a36b-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 305253118871903556, path: /dev/gptid/2c0b1b10-72e7-11ef-917c-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 8675270927048567072, path: /dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 7922966623353351330, path: /dev/gptid/87f22765-7360-11ef-88e2-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 6: disk, guid: 9190466072889394239, path: /dev/gptid/06e7c7f0-7206-11ef-9a1b-00074317efd0, healthy
1782227639   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 7: disk, guid: 5433800611810275809, path: /dev/gptid/fee51789-7308-11ef-8146-00074317efd0, healthy
1782227640   spa.c:8392:spa_async_request(): spa=$import async request task=2048
1782227640   spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): LOADED
1782227640   spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): UNLOADING
1782227640   spa.c:6113:spa_import(): spa_import: importing Spinners, max_txg=27881836 (RECOVERY MODE)
1782227640   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): LOADING
1782227640   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0': best uberblock found for spa Spinners. txg 27857560
1782227640   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): using uberblock with txg=27857560
1782227640   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
1782227723   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_verify found 1 metadata errors and 1 data errors
1782227723   spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config trusted): FAILED: spa_load_verify failed [error=5]
1782227723   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): UNLOADING
1782227723   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_retry: rewind, max txg: 27857559
1782227723   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): LOADING
1782227723   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0': best uberblock found for spa Spinners. txg 27857559
1782227723   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): using uberblock with txg=27857559
1782227724   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
1782227806   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_verify found 1 metadata errors and 1 data errors
1782227806   spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config trusted): FAILED: spa_load_verify failed [error=5]
1782227806   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): UNLOADING
1782227806   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_retry: rewind, max txg: 27857558
1782227806   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): LOADING
1782227806   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0': best uberblock found for spa Spinners. txg 27857558
1782227806   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): using uberblock with txg=27857558
1782227807   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
1782227890   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_verify found 1 metadata errors and 1 data errors
1782227890   spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config trusted): FAILED: spa_load_verify failed [error=5]
1782227890   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): UNLOADING
1782227890   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_retry: rewind, max txg: 27857557
1782227890   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): LOADING
1782227890   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0': best uberblock found for spa Spinners. txg 27857557
1782227890   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): using uberblock with txg=27857557
1782227891   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
1782227974   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_verify found 1 metadata errors and 1 data errors
1782227974   spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config trusted): FAILED: spa_load_verify failed [error=5]
1782227974   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): UNLOADING
1782227974   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_retry: rewind, max txg: 27857556
1782227974   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): LOADING
1782227974   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0': best uberblock found for spa Spinners. txg 27857556
1782227974   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): using uberblock with txg=27857556
1782227974   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.
1782228057   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_verify found 1 metadata errors and 1 data errors
1782228057   spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config trusted): FAILED: spa_load_verify failed [error=5]
1782228057   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): UNLOADING
1782228057   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_retry: rewind, max txg: 27857555
1782228057   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): LOADING
1782228058   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0': best uberblock found for spa Spinners. txg 27857555
1782228058   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): using uberblock with txg=27857555
1782228058   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): performing a complete scan of the pool since extreme rewind is on. This may take a very long time.

Then a little later we end up here.

1782229245   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_verify found 39 metadata errors and 896 data errors
1782229245   spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config trusted): FAILED: spa_load_verify failed [error=6]
1782229245   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): UNLOADING
1782229245   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): spa_load_retry: rewind, max txg: 27857541
1782229245   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config trusted): LOADING
1782229252   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/2c0b1b10-72e7-11ef-917c-00074317efd0': vdev_geom_open: failed to open [error=2]
1782229252   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0': vdev_geom_open: failed to open [error=2]
1782229252   vdev.c:161:vdev_dbgmsg(): disk vdev '/dev/gptid/5d9b589e-72a0-11ef-a36b-00074317efd0': vdev_geom_open: failed to open [error=2]
1782229252   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): vdev tree has 1 missing top-level vdevs.
1782229252   spa_misc.c:419:spa_load_note(): spa_load(Spinners, config untrusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
1782229252   spa_misc.c:404:spa_load_failed(): spa_load(Spinners, config untrusted): FAILED: unable to open vdev tree [error=2]
1782229252   vdev.c:216:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 15449366848463911438, path: N/A, can't open
1782229252   vdev.c:216:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 3044575188808327488, path: N/A, can't open
1782229252   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 12608461263144079861, path: /dev/gptid/c6e19795-7228-11ef-ba7f-00074317efd0, healthy
1782229252   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 8494155210588754851, path: /dev/gptid/04e69d29-7330-11ef-a9f6-00074317efd0, healthy
1782229252   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 8477440304910353809, path: /dev/gptid/5d9b589e-72a0-11ef-a36b-00074317efd0, can't open
1782229252   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 305253118871903556, path: /dev/gptid/2c0b1b10-72e7-11ef-917c-00074317efd0, can't open
1782229252   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 8675270927048567072, path: /dev/gptid/78242ea7-72c2-11ef-a5f8-00074317efd0, can't open
1782229252   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 7922966623353351330, path: /dev/gptid/87f22765-7360-11ef-88e2-00074317efd0, healthy
1782229252   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 6: disk, guid: 9190466072889394239, path: /dev/gptid/06e7c7f0-7206-11ef-9a1b-00074317efd0, healthy
1782229252   vdev.c:216:vdev_dbgmsg_print_tree():       vdev 7: disk, guid: 5433800611810275809, path: /dev/gptid/fee51789-7308-11ef-8146-00074317efd0, healthy

3 devices that it can’t talk to are just too much. Those 3 devices are in the same 4 bay cage. Rules out the controller as the other 4 drive array attached to the same controller is in perfect working order. Unfortunately for me that’s a remote replication of a box at the office. Just my luck. If it died it would have had zero impact.

I do greatly appreciate the guidance and help.

Kenny_Duval · June 25, 2026, 9:18pm

New hardware is here. Truenas Scale 25 installed. I’ve moved the drives for the pool over. I can see them all with a zpool import at the command line. If I try to import as a read-only pool can I get it out of a read-only state?

When I run zpool import -nfF against the pool it comes back to a clean prompt with no error.

SmallBarky · June 25, 2026, 9:29pm

Try an import using the GUI. It may import your pool as normal and it is worth a try on the new hardware.

Topic		Replies	Views
TrueNAS stuck with some services after a disk has been replaced TrueNAS General SCALE , ZFS	37	433	June 20, 2025
Help recovering pool (if possible) TrueNAS General SCALE	26	220	April 30, 2025
Pool in faulted state, metadata is corrupted, I/O error TrueNAS General Import-problem	34	311	March 7, 2026
URGENT: RAIDZ1 Pool UNAVAIL After Replace Attempt — All Disks Healthy, Labels Intact, Need Help Assembling Pool TrueNAS General SCALE , ZFS , Import-problem	53	410	December 4, 2025
ZFS import crashing system TrueNAS General SCALE , Hardware , Import-problem	44	378	November 20, 2025

Offline Z2 pool after drive replacement

Related topics