I lost my main 6-disk RAIDZ2 pool this night after two subsequent device got removed from the pool. It got from DEGRADED to SUSPENDED within 5mns as far as alerting emails can tell. The last email I got was pool state is SUSPENDED The following disks are not healthy (one is FAULTED, one is REMOVE…

[image] winnielinnie: The next step would be to reboot and run a full scrub on the pool. Rebooting will probably mean the pool doesn’t import as we never changed the state of the pool, we’re just browsing it at the previous state. @coolnodje if you know of any very recently modified files yo…

[image] coolnodje: I don’t understand the LABEL meaning, even looking at zdb.8 — OpenZFS documentation . But it’s LABEL 0 for all the 6 disks. [image] coolnodje: labels = 0 1 2 3 That line in the zdb -l output identifies the labels it was able to read. I believe zdb -l will show th…

So I’ve been tagged in here to look at this case of weird mismatching txg values and labels that just seem to have gone silly. From a shell, in your home directory on the boot drive, can you do a dd if=/dev/sda of=sda.bin count=32 bs=1M - after that, you’ll need to SCP/SFTP into your system and ret…

@coolnodje In addition to gathering the 32M chunks - I don’t believe the zpool history can target an exported pool, so a written one will have to suffice - can you describe the version of TrueNAS you were on when the pool was created, and any upgrades that have happened since? I get the impression t…

@HoneyBadger Let me know the place you want me to upload the files to. You want a 32Mb chunk from each drive, from sda to sdf correct? This pool is actually fairly old. It was my initial pool back in, probably, 2012 I think, starting with Freenas 9.3 or so. It evolved all along with each version, …

[image] coolnodje: You want a 32Mb chunk from each drive, from sda to sdf correct? Correct. [image] coolnodje: This pool is actually fairly old. It was my initial pool back in, probably, 2012 I think, starting with Freenas 9.3 or so. It evolved all along with each version, and grew from …

[image] Chris Peredun: And just for giggles, let’s get the contents of /proc/spl/kstat/zfs/dbgmsg immediately following the zpool import subramanya command. You want a paste here or a file along with the chunks? AFAICT It seems to basically choke on the txg: label discarded as txg is too lar…

A paste here will be fine, it should be fairly short. Just mostly wanting to see where it’s starting the txg walk and how far it goes back before hurking. [image] coolnodje: The CORE to SCALE migration happened august 2024. I’m assuming you went from CORE 13 → SCALE 24.04 at that point, and …

[image] coolnodje: Any pointer on pool labels documentation? I still don’t how it works really. I do not have any, I do not know if @HoneyBadger does. Essentially ZFS labels are written to the start and end of each logical disk (disk or partition) used by ZFS. There are 4 copies for redundan…

Bringing back a SUSPENDED pool that doesn't show up with `zpool status`

TrueNAS General

HoneyBadger June 5, 2025, 8:08pm 46

And just for giggles, let’s get the contents of /proc/spl/kstat/zfs/dbgmsg immediately following the zpool import subramanya command.

Need help with missing pool

Topic		Replies	Views
Unreadable and uncorrectable sectors on two drives TrueNAS General SCALE , Issue , Hardware	16	1135	October 26, 2024
Cant import pool I/O error. Am I goosed? TrueNAS General SCALE , ZFS	7	192	March 6, 2025
Can't import a pool, but no errors TrueNAS General SCALE , Hardware , ZFS , Import-problem	12	254	November 27, 2025
SMART 'failed to read smart attribute data' alert on scale TrueNAS General	21	1170	February 14, 2025
S.M.A.R.T concerns for "new" disks TrueNAS General SCALE , SMART	1	99	August 27, 2024

Bringing back a SUSPENDED pool that doesn't show up with `zpool status`

Related topics