ryanr
October 24, 2025, 9:15am
21
@HoneyBadger I did this:
# zdb -e -bcsvL datatank
10.9T completed ( 86MB/s) estimated time remaining: 4hr 32min 51sec cbc
Traversing all blocks to verify checksums ...
zdb_blkptr_cb: Got error 52 reading <0, 0, 0, 48d> DVA[0]=<0:b2b011ee000:1000> DVA[1]=<0:627512f000:1000> DVA[2]=<0:16190579000:1000> [L0 DMU dnode] fletcher4 lz4 unencrypted LE contiguous unique triple size=4000L/1000P birth=15489506L/15489506P fill=6 cksum=00000033cfab9dad:0000bb89c5773c43:0154eb3d0fa607e0:9efc5bfdadc50569 -- skipping
zdb_blkptr_cb: Got error 52 reading <0, 0, 0, ed3> DVA[0]=<0:b2b014e6000:1000> DVA[1]=<0:63006b6000:1000> DVA[2]=<0:16300156000:1000> [L0 DMU dnode] fletcher4 lz4 unencrypted LE contiguous unique triple size=4000L/1000P birth=15489510L/15489510P fill=5 cksum=00000033fb3b041c:0000bf13557f068a:0160d50a122f0178:b45d716c116f0e1a -- skipping
zdb_blkptr_cb: Got error 52 reading <0, 0, 0, ed6> DVA[0]=<0:b2b014e7000:1000> DVA[1]=<0:63006b7000:1000> DVA[2]=<0:16300157000:1000> [L0 DMU dnode] fletcher4 lz4 unencrypted LE contiguous unique triple size=4000L/1000P birth=15489510L/15489510P fill=6 cksum=0000003fb735b76c:0000e5e4ed8f80be:01a0dbff71baa88e:fa78d8051e3db0ce -- skipping
zdb_blkptr_cb: Got error 52 reading <23189, 0, -1, 0> DVA[0]=<0:b2b0123a000:1000> DVA[1]=<0:6300421000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=15489507L/15489507P fill=2 cksum=000000058454d62e:000014227c2aed71:0024ef2eed625948:2d662b7e33fe764d -- skipping
Error counts:
errno count
52 4
bp count: 105560728
ganged count: 0
bp logical: 12095475617792 avg: 114583
bp physical: 11924163283456 avg: 112960 compression: 1.01
bp allocated: 11932194775040 avg: 113036 compression: 1.01
bp deduped: 0 ref>1: 0 deduplication: 1.00
bp cloned: 0 count: 0
Normal class: 13415743545344 used: 96.05%
Embedded log class 2017865728 used: 11.75%
additional, non-pointer bps of type 0: 23606
Dittoed blocks on same vdev: 793631
Dittoed blocks in same metaslab: 9
indirect vdev id 1 has 96044 segments (93.8K in memory)
indirect vdev id 2 has 879320 segments (859K in memory)
Blocks LSIZE PSIZE ASIZE avg comp %Total Type
- - - - - - - unallocated
2 32K 12K 36K 18K 2.67 0.00 object directory
14 38.5K 10.5K 168K 12K 3.67 0.00 object array
1 16K 4K 12K 12K 4.00 0.00 packed nvlist
- - - - - - - packed nvlist size
48.9K 6.09G 399M 1.17G 24.5K 15.65 0.01 bpobj
- - - - - - - bpobj header
- - - - - - - SPA space map header
3.55K 365M 161M 483M 136K 2.27 0.00 SPA space map
4 160K 160K 160K 40K 1.00 0.00 ZIL intent log
28.1K 518M 117M 235M 8.35K 4.44 0.00 DMU dnode
112 448K 448K 900K 8.04K 1.00 0.00 DMU objset
- - - - - - - DSL directory
67 34.5K 4K 72K 1.07K 8.62 0.00 DSL directory child map
71 226K 57.5K 276K 3.89K 3.92 0.00 DSL dataset snap map
98 1016K 260K 888K 9.06K 3.91 0.00 DSL props
- - - - - - - DSL dataset
- - - - - - - ZFS znode
- - - - - - - ZFS V0 ACL
86.9M 10.8T 10.6T 10.6T 125K 1.01 98.06 ZFS plain file
60.5K 706M 185M 463M 7.65K 3.82 0.00 ZFS directory
43 32K 32K 344K 8K 1.00 0.00 ZFS master node
105 172K 8K 16K 156 21.44 0.00 ZFS delete queue
13.6M 219G 213G 213G 15.6K 1.03 1.92 zvol object
- - - - - - - zvol prop
- - - - - - - other uint8[]
- - - - - - - other uint64[]
- - - - - - - other ZAP
- - - - - - - persistent error log
106 13.2M 1.43M 4.28M 41.3K 9.23 0.00 SPA history
- - - - - - - SPA history offsets
- - - - - - - Pool properties
- - - - - - - DSL permissions
- - - - - - - ZFS ACL
- - - - - - - ZFS SYSACL
- - - - - - - FUID table
- - - - - - - FUID table size
23 15.5K 4.50K 24K 1.04K 3.44 0.00 DSL dataset next clones
- - - - - - - scan work queue
315 168K 17.5K 104K 338 9.63 0.00 ZFS user/group/project used
- - - - - - - ZFS user/group/project quota
- - - - - - - snapshot refcount tags
- - - - - - - DDT ZAP algorithm
- - - - - - - DDT statistics
59 29.5K 29.5K 472K 8K 1.00 0.00 System attributes
- - - - - - - SA master node
43 64.5K 64.5K 344K 8K 1.00 0.00 SA attr registration
110 1.72M 440K 880K 8K 4.00 0.00 SA attr layouts
- - - - - - - scan translations
- - - - - - - deduplicated block
210 260K 151K 804K 3.83K 1.72 0.00 DSL deadlist map
- - - - - - - DSL deadlist map hdr
2 5K 4.50K 24K 12K 1.11 0.00 DSL dir clones
1 128K 4K 12K 12K 32.00 0.00 bpobj subobj
- - - - - - - deferred free
- - - - - - - dedup ditto
487 28.8M 12.0M 36.6M 77.0K 2.39 0.00 other
101M 11.0T 10.8T 10.9T 110K 1.01 100.00 Total
894K 32.1G 6.55G 13.7G 15.7K 4.90 0.12 Metadata Total
Block Size Histogram
block psize lsize asize
size Count Size Cum. Count Size Cum. Count Size Cum.
512: 27.7K 13.9M 13.9M 27.7K 13.9M 13.9M 0 0 0
1K: 37.1K 45.9M 59.7M 37.1K 45.9M 59.7M 0 0 0
2K: 33.0K 86.4M 146M 33.0K 86.4M 146M 0 0 0
4K: 795K 3.11G 3.25G 21.7K 117M 263M 417K 1.63G 1.63G
8K: 1.15M 12.3G 15.6G 14.5K 161M 424M 1.22M 11.6G 13.2G
16K: 12.9M 208G 223G 13.7M 219G 219G 13.3M 217G 230G
32K: 444K 20.8G 244G 771K 24.4G 243G 442K 20.6G 251G
64K: 1.60M 153G 397G 23.1K 2.00G 245G 1.61M 153G 404G
128K: 83.7M 10.5T 10.8T 86.1M 10.8T 11.0T 83.7M 10.5T 10.9T
256K: 0 0 10.8T 0 0 11.0T 1.14K 319M 10.9T
512K: 0 0 10.8T 0 0 11.0T 0 0 10.9T
1M: 0 0 10.8T 0 0 11.0T 0 0 10.9T
2M: 0 0 10.8T 0 0 11.0T 0 0 10.9T
4M: 0 0 10.8T 0 0 11.0T 0 0 10.9T
8M: 0 0 10.8T 0 0 11.0T 0 0 10.9T
16M: 0 0 10.8T 0 0 11.0T 0 0 10.9T
capacity operations bandwidth ---- errors ----
description used avail read write read write read write cksum
datatank 12.2T 528G 2.14K 0 254M 0 0 0 0
mirror 12.2T 528G 2.14K 0 254M 0 0 0 0
/dev/sde2 1.07K 0 126M 0 0 0 137
/dev/sdd2 1.07K 0 127M 0 0 0 137
indirect 0 0 0 0 0 0 0
indirect 0 0 0 0 0 0 0
Is it helpful?
ryanr
October 27, 2025, 3:08pm
22
Running a forced import on half of the mirror just gives me a null pointer dereference from the kernel, then zpool seems to do nothing forever:
# zpool import -d /dev/sde2 datatank -fFX
Yeah, unfortunately thereâs likely severe metadata corruption from the double-mount.
Try dumping a full label list including uberblocks with zdb -ul /dev/sde2 and we can see about trying to spelunk for one to use with -T (probably just âpick the earliest available optionâ) but if it was double-mounted for long enough to rotate through all 32 available positions then itâs been pretty scrambled.
ryanr
October 28, 2025, 1:46pm
24
Here is what I get:
# zdb -ul /dev/sde2
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'datatank'
state: 0
txg: 15493413
pool_guid: 16911827047402167892
errata: 0
hostid: 383395800
hostname: 'datatank'
top_guid: 11077806832579153414
guid: 8569546290774946668
vdev_children: 3
vdev_tree:
type: 'mirror'
id: 0
guid: 11077806832579153414
whole_disk: 0
metaslab_array: 65
metaslab_shift: 34
ashift: 12
asize: 13998367178752
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 8569546290774946668
path: '/dev/sdb'
devid: 'scsi-0QEMU_QEMU_HARDDISK_drive-scsi2'
phys_path: 'pci-0000:01:03.0-scsi-0:0:0:2'
whole_disk: 1
DTL: 21406
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 13180809383598811762
path: '/dev/sda'
devid: 'scsi-0QEMU_QEMU_HARDDISK_drive-scsi1'
phys_path: 'pci-0000:01:02.0-scsi-0:0:0:1'
vdev_enc_sysfs_path: '/sys/class/enclosure/6:0:0:0/Slot 05'
whole_disk: 1
DTL: 9036
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
com.delphix:device_removal
com.klarasystems:vdev_zaps_v2
labels = 0 1 2 3
Uberblock[0]
magic = 0000000000bab10c
version = 5000
txg = 15493440
guid_sum = 15792786418798772495
timestamp = 1759868137 UTC = Tue Oct 7 22:15:37 2025
bp = DVA[0]=<0:16300a6f000:1000> DVA[1]=<0:6309578000:1000> DVA[2]=<0:237c920d000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493440L/15493440P fill=2677 cksum=00000003e04a8997:00000efb17f68653:001cfb6936b50236:256bd57a629cdbbe
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[1]
magic = 0000000000bab10c
version = 5000
txg = 15493473
guid_sum = 15792786418798772495
timestamp = 1759868367 UTC = Tue Oct 7 22:19:27 2025
bp = DVA[0]=<0:b2b016a1000:1000> DVA[1]=<0:63073f0000:1000> DVA[2]=<0:16300346000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493473L/15493473P fill=2680 cksum=000000047c3f1f11:00001153776d77f8:0021805f61411d90:2b3beee9623d0272
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[2]
magic = 0000000000bab10c
version = 5000
txg = 15493602
guid_sum = 15792786418798772495
timestamp = 1759869028 UTC = Tue Oct 7 22:30:28 2025
bp = DVA[0]=<0:b2b01729000:1000> DVA[1]=<0:6307a1b000:1000> DVA[2]=<0:163003ea000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493602L/15493602P fill=2682 cksum=000000051073f296:0000138a36118842:0025bd1a77c917e4:30a570a37eea995f
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[3]
magic = 0000000000bab10c
version = 5000
txg = 15493443
guid_sum = 15792786418798772495
timestamp = 1759868138 UTC = Tue Oct 7 22:15:38 2025
bp = DVA[0]=<0:16303310000:1000> DVA[1]=<0:6309634000:1000> DVA[2]=<0:237c9d07000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493443L/15493443P fill=2679 cksum=00000003c3787efd:00000e8a65863529:001c1ef141800285:244c2090c161dc08
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[4]
magic = 0000000000bab10c
version = 5000
txg = 15493476
guid_sum = 15792786418798772495
timestamp = 1759868383 UTC = Tue Oct 7 22:19:43 2025
bp = DVA[0]=<0:b2b016c8000:1000> DVA[1]=<0:630795d000:1000> DVA[2]=<0:16300369000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493476L/15493476P fill=2681 cksum=000000051c433cef:000013ba0f4edb14:00261dd7a16cc525:31279be5ab9b2011
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[5]
magic = 0000000000bab10c
version = 5000
txg = 15493477
guid_sum = 15792786418798772495
timestamp = 1759868388 UTC = Tue Oct 7 22:19:48 2025
bp = DVA[0]=<0:b2b0170f000:1000> DVA[1]=<0:63079f7000:1000> DVA[2]=<0:163003c9000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493477L/15493477P fill=2681 cksum=000000055d76b06d:000014b404286a23:0027fd7538b3eb06:338dc408e0dde1ba
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[6]
magic = 0000000000bab10c
version = 5000
txg = 15493446
guid_sum = 15792786418798772495
timestamp = 1759868139 UTC = Tue Oct 7 22:15:39 2025
bp = DVA[0]=<0:16303fa8000:1000> DVA[1]=<0:630970d000:1000> DVA[2]=<0:237bf8e1000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493446L/15493446P fill=2677 cksum=00000002f6ab5970:00000b789b49604b:001639ebd04e8716:1cbe35fa0573eeb9
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[7]
magic = 0000000000bab10c
version = 5000
txg = 15493447
guid_sum = 15792786418798772495
timestamp = 1759868140 UTC = Tue Oct 7 22:15:40 2025
bp = DVA[0]=<0:163049e5000:1000> DVA[1]=<0:630972c000:1000> DVA[2]=<0:237c9d6d000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493447L/15493447P fill=2677 cksum=00000003615ef27e:00000d119ff10dbf:00194aad36eed909:20ab057cf25a5124
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[8]
magic = 0000000000bab10c
version = 5000
txg = 15493448
guid_sum = 15792786418798772495
timestamp = 1759868140 UTC = Tue Oct 7 22:15:40 2025
bp = DVA[0]=<0:16304a37000:1000> DVA[1]=<0:6309e26000:1000> DVA[2]=<0:2378b7f6000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493448L/15493448P fill=2678 cksum=00000003dc4fef7d:00000ee801d1b41d:001ccf72119eedad:252a263137ca05a2
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[9]
magic = 0000000000bab10c
version = 5000
txg = 15493449
guid_sum = 15792786418798772495
timestamp = 1759868140 UTC = Tue Oct 7 22:15:40 2025
bp = DVA[0]=<0:16304a59000:1000> DVA[1]=<0:6309e38000:1000> DVA[2]=<0:237f2b2a000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493449L/15493449P fill=2679 cksum=00000003d98c3bdd:00000edc2387c91f:001cb63e74154f15:2506c78bbd1f4946
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[10]
magic = 0000000000bab10c
version = 5000
txg = 15493610
guid_sum = 15792786418798772495
timestamp = 1759869069 UTC = Tue Oct 7 22:31:09 2025
bp = DVA[0]=<0:b2b0175c000:1000> DVA[1]=<0:6308243000:1000> DVA[2]=<0:16300430000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493610L/15493610P fill=2682 cksum=000000048b6aee34:0000118da5e9a825:0021f00db219cfc3:2bcaffce610bfc0c
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[11]
magic = 0000000000bab10c
version = 5000
txg = 15493611
guid_sum = 15792786418798772495
timestamp = 1759869074 UTC = Tue Oct 7 22:31:14 2025
bp = DVA[0]=<0:b2b01773000:1000> DVA[1]=<0:630827a000:1000> DVA[2]=<0:16300450000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493611L/15493611P fill=2682 cksum=0000000430f94d9a:000010354a88b6c0:001f5fdc345f2726:288893ac868da29b
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[12]
magic = 0000000000bab10c
version = 5000
txg = 15493452
guid_sum = 15792786418798772495
timestamp = 1759868142 UTC = Tue Oct 7 22:15:42 2025
bp = DVA[0]=<0:163056b9000:1000> DVA[1]=<0:6309ecb000:1000> DVA[2]=<0:237c9451000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493452L/15493452P fill=2682 cksum=00000002d12aaff7:00000ae9eef3fae1:00152a44ace2bedf:1b650e19241f1e3b
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[13]
magic = 0000000000bab10c
version = 5000
txg = 15493613
guid_sum = 15792786418798772495
timestamp = 1759869084 UTC = Tue Oct 7 22:31:24 2025
bp = DVA[0]=<0:b2b017ac000:1000> DVA[1]=<0:63086b1000:1000> DVA[2]=<0:16300475000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493613L/15493613P fill=2682 cksum=00000003843cdb99:00000d9d67391cfb:001a62c6fc25af88:2221242bcd079672
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[14]
magic = 0000000000bab10c
version = 5000
txg = 15493454
guid_sum = 15792786418798772495
timestamp = 1759868142 UTC = Tue Oct 7 22:15:42 2025
bp = DVA[0]=<0:16305909000:1000> DVA[1]=<0:6309f46000:1000> DVA[2]=<0:b2b01898000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493454L/15493454P fill=2682 cksum=0000000499822a14:000011c857375144:002269b1263b99c4:2c726079e31f838e
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[15]
magic = 0000000000bab10c
version = 5000
txg = 15493455
guid_sum = 15792786418798772495
timestamp = 1759868143 UTC = Tue Oct 7 22:15:43 2025
bp = DVA[0]=<0:16306eb3000:1000> DVA[1]=<0:6309f73000:1000> DVA[2]=<0:b2b018cb000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493455L/15493455P fill=2683 cksum=000000040373ed2e:00000f8826d63355:001e16564416165f:26e6278cc38c2a5e
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[16]
magic = 0000000000bab10c
version = 5000
txg = 15493456
guid_sum = 15792786418798772495
timestamp = 1759868280 UTC = Tue Oct 7 22:18:00 2025
bp = DVA[0]=<0:b2b0164b000:1000> DVA[1]=<0:6307392000:1000> DVA[2]=<0:163002ec000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493456L/15493456P fill=2678 cksum=000000042add874f:0000101d5418afdc:001f30d46dec56a3:284b004b3e651523
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[17]
magic = 0000000000bab10c
version = 5000
txg = 15493457
guid_sum = 15792786418798772495
timestamp = 1759868285 UTC = Tue Oct 7 22:18:05 2025
bp = DVA[0]=<0:b2b01674000:1000> DVA[1]=<0:63073cb000:1000> DVA[2]=<0:16300316000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493457L/15493457P fill=2679 cksum=00000004ca7b11bb:0000127f66aba9e4:0023bfe9e03dbdab:2e1cf61e4ac6c678
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[18]
magic = 0000000000bab10c
version = 5000
txg = 15493458
guid_sum = 15792786418798772495
timestamp = 1759868144 UTC = Tue Oct 7 22:15:44 2025
bp = DVA[0]=<0:16306fbe000:1000> DVA[1]=<0:630a017000:1000> DVA[2]=<0:b2b01983000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493458L/15493458P fill=2685 cksum=000000043b5f48fc:0000105bef7fbc89:001fa7ba6b94abaf:28e1c1dda97b754b
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[19]
magic = 0000000000bab10c
version = 5000
txg = 15493427
guid_sum = 15792786418798772495
timestamp = 1759868133 UTC = Tue Oct 7 22:15:33 2025
bp = DVA[0]=<0:163004d2000:1000> DVA[1]=<0:6308ade000:1000> DVA[2]=<0:237c9d6d000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493427L/15493427P fill=2684 cksum=0000000433b05152:00001037cc14b7f8:001f55528d3c8af8:286731eee1a36224
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[20]
magic = 0000000000bab10c
version = 5000
txg = 15493428
guid_sum = 15792786418798772495
timestamp = 1759868133 UTC = Tue Oct 7 22:15:33 2025
bp = DVA[0]=<0:163004f6000:1000> DVA[1]=<0:6308b29000:1000> DVA[2]=<0:237a129f000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493428L/15493428P fill=2684 cksum=00000003460f1e3b:00000ca905bb0c1c:00188227bff6806f:1faa7c9a399e2a69
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[21]
magic = 0000000000bab10c
version = 5000
txg = 15493429
guid_sum = 15792786418798772495
timestamp = 1759868134 UTC = Tue Oct 7 22:15:34 2025
bp = DVA[0]=<0:1630054a000:1000> DVA[1]=<0:6308b6e000:1000> DVA[2]=<0:237c9d3e000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493429L/15493429P fill=2684 cksum=00000004aa5db128:000012007283fb06:0022c4c3f7e54aef:2cd174578806c4de
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[22]
magic = 0000000000bab10c
version = 5000
txg = 15493430
guid_sum = 15792786418798772495
timestamp = 1759868147 UTC = Tue Oct 7 22:15:47 2025
bp = DVA[0]=<0:b2b015a6000:1000> DVA[1]=<0:630729c000:1000> DVA[2]=<0:16300220000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493430L/15493430P fill=2677 cksum=00000003b9a92519:00000e69ae45e1f6:001be9bb4a78a061:2414773366a6fcc8
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[23]
magic = 0000000000bab10c
version = 5000
txg = 15493431
guid_sum = 15792786418798772495
timestamp = 1759868134 UTC = Tue Oct 7 22:15:34 2025
bp = DVA[0]=<0:163005e4000:1000> DVA[1]=<0:6308c69000:1000> DVA[2]=<0:237c9ca1000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493431L/15493431P fill=2681 cksum=0000000390de4bd4:00000dc60f88771e:001aa1b87e9f950b:225e38a7a88959a7
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[24]
magic = 0000000000bab10c
version = 5000
txg = 15493432
guid_sum = 15792786418798772495
timestamp = 1759868135 UTC = Tue Oct 7 22:15:35 2025
bp = DVA[0]=<0:1630072a000:1000> DVA[1]=<0:6308dc2000:1000> DVA[2]=<0:237b195d000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493432L/15493432P fill=2682 cksum=000000031896d908:00000bfc9837e766:00173aea00056665:1e0c108b85b7bcd5
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[25]
magic = 0000000000bab10c
version = 5000
txg = 15493433
guid_sum = 15792786418798772495
timestamp = 1759868162 UTC = Tue Oct 7 22:16:02 2025
bp = DVA[0]=<0:b2b015d0000:1000> DVA[1]=<0:63072c6000:1000> DVA[2]=<0:1630024e000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493433L/15493433P fill=2678 cksum=000000045cedbb4c:000010da234123d3:002095240847b0c4:2a0b9d4373dbb2a1
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[26]
magic = 0000000000bab10c
version = 5000
txg = 15493434
guid_sum = 15792786418798772495
timestamp = 1759868168 UTC = Tue Oct 7 22:16:08 2025
bp = DVA[0]=<0:b2b0160e000:1000> DVA[1]=<0:63072f5000:1000> DVA[2]=<0:16300289000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493434L/15493434P fill=2677 cksum=00000004344ee74b:00001042b1139a99:001f7abf6c9a9ba2:28ac7f4bc3ff5a88
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[27]
magic = 0000000000bab10c
version = 5000
txg = 15493435
guid_sum = 15792786418798772495
timestamp = 1759868136 UTC = Tue Oct 7 22:15:36 2025
bp = DVA[0]=<0:1630086e000:1000> DVA[1]=<0:6308f95000:1000> DVA[2]=<0:237c9d6d000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493435L/15493435P fill=2676 cksum=00000004893838cd:0000117f50518d42:0021c90a16c8b8dc:2b8a1451654ee657
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[28]
magic = 0000000000bab10c
version = 5000
txg = 15493436
guid_sum = 15792786418798772495
timestamp = 1759868178 UTC = Tue Oct 7 22:16:18 2025
bp = DVA[0]=<0:b2b01625000:1000> DVA[1]=<0:6307343000:1000> DVA[2]=<0:163002b2000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493436L/15493436P fill=2678 cksum=00000004539e9ebf:000010b7eebcf14b:002056597e23c3bf:29bed822261c523d
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[29]
magic = 0000000000bab10c
version = 5000
txg = 15493437
guid_sum = 15792786418798772495
timestamp = 1759868136 UTC = Tue Oct 7 22:15:36 2025
bp = DVA[0]=<0:163008f7000:1000> DVA[1]=<0:63094fb000:1000> DVA[2]=<0:237c9bd4000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493437L/15493437P fill=2677 cksum=0000000319e19c98:00000c0216aa700a:0017467fcef025d0:1e1c3e4b925aa050
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[30]
magic = 0000000000bab10c
version = 5000
txg = 15493438
guid_sum = 15792786418798772495
timestamp = 1759868137 UTC = Tue Oct 7 22:15:37 2025
bp = DVA[0]=<0:16300915000:1000> DVA[1]=<0:6309518000:1000> DVA[2]=<0:237a1b55000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493438L/15493438P fill=2676 cksum=0000000376f6cf87:00000d630ac08294:0019e4497f6990ad:216c618b5a723e31
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[31]
magic = 0000000000bab10c
version = 5000
txg = 15493439
guid_sum = 15792786418798772495
timestamp = 1759868137 UTC = Tue Oct 7 22:15:37 2025
bp = DVA[0]=<0:16300948000:1000> DVA[1]=<0:6309552000:1000> DVA[2]=<0:237bf84e000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=15493439L/15493439P fill=2677 cksum=00000003ee712f9b:00000f2dc4666278:001d55cb3f249412:25d6d0f0ed8987a2
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Youâve got a lot of overlapping timestamps on your uberblocks:
timestamp = 1759868137 UTC = Tue Oct 7 22:15:37 2025 | x137
timestamp = 1759868367 UTC = Tue Oct 7 22:19:27 2025
timestamp = 1759869028 UTC = Tue Oct 7 22:30:28 2025
timestamp = 1759868138 UTC = Tue Oct 7 22:15:38 2025
timestamp = 1759868383 UTC = Tue Oct 7 22:19:43 2025
timestamp = 1759868388 UTC = Tue Oct 7 22:19:48 2025
timestamp = 1759868139 UTC = Tue Oct 7 22:15:39 2025
timestamp = 1759868140 UTC = Tue Oct 7 22:15:40 2025 | x140
timestamp = 1759868140 UTC = Tue Oct 7 22:15:40 2025 | x140
timestamp = 1759868140 UTC = Tue Oct 7 22:15:40 2025 | x140
timestamp = 1759869069 UTC = Tue Oct 7 22:31:09 2025
timestamp = 1759869074 UTC = Tue Oct 7 22:31:14 2025
timestamp = 1759868142 UTC = Tue Oct 7 22:15:42 2025 | x142
timestamp = 1759869084 UTC = Tue Oct 7 22:31:24 2025
timestamp = 1759868142 UTC = Tue Oct 7 22:15:42 2025 | x142
timestamp = 1759868143 UTC = Tue Oct 7 22:15:43 2025
timestamp = 1759868280 UTC = Tue Oct 7 22:18:00 2025
timestamp = 1759868285 UTC = Tue Oct 7 22:18:05 2025
timestamp = 1759868144 UTC = Tue Oct 7 22:15:44 2025
timestamp = 1759868133 UTC = Tue Oct 7 22:15:33 2025 | x133
timestamp = 1759868133 UTC = Tue Oct 7 22:15:33 2025 | x133
timestamp = 1759868134 UTC = Tue Oct 7 22:15:34 2025 | x134
timestamp = 1759868147 UTC = Tue Oct 7 22:15:47 2025
timestamp = 1759868134 UTC = Tue Oct 7 22:15:34 2025 | x134
timestamp = 1759868135 UTC = Tue Oct 7 22:15:35 2025
timestamp = 1759868162 UTC = Tue Oct 7 22:16:02 2025
timestamp = 1759868168 UTC = Tue Oct 7 22:16:08 2025
timestamp = 1759868136 UTC = Tue Oct 7 22:15:36 2025 | x136
timestamp = 1759868178 UTC = Tue Oct 7 22:16:18 2025
timestamp = 1759868136 UTC = Tue Oct 7 22:15:36 2025 | x136
timestamp = 1759868137 UTC = Tue Oct 7 22:15:37 2025 | x137
timestamp = 1759868137 UTC = Tue Oct 7 22:15:37 2025 | x137
This doesnât in itself mean guaranteed that the pool is in trouble, but thatâs an awful lot of closely-packed overlapping times for ZFS to have to figure out which one is correct.
