My best guess is it would be, yes. Thursday morning is when the first reset happened.
Not fantastic, but we at least have a better list of opportunities to stick our noses in.
Can you run the for loop but grepping for txg = 1051387 instead of just txg?
If we get all of your member disks reporting that one present, maybe we try entering there.
it does look like it’s on most drives:
root@truenas[/home/truenas_admin]# for dev in /dev/sd*[0-9] /dev/nvme*n*p[0-9]; do
[ -b "$dev" ] || continue
txg=$(sudo zdb -ul "$dev" 2>/dev/null | grep -i "txg = 1051387" | head -1)
if [ -n "$txg" ]; then
echo "$dev -> $txg"
fi
done
/dev/sdac1 -> txg = 1051387
/dev/sdad1 -> txg = 1051387
/dev/sdaf1 -> txg = 1051387
/dev/sdag1 -> txg = 1051387
/dev/sdah1 -> txg = 1051387
/dev/sdak1 -> txg = 1051387
/dev/sdal1 -> txg = 1051387
/dev/sdam1 -> txg = 1051387
/dev/sdan1 -> txg = 1051387
/dev/sdaq1 -> txg = 1051387
/dev/sdar1 -> txg = 1051387
/dev/sdau1 -> txg = 1051387
/dev/sdb1 -> txg = 1051387
/dev/sdh1 -> txg = 1051387
/dev/sdi1 -> txg = 1051387
/dev/sdj1 -> txg = 1051387
/dev/sdl1 -> txg = 1051387
/dev/sdm1 -> txg = 1051387
/dev/sdn1 -> txg = 1051387
/dev/sdp1 -> txg = 1051387
/dev/sdq1 -> txg = 1051387
/dev/sdr1 -> txg = 1051387
/dev/sds1 -> txg = 1051387
/dev/sdu1 -> txg = 1051387
/dev/sdv1 -> txg = 1051387
/dev/sdy1 -> txg = 1051387
Oh no, i just noticed… my NVME’s aren’t in there…
Since your NVMe drives are your metadata vdevs, yes, that’s charitably phrased as “kind of a problem”.
Do they show up in lspci output?
1 Like
they sure do:
08:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD
0b:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD
also in lsblk:
nvme1n1 259:0 0 1.8T 0 disk
└─nvme1n1p1 259:2 0 1.8T 0 part
nvme0n1 259:1 0 1.8T 0 disk
└─nvme0n1p1 259:3 0 1.8T 0 part
And remember, I can mount the pool in Read Only mode without crashing.
Query them both directly with zdb -ul /dev/nvmeXn1p1 and see what comes back.
I just scrolled up and caught that part amidst the zpool status output - so the metadata that’s there is at least intact.
They are WD SN750 drives though which there is a long thread of people reporting odd/intermittent stability issues with on the OpenZFS github:
here we go:
root@truenas[/home/truenas_admin]# zdb -ul /dev/nvme0n1p1
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'Oxidizer'
state: 0
txg: 694409
pool_guid: 10025228163071528045
errata: 0
hostid: 1031674282
hostname: 'srv-nas'
top_guid: 3210901141434158234
guid: 10649673995229025202
vdev_children: 7
vdev_tree:
type: 'mirror'
id: 4
guid: 3210901141434158234
metaslab_array: 679
metaslab_shift: 34
ashift: 12
asize: 1998246641664
is_log: 0
create_txg: 84990
children[0]:
type: 'disk'
id: 0
guid: 10649673995229025202
path: '/dev/disk/by-partuuid/0b89f764-f06a-4bb9-acfb-b985b8120226'
whole_disk: 0
DTL: 20431
create_txg: 84990
children[1]:
type: 'disk'
id: 1
guid: 16231000486363755697
path: '/dev/disk/by-partuuid/3eb27884-6106-4ae2-bbec-1eefbb5bb449'
whole_disk: 0
DTL: 28292
create_txg: 84990
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
com.klarasystems:vdev_zaps_v2
labels = 0 1 2 3
Uberblock[0]
magic = 0000000000bab10c
version = 5000
txg = 1051360
guid_sum = 6731355438852460797
timestamp = 1762453996 UTC = Thu Nov 6 19:33:16 2025
bp = DVA[0]=<4:6c006a0000:1000> DVA[1]=<4:70002f7000:1000> DVA[2]=<4:74000a3000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051360L/1051360P fill=16054 cksum=0000000580dd9733:00001528a994d81f:0028b81f02fae913:344c7a751983ab4d
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[1]
magic = 0000000000bab10c
version = 5000
txg = 1051329
guid_sum = 6731355438852460797
timestamp = 1762433998 UTC = Thu Nov 6 13:59:58 2025
bp = DVA[0]=<4:6c000aa000:1000> DVA[1]=<4:7000044000:1000> DVA[2]=<4:8800011000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051329L/1051329P fill=16081 cksum=0000000465536602:000010e76de98452:002089a640f845bc:29cc9325ad5b9964
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[2]
magic = 0000000000bab10c
version = 5000
txg = 1051330
guid_sum = 6731355438852460797
timestamp = 1762434648 UTC = Thu Nov 6 14:10:48 2025
bp = DVA[0]=<4:4800216000:1000> DVA[1]=<4:4c0021f000:1000> DVA[2]=<4:74000ad000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051330L/1051330P fill=16079 cksum=000000045768d2e6:000010b7ef430936:002039e43f38e025:297530f6f87a9b48
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[3]
magic = 0000000000bab10c
version = 5000
txg = 1051363
guid_sum = 6731355438852460797
timestamp = 1762455935 UTC = Thu Nov 6 20:05:35 2025
bp = DVA[0]=<4:6c00233000:1000> DVA[1]=<4:70000af000:1000> DVA[2]=<4:7c000e0000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051363L/1051363P fill=16046 cksum=00000003e5c426c3:00000f02ea6a9ef2:001cf0cc820f8e89:253cf8470ba4a79f
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[4]
magic = 0000000000bab10c
version = 5000
txg = 1051300
guid_sum = 6731355438852460797
timestamp = 1762429406 UTC = Thu Nov 6 12:43:26 2025
bp = DVA[0]=<4:4e2aabc000:1000> DVA[1]=<4:6ea68f0000:1000> DVA[2]=<4:824f106000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051300L/1051300P fill=16121 cksum=000000042619de3e:00000ff81208ecea:001ec4504f234885:278fe020a2260e8d
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[5]
magic = 0000000000bab10c
version = 5000
txg = 1051365
guid_sum = 6731355438852460797
timestamp = 1762457236 UTC = Thu Nov 6 20:27:16 2025
bp = DVA[0]=<4:6c00612000:1000> DVA[1]=<4:700011a000:1000> DVA[2]=<4:7c000e0000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051365L/1051365P fill=16046 cksum=0000000346671c9b:00000ca1da24a1e9:001863bb837c8750:1f6db20f1f9a294f
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[6]
magic = 0000000000bab10c
version = 5000
txg = 1051334
guid_sum = 6731355438852460797
timestamp = 1762437234 UTC = Thu Nov 6 14:53:54 2025
bp = DVA[0]=<4:48001db000:1000> DVA[1]=<4:4c001f8000:1000> DVA[2]=<4:780003a000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051334L/1051334P fill=16073 cksum=00000004bdfcbc75:0000123a42c3b204:002311c7e6071641:2d07f787d5962a13
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[7]
magic = 0000000000bab10c
version = 5000
txg = 1051303
guid_sum = 6731355438852460797
timestamp = 1762429413 UTC = Thu Nov 6 12:43:33 2025
bp = DVA[0]=<4:4e2ab58000:1000> DVA[1]=<4:6ea698c000:1000> DVA[2]=<4:824f290000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051303L/1051303P fill=16118 cksum=00000004d06c4dfe:00001286fb878892:0023b0b0fa1ce97d:2de28f52b46294a9
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[8]
magic = 0000000000bab10c
version = 5000
txg = 1051336
guid_sum = 6731355438852460797
timestamp = 1762438515 UTC = Thu Nov 6 15:15:15 2025
bp = DVA[0]=<4:6c0023e000:1000> DVA[1]=<4:7000078000:1000> DVA[2]=<4:7400098000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051336L/1051336P fill=16073 cksum=00000003daef2992:00000ed42e212d43:001c8cf1ac6ee77f:24afd4e60dd9bd06
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[9]
magic = 0000000000bab10c
version = 5000
txg = 1051337
guid_sum = 6731355438852460797
timestamp = 1762439156 UTC = Thu Nov 6 15:25:56 2025
bp = DVA[0]=<4:4800118000:1000> DVA[1]=<4:4c001de000:1000> DVA[2]=<4:7c00029000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051337L/1051337P fill=16072 cksum=00000004c9e74520:000012708d60c44b:00238a9b1f4da839:2db82de35b700ae9
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[10]
magic = 0000000000bab10c
version = 5000
txg = 1051338
guid_sum = 6731355438852460797
timestamp = 1762439804 UTC = Thu Nov 6 15:36:44 2025
bp = DVA[0]=<4:6c00353000:1000> DVA[1]=<4:7000077000:1000> DVA[2]=<4:7c00076000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051338L/1051338P fill=16071 cksum=000000044b7670d8:0000108ad47d3183:001fe4b28d95f368:2909dae9a0811291
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[11]
magic = 0000000000bab10c
version = 5000
txg = 1051179
guid_sum = 6731355438852460797
timestamp = 1762429177 UTC = Thu Nov 6 12:39:37 2025
bp = DVA[0]=<4:768b632000:1000> DVA[1]=<4:6ea54f7000:1000> DVA[2]=<4:86b12aa000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051179L/1051179P fill=16115 cksum=00000004411ab1f8:0000106334d635ab:001f98d0848ba57f:28a8d92c317e0b05
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[12]
magic = 0000000000bab10c
version = 5000
txg = 1051372
guid_sum = 6731355438852460797
timestamp = 1762461781 UTC = Thu Nov 6 21:43:01 2025
bp = DVA[0]=<4:6c00800000:1000> DVA[1]=<4:70000e5000:1000> DVA[2]=<4:740014b000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051372L/1051372P fill=16030 cksum=00000004b042a56a:000012064f3ccdb7:0022af6045e427b7:2c8b94b87fc4c6e1
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[13]
magic = 0000000000bab10c
version = 5000
txg = 1051309
guid_sum = 6731355438852460797
timestamp = 1762429423 UTC = Thu Nov 6 12:43:43 2025
bp = DVA[0]=<4:4e2ac24000:1000> DVA[1]=<4:6ea6a58000:1000> DVA[2]=<4:824f529000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051309L/1051309P fill=16119 cksum=00000004091e9774:00000f89f1bb2aa2:001df2dbcb59c14b:2686029ce1cb49cd
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[14]
magic = 0000000000bab10c
version = 5000
txg = 1051342
guid_sum = 6731355438852460797
timestamp = 1762442392 UTC = Thu Nov 6 16:19:52 2025
bp = DVA[0]=<4:6c0012d000:1000> DVA[1]=<4:7000078000:1000> DVA[2]=<4:7c00198000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051342L/1051342P fill=16068 cksum=00000004674794cf:000010f3f1950ef0:0020ab7b92912b8e:2a04b5b91c5e611a
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[15]
magic = 0000000000bab10c
version = 5000
txg = 1051375
guid_sum = 6731355438852460797
timestamp = 1762468470 UTC = Thu Nov 6 23:34:30 2025
bp = DVA[0]=<4:7400735000:1000> DVA[1]=<4:780031b000:1000> DVA[2]=<4:7c00326000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051375L/1051375P fill=16025 cksum=000000049bd539d9:000011ba9b6a8a17:0022231a50449806:2bde501fc1f54964
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[16]
magic = 0000000000bab10c
version = 5000
txg = 1051344
guid_sum = 6731355438852460797
timestamp = 1762443675 UTC = Thu Nov 6 16:41:15 2025
bp = DVA[0]=<4:6c0010f000:1000> DVA[1]=<4:700007c000:1000> DVA[2]=<4:7c00072000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051344L/1051344P fill=16069 cksum=00000004750e844c:000011293716e6ad:0021128f9ee99fb9:2a89ca5b2f3da665
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[17]
magic = 0000000000bab10c
version = 5000
txg = 1051217
guid_sum = 6731355438852460797
timestamp = 1762429233 UTC = Thu Nov 6 12:40:33 2025
bp = DVA[0]=<4:768bc4f000:1000> DVA[1]=<4:6ea5b14000:1000> DVA[2]=<4:86b314c000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051217L/1051217P fill=16114 cksum=0000000419a47024:00000fc74611103d:001e64b90cba947b:2712fe4a20758b81
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[18]
magic = 0000000000bab10c
version = 5000
txg = 1051346
guid_sum = 6731355438852460797
timestamp = 1762444975 UTC = Thu Nov 6 17:02:55 2025
bp = DVA[0]=<4:6c003c8000:1000> DVA[1]=<4:700007f000:1000> DVA[2]=<4:800011f000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051346L/1051346P fill=16064 cksum=000000051c86575f:000013ab0a10d3dc:0025e1bc1480b0f7:30b1e0adf32b7883
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[19]
magic = 0000000000bab10c
version = 5000
txg = 1051379
guid_sum = 6731355438852460797
timestamp = 1762502960 UTC = Fri Nov 7 09:09:20 2025
bp = DVA[0]=<4:6c0075d000:1000> DVA[1]=<4:7000465000:1000> DVA[2]=<4:740084f000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051379L/1051379P fill=16019 cksum=00000004270ee3cb:00000ff9a2ada2ea:001ec3204db7e891:2788ca1a7eeeecae
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[20]
magic = 0000000000bab10c
version = 5000
txg = 1051380
guid_sum = 6731355438852460797
timestamp = 1762508797 UTC = Fri Nov 7 10:46:37 2025
bp = DVA[0]=<4:6c00a31000:1000> DVA[1]=<4:7000599000:1000> DVA[2]=<4:7400980000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051380L/1051380P fill=16018 cksum=00000004132388c8:00000fb28436ff62:001e44d7ece867dc:26f44569c69b2e15
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[21]
magic = 0000000000bab10c
version = 5000
txg = 1051253
guid_sum = 6731355438852460797
timestamp = 1762429291 UTC = Thu Nov 6 12:41:31 2025
bp = DVA[0]=<4:4a56dde000:1000> DVA[1]=<4:7a6ac88000:1000> DVA[2]=<4:86b4e0e000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051253L/1051253P fill=16103 cksum=0000000418f98fd3:00000fc38ff41a69:001e5b601a9bb71a:2704276e3c89244f
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[22]
magic = 0000000000bab10c
version = 5000
txg = 1051382
guid_sum = 6731355438852460797
timestamp = 1762519873 UTC = Fri Nov 7 13:51:13 2025
bp = DVA[0]=<4:6c008ea000:1000> DVA[1]=<4:7000571000:1000> DVA[2]=<4:7c00386000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051382L/1051382P fill=16016 cksum=00000003de61a48f:00000ee5f73b7b4a:001cb7fe6697b8c8:24f295742cd6aeab
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[23]
magic = 0000000000bab10c
version = 5000
txg = 1051383
guid_sum = 6731355438852460797
timestamp = 1762554600 UTC = Fri Nov 7 23:30:00 2025
bp = DVA[0]=<4:6c00ab9000:1000> DVA[1]=<4:7000628000:1000> DVA[2]=<4:74009ed000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051383L/1051383P fill=16012 cksum=0000000388056343:00000d9b80a943de:001a3f0ad8350da1:21c97f21152091e0
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[24]
magic = 0000000000bab10c
version = 5000
txg = 1051384
guid_sum = 6731355438852460797
timestamp = 1762557335 UTC = Sat Nov 8 00:15:35 2025
bp = DVA[0]=<4:4800a6f000:1000> DVA[1]=<4:4c00743000:1000> DVA[2]=<4:7c000ea000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051384L/1051384P fill=16013 cksum=00000002981af6f2:00000a05891532e7:0013611e47902f64:1902f2cd9820b124
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[25]
magic = 0000000000bab10c
version = 5000
txg = 1051257
guid_sum = 6731355438852460797
timestamp = 1762429299 UTC = Thu Nov 6 12:41:39 2025
bp = DVA[0]=<4:4a56e7b000:1000> DVA[1]=<4:7a6ad25000:1000> DVA[2]=<4:86b51b7000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051257L/1051257P fill=16106 cksum=00000004e84d765e:000012e31909e55b:00246286dba2554b:2ec7a8eb6d048633
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[26]
magic = 0000000000bab10c
version = 5000
txg = 1051322
guid_sum = 6731355438852460797
timestamp = 1762429474 UTC = Thu Nov 6 12:44:34 2025
bp = DVA[0]=<4:72ea89d000:1000> DVA[1]=<4:671a808000:1000> DVA[2]=<4:87338c4000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051322L/1051322P fill=16096 cksum=00000004d427fa91:000012910afe6c29:0023bbdc4699a73b:2de67954967bcbd8
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[27]
magic = 0000000000bab10c
version = 5000
txg = 1051323
guid_sum = 6731355438852460797
timestamp = 1762430120 UTC = Thu Nov 6 12:55:20 2025
bp = DVA[0]=<4:6c00005000:1000> DVA[1]=<4:7000005000:1000> DVA[2]=<4:740002a000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051323L/1051323P fill=16093 cksum=0000000414f08939:00000fb273296561:001e37409e5aa3f0:26d182e26ecc1a8d
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[28]
magic = 0000000000bab10c
version = 5000
txg = 1051324
guid_sum = 6731355438852460797
timestamp = 1762430759 UTC = Thu Nov 6 13:05:59 2025
bp = DVA[0]=<4:7400011000:1000> DVA[1]=<4:800004e000:1000> DVA[2]=<4:8800002000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051324L/1051324P fill=16091 cksum=00000005b888d3ba:0000160146a7ca2b:002a5deb0dbbf40a:3670830d7551044c
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[29]
magic = 0000000000bab10c
version = 5000
txg = 1051357
guid_sum = 6731355438852460797
timestamp = 1762452057 UTC = Thu Nov 6 19:00:57 2025
bp = DVA[0]=<4:480040c000:1000> DVA[1]=<4:4c003b2000:1000> DVA[2]=<4:7400108000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051357L/1051357P fill=16063 cksum=000000049de8cf19:000011c5c71f34d5:00223ed1890ef42d:2c0a01923963a2af
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[30]
magic = 0000000000bab10c
version = 5000
txg = 1051390
guid_sum = 6731355438852460797
timestamp = 1762871543 UTC = Tue Nov 11 15:32:23 2025
bp = DVA[0]=<4:4800dc2000:1000> DVA[1]=<4:4c014d6000:1000> DVA[2]=<4:740014d000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051390L/1051390P fill=15998 cksum=00000003ad91f3a4:00000e2898bb3006:001b482875ba1952:23159d2ffb7e3863
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[31]
magic = 0000000000bab10c
version = 5000
txg = 1051327
guid_sum = 6731355438852460797
timestamp = 1762432700 UTC = Thu Nov 6 13:38:20 2025
bp = DVA[0]=<4:700003b000:1000> DVA[1]=<4:7400030000:1000> DVA[2]=<4:7c000c8000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051327L/1051327P fill=16081 cksum=00000004018a4b67:00000f6b84ca24f8:001db5e784111b8f:2634bb11ffbfad7e
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
root@truenas[/home/truenas_admin]# zdb -ul /dev/nvme1n1p1
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'Oxidizer'
state: 0
txg: 694409
pool_guid: 10025228163071528045
errata: 0
hostid: 1031674282
hostname: 'srv-nas'
top_guid: 3210901141434158234
guid: 16231000486363755697
vdev_children: 7
vdev_tree:
type: 'mirror'
id: 4
guid: 3210901141434158234
metaslab_array: 679
metaslab_shift: 34
ashift: 12
asize: 1998246641664
is_log: 0
create_txg: 84990
children[0]:
type: 'disk'
id: 0
guid: 10649673995229025202
path: '/dev/disk/by-partuuid/0b89f764-f06a-4bb9-acfb-b985b8120226'
whole_disk: 0
DTL: 20431
create_txg: 84990
children[1]:
type: 'disk'
id: 1
guid: 16231000486363755697
path: '/dev/disk/by-partuuid/3eb27884-6106-4ae2-bbec-1eefbb5bb449'
whole_disk: 0
DTL: 28292
create_txg: 84990
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
com.klarasystems:vdev_zaps_v2
labels = 0 1 2 3
Uberblock[0]
magic = 0000000000bab10c
version = 5000
txg = 1051360
guid_sum = 6731355438852460797
timestamp = 1762453996 UTC = Thu Nov 6 19:33:16 2025
bp = DVA[0]=<4:6c006a0000:1000> DVA[1]=<4:70002f7000:1000> DVA[2]=<4:74000a3000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051360L/1051360P fill=16054 cksum=0000000580dd9733:00001528a994d81f:0028b81f02fae913:344c7a751983ab4d
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[1]
magic = 0000000000bab10c
version = 5000
txg = 1051329
guid_sum = 6731355438852460797
timestamp = 1762433998 UTC = Thu Nov 6 13:59:58 2025
bp = DVA[0]=<4:6c000aa000:1000> DVA[1]=<4:7000044000:1000> DVA[2]=<4:8800011000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051329L/1051329P fill=16081 cksum=0000000465536602:000010e76de98452:002089a640f845bc:29cc9325ad5b9964
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[2]
magic = 0000000000bab10c
version = 5000
txg = 1051330
guid_sum = 6731355438852460797
timestamp = 1762434648 UTC = Thu Nov 6 14:10:48 2025
bp = DVA[0]=<4:4800216000:1000> DVA[1]=<4:4c0021f000:1000> DVA[2]=<4:74000ad000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051330L/1051330P fill=16079 cksum=000000045768d2e6:000010b7ef430936:002039e43f38e025:297530f6f87a9b48
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[3]
magic = 0000000000bab10c
version = 5000
txg = 1051363
guid_sum = 6731355438852460797
timestamp = 1762455935 UTC = Thu Nov 6 20:05:35 2025
bp = DVA[0]=<4:6c00233000:1000> DVA[1]=<4:70000af000:1000> DVA[2]=<4:7c000e0000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051363L/1051363P fill=16046 cksum=00000003e5c426c3:00000f02ea6a9ef2:001cf0cc820f8e89:253cf8470ba4a79f
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[4]
magic = 0000000000bab10c
version = 5000
txg = 1051300
guid_sum = 6731355438852460797
timestamp = 1762429406 UTC = Thu Nov 6 12:43:26 2025
bp = DVA[0]=<4:4e2aabc000:1000> DVA[1]=<4:6ea68f0000:1000> DVA[2]=<4:824f106000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051300L/1051300P fill=16121 cksum=000000042619de3e:00000ff81208ecea:001ec4504f234885:278fe020a2260e8d
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[5]
magic = 0000000000bab10c
version = 5000
txg = 1051365
guid_sum = 6731355438852460797
timestamp = 1762457236 UTC = Thu Nov 6 20:27:16 2025
bp = DVA[0]=<4:6c00612000:1000> DVA[1]=<4:700011a000:1000> DVA[2]=<4:7c000e0000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051365L/1051365P fill=16046 cksum=0000000346671c9b:00000ca1da24a1e9:001863bb837c8750:1f6db20f1f9a294f
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[6]
magic = 0000000000bab10c
version = 5000
txg = 1051334
guid_sum = 6731355438852460797
timestamp = 1762437234 UTC = Thu Nov 6 14:53:54 2025
bp = DVA[0]=<4:48001db000:1000> DVA[1]=<4:4c001f8000:1000> DVA[2]=<4:780003a000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051334L/1051334P fill=16073 cksum=00000004bdfcbc75:0000123a42c3b204:002311c7e6071641:2d07f787d5962a13
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[7]
magic = 0000000000bab10c
version = 5000
txg = 1051303
guid_sum = 6731355438852460797
timestamp = 1762429413 UTC = Thu Nov 6 12:43:33 2025
bp = DVA[0]=<4:4e2ab58000:1000> DVA[1]=<4:6ea698c000:1000> DVA[2]=<4:824f290000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051303L/1051303P fill=16118 cksum=00000004d06c4dfe:00001286fb878892:0023b0b0fa1ce97d:2de28f52b46294a9
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[8]
magic = 0000000000bab10c
version = 5000
txg = 1051336
guid_sum = 6731355438852460797
timestamp = 1762438515 UTC = Thu Nov 6 15:15:15 2025
bp = DVA[0]=<4:6c0023e000:1000> DVA[1]=<4:7000078000:1000> DVA[2]=<4:7400098000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051336L/1051336P fill=16073 cksum=00000003daef2992:00000ed42e212d43:001c8cf1ac6ee77f:24afd4e60dd9bd06
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[9]
magic = 0000000000bab10c
version = 5000
txg = 1051337
guid_sum = 6731355438852460797
timestamp = 1762439156 UTC = Thu Nov 6 15:25:56 2025
bp = DVA[0]=<4:4800118000:1000> DVA[1]=<4:4c001de000:1000> DVA[2]=<4:7c00029000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051337L/1051337P fill=16072 cksum=00000004c9e74520:000012708d60c44b:00238a9b1f4da839:2db82de35b700ae9
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[10]
magic = 0000000000bab10c
version = 5000
txg = 1051338
guid_sum = 6731355438852460797
timestamp = 1762439804 UTC = Thu Nov 6 15:36:44 2025
bp = DVA[0]=<4:6c00353000:1000> DVA[1]=<4:7000077000:1000> DVA[2]=<4:7c00076000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051338L/1051338P fill=16071 cksum=000000044b7670d8:0000108ad47d3183:001fe4b28d95f368:2909dae9a0811291
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[11]
magic = 0000000000bab10c
version = 5000
txg = 1051179
guid_sum = 6731355438852460797
timestamp = 1762429177 UTC = Thu Nov 6 12:39:37 2025
bp = DVA[0]=<4:768b632000:1000> DVA[1]=<4:6ea54f7000:1000> DVA[2]=<4:86b12aa000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051179L/1051179P fill=16115 cksum=00000004411ab1f8:0000106334d635ab:001f98d0848ba57f:28a8d92c317e0b05
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[12]
magic = 0000000000bab10c
version = 5000
txg = 1051372
guid_sum = 6731355438852460797
timestamp = 1762461781 UTC = Thu Nov 6 21:43:01 2025
bp = DVA[0]=<4:6c00800000:1000> DVA[1]=<4:70000e5000:1000> DVA[2]=<4:740014b000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051372L/1051372P fill=16030 cksum=00000004b042a56a:000012064f3ccdb7:0022af6045e427b7:2c8b94b87fc4c6e1
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[13]
magic = 0000000000bab10c
version = 5000
txg = 1051309
guid_sum = 6731355438852460797
timestamp = 1762429423 UTC = Thu Nov 6 12:43:43 2025
bp = DVA[0]=<4:4e2ac24000:1000> DVA[1]=<4:6ea6a58000:1000> DVA[2]=<4:824f529000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051309L/1051309P fill=16119 cksum=00000004091e9774:00000f89f1bb2aa2:001df2dbcb59c14b:2686029ce1cb49cd
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[14]
magic = 0000000000bab10c
version = 5000
txg = 1051342
guid_sum = 6731355438852460797
timestamp = 1762442392 UTC = Thu Nov 6 16:19:52 2025
bp = DVA[0]=<4:6c0012d000:1000> DVA[1]=<4:7000078000:1000> DVA[2]=<4:7c00198000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051342L/1051342P fill=16068 cksum=00000004674794cf:000010f3f1950ef0:0020ab7b92912b8e:2a04b5b91c5e611a
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[15]
magic = 0000000000bab10c
version = 5000
txg = 1051375
guid_sum = 6731355438852460797
timestamp = 1762468470 UTC = Thu Nov 6 23:34:30 2025
bp = DVA[0]=<4:7400735000:1000> DVA[1]=<4:780031b000:1000> DVA[2]=<4:7c00326000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051375L/1051375P fill=16025 cksum=000000049bd539d9:000011ba9b6a8a17:0022231a50449806:2bde501fc1f54964
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[16]
magic = 0000000000bab10c
version = 5000
txg = 1051344
guid_sum = 6731355438852460797
timestamp = 1762443675 UTC = Thu Nov 6 16:41:15 2025
bp = DVA[0]=<4:6c0010f000:1000> DVA[1]=<4:700007c000:1000> DVA[2]=<4:7c00072000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051344L/1051344P fill=16069 cksum=00000004750e844c:000011293716e6ad:0021128f9ee99fb9:2a89ca5b2f3da665
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[17]
magic = 0000000000bab10c
version = 5000
txg = 1051217
guid_sum = 6731355438852460797
timestamp = 1762429233 UTC = Thu Nov 6 12:40:33 2025
bp = DVA[0]=<4:768bc4f000:1000> DVA[1]=<4:6ea5b14000:1000> DVA[2]=<4:86b314c000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051217L/1051217P fill=16114 cksum=0000000419a47024:00000fc74611103d:001e64b90cba947b:2712fe4a20758b81
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[18]
magic = 0000000000bab10c
version = 5000
txg = 1051346
guid_sum = 6731355438852460797
timestamp = 1762444975 UTC = Thu Nov 6 17:02:55 2025
bp = DVA[0]=<4:6c003c8000:1000> DVA[1]=<4:700007f000:1000> DVA[2]=<4:800011f000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051346L/1051346P fill=16064 cksum=000000051c86575f:000013ab0a10d3dc:0025e1bc1480b0f7:30b1e0adf32b7883
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[19]
magic = 0000000000bab10c
version = 5000
txg = 1051347
guid_sum = 6731355438852460797
timestamp = 1762445625 UTC = Thu Nov 6 17:13:45 2025
bp = DVA[0]=<4:6c0028a000:1000> DVA[1]=<4:70000cc000:1000> DVA[2]=<4:7c000ab000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051347L/1051347P fill=16063 cksum=00000003f6ad5540:00000f428b9da1a7:001d6895c4e1b9f3:25d366ba9e2714f4
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[20]
magic = 0000000000bab10c
version = 5000
txg = 1051380
guid_sum = 6731355438852460797
timestamp = 1762508797 UTC = Fri Nov 7 10:46:37 2025
bp = DVA[0]=<4:6c00a31000:1000> DVA[1]=<4:7000599000:1000> DVA[2]=<4:7400980000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051380L/1051380P fill=16018 cksum=00000004132388c8:00000fb28436ff62:001e44d7ece867dc:26f44569c69b2e15
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[21]
magic = 0000000000bab10c
version = 5000
txg = 1051253
guid_sum = 6731355438852460797
timestamp = 1762429291 UTC = Thu Nov 6 12:41:31 2025
bp = DVA[0]=<4:4a56dde000:1000> DVA[1]=<4:7a6ac88000:1000> DVA[2]=<4:86b4e0e000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051253L/1051253P fill=16103 cksum=0000000418f98fd3:00000fc38ff41a69:001e5b601a9bb71a:2704276e3c89244f
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[22]
magic = 0000000000bab10c
version = 5000
txg = 1051382
guid_sum = 6731355438852460797
timestamp = 1762519873 UTC = Fri Nov 7 13:51:13 2025
bp = DVA[0]=<4:6c008ea000:1000> DVA[1]=<4:7000571000:1000> DVA[2]=<4:7c00386000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051382L/1051382P fill=16016 cksum=00000003de61a48f:00000ee5f73b7b4a:001cb7fe6697b8c8:24f295742cd6aeab
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[23]
magic = 0000000000bab10c
version = 5000
txg = 1051383
guid_sum = 6731355438852460797
timestamp = 1762554600 UTC = Fri Nov 7 23:30:00 2025
bp = DVA[0]=<4:6c00ab9000:1000> DVA[1]=<4:7000628000:1000> DVA[2]=<4:74009ed000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051383L/1051383P fill=16012 cksum=0000000388056343:00000d9b80a943de:001a3f0ad8350da1:21c97f21152091e0
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[24]
magic = 0000000000bab10c
version = 5000
txg = 1051384
guid_sum = 6731355438852460797
timestamp = 1762557335 UTC = Sat Nov 8 00:15:35 2025
bp = DVA[0]=<4:4800a6f000:1000> DVA[1]=<4:4c00743000:1000> DVA[2]=<4:7c000ea000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051384L/1051384P fill=16013 cksum=00000002981af6f2:00000a05891532e7:0013611e47902f64:1902f2cd9820b124
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[25]
magic = 0000000000bab10c
version = 5000
txg = 1051257
guid_sum = 6731355438852460797
timestamp = 1762429299 UTC = Thu Nov 6 12:41:39 2025
bp = DVA[0]=<4:4a56e7b000:1000> DVA[1]=<4:7a6ad25000:1000> DVA[2]=<4:86b51b7000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051257L/1051257P fill=16106 cksum=00000004e84d765e:000012e31909e55b:00246286dba2554b:2ec7a8eb6d048633
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[26]
magic = 0000000000bab10c
version = 5000
txg = 1051322
guid_sum = 6731355438852460797
timestamp = 1762429474 UTC = Thu Nov 6 12:44:34 2025
bp = DVA[0]=<4:72ea89d000:1000> DVA[1]=<4:671a808000:1000> DVA[2]=<4:87338c4000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051322L/1051322P fill=16096 cksum=00000004d427fa91:000012910afe6c29:0023bbdc4699a73b:2de67954967bcbd8
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[27]
magic = 0000000000bab10c
version = 5000
txg = 1051323
guid_sum = 6731355438852460797
timestamp = 1762430120 UTC = Thu Nov 6 12:55:20 2025
bp = DVA[0]=<4:6c00005000:1000> DVA[1]=<4:7000005000:1000> DVA[2]=<4:740002a000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051323L/1051323P fill=16093 cksum=0000000414f08939:00000fb273296561:001e37409e5aa3f0:26d182e26ecc1a8d
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[28]
magic = 0000000000bab10c
version = 5000
txg = 1051324
guid_sum = 6731355438852460797
timestamp = 1762430759 UTC = Thu Nov 6 13:05:59 2025
bp = DVA[0]=<4:7400011000:1000> DVA[1]=<4:800004e000:1000> DVA[2]=<4:8800002000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051324L/1051324P fill=16091 cksum=00000005b888d3ba:0000160146a7ca2b:002a5deb0dbbf40a:3670830d7551044c
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[29]
magic = 0000000000bab10c
version = 5000
txg = 1051357
guid_sum = 6731355438852460797
timestamp = 1762452057 UTC = Thu Nov 6 19:00:57 2025
bp = DVA[0]=<4:480040c000:1000> DVA[1]=<4:4c003b2000:1000> DVA[2]=<4:7400108000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051357L/1051357P fill=16063 cksum=000000049de8cf19:000011c5c71f34d5:00223ed1890ef42d:2c0a01923963a2af
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[30]
magic = 0000000000bab10c
version = 5000
txg = 1051390
guid_sum = 6731355438852460797
timestamp = 1762871543 UTC = Tue Nov 11 15:32:23 2025
bp = DVA[0]=<4:4800dc2000:1000> DVA[1]=<4:4c014d6000:1000> DVA[2]=<4:740014d000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051390L/1051390P fill=15998 cksum=00000003ad91f3a4:00000e2898bb3006:001b482875ba1952:23159d2ffb7e3863
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Uberblock[31]
magic = 0000000000bab10c
version = 5000
txg = 1051327
guid_sum = 6731355438852460797
timestamp = 1762432700 UTC = Thu Nov 6 13:38:20 2025
bp = DVA[0]=<4:700003b000:1000> DVA[1]=<4:7400030000:1000> DVA[2]=<4:7c000c8000:1000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique triple size=1000L/1000P birth=1051327L/1051327P fill=16081 cksum=00000004018a4b67:00000f6b84ca24f8:001db5e784111b8f:2634bb11ffbfad7e
mmp_magic = 00000000a11cea11
mmp_delay = 0
mmp_valid = 0
checkpoint_txg = 0
raidz_reflow state=0 off=0
labels = 0 1 2 3
Looks like I need to adjust my mindset for recovering pools with special a bit here, they’re bouncing the uberblocks between main pool and special.
Try 1051390 perhaps?
I think what we need to do is pipe the output of journalctl --follow to another machine over SSH (or maybe to a file not on the impacted pool?) so that we can grab what’s actually causing the crash and kernel panic.
I’ve been connecting over SSH this whole time, so that’s no problem… Here is what Journalctl tells me when running a regular import:
root@truenas[/home/truenas_admin]# journalctl --follow
Nov 11 16:37:53 truenas systemd-logind[4426]: New session 4 of user truenas_admin.
Nov 11 16:37:53 truenas systemd[1]: Started session-4.scope - Session 4 of User truenas_admin.
Nov 11 16:37:53 truenas sshd[53223]: pam_unix(sshd:session): session opened for user truenas_admin(uid=950) by (uid=0)
Nov 11 16:37:53 truenas sshd[53223]: pam_env(sshd:session): deprecated reading of user environment enabled
Nov 11 16:37:56 truenas sudo[53264]: @cee:{"sudo":{"accept":{"uuid":"681033b4f3-5930-4281-703f-63769de8c3","server_time":{"seconds":1762875476,"nanoseconds":399710536,"iso8601":"20251111153756Z","localtime":"Nov 11 16:37:56"},"submit_time":{"seconds":1762875475,"nanoseconds":148136266,"iso8601":"20251111153755Z","localtime":"Nov 11 16:37:55"},"submituser":"truenas_admin","command":"/bin/su","runuser":"root","runcwd":"/home/truenas_admin","ttyname":"/dev/pts/1","submithost":"truenas.local","submitcwd":"/home/truenas_admin","runuid":0,"columns":209,"lines":88,"runargv":["su"],"runenv":["PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin","TERM=xterm-256color","LC_ALL=C.UTF-8","MAIL=/var/mail/root","LOGNAME=root","USER=root","HOME=/root","SHELL=/usr/bin/zsh","SUDO_COMMAND=/bin/su","SUDO_USER=truenas_admin","SUDO_UID=950","SUDO_GID=950"]}}}
Nov 11 16:37:56 truenas sudo[53264]: pam_unix(sudo:session): session opened for user root(uid=0) by truenas_admin(uid=950)
Nov 11 16:37:56 truenas su[53265]: (to root) root on pts/1
Nov 11 16:37:56 truenas su[53265]: pam_unix(su:session): session opened for user root(uid=0) by truenas_admin(uid=0)
Nov 11 16:37:56 truenas sudo[53264]: @cee:{"sudo":{"accept":{"uuid":"7261474171-cce3-45aa-88b3-cf2515445e","server_time":{"seconds":1762875476,"nanoseconds":410395050,"iso8601":"20251111153756Z","localtime":"Nov 11 16:37:56"},"submit_time":{"seconds":1762875475,"nanoseconds":148136266,"iso8601":"20251111153755Z","localtime":"Nov 11 16:37:55"},"submituser":"truenas_admin","command":"/usr/bin/zsh","runuser":"root","runcwd":"/home/truenas_admin","ttyname":"/dev/pts/1","submithost":"truenas.local","submitcwd":"/home/truenas_admin","runuid":0,"columns":209,"lines":88,"runargv":["/usr/bin/zsh"],"runenv":["PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin","TERM=xterm-256color","LC_ALL=C.UTF-8","MAIL=/var/mail/root","LOGNAME=root","USER=root","HOME=/root","SHELL=/usr/bin/zsh","SUDO_COMMAND=/bin/su","SUDO_USER=truenas_admin","SUDO_UID=950","SUDO_GID=950"]}}}
Nov 11 16:38:07 truenas sudo[53264]: @cee:{"sudo":{"accept":{"uuid":"f80151b865-74e8-4db1-816f-b2a362b483","server_time":{"seconds":1762875487,"nanoseconds":320410151,"iso8601":"20251111153807Z","localtime":"Nov 11 16:38:07"},"submit_time":{"seconds":1762875475,"nanoseconds":148136266,"iso8601":"20251111153755Z","localtime":"Nov 11 16:37:55"},"submituser":"truenas_admin","command":"/bin/journalctl","runuser":"root","runcwd":"/home/truenas_admin","ttyname":"/dev/pts/1","submithost":"truenas.local","submitcwd":"/home/truenas_admin","runuid":0,"columns":209,"lines":88,"runargv":["/bin/journalctl","--follow"],"runenv":["PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin","TERM=xterm-256color","LC_ALL=C.UTF-8","MAIL=/var/mail/root","LOGNAME=root","USER=root","HOME=/root","SHELL=/usr/bin/zsh","SUDO_COMMAND=/bin/su","SUDO_USER=truenas_admin","SUDO_UID=950","SUDO_GID=950","SHLVL=1","PWD=/home/truenas_admin","OLDPWD=/home/truenas_admin","_=/bin/journalctl"]}}}
Nov 11 16:38:11 truenas sudo[6824]: @cee:{"sudo":{"accept":{"uuid":"c16092d047-ea80-42b8-a2f0-60ed9f6051","server_time":{"seconds":1762875491,"nanoseconds":903147127,"iso8601":"20251111153811Z","localtime":"Nov 11 16:38:11"},"submit_time":{"seconds":1762872319,"nanoseconds":308902255,"iso8601":"20251111144519Z","localtime":"Nov 11 15:45:19"},"submituser":"truenas_admin","command":"/sbin/zpool","runuser":"root","runcwd":"/home/truenas_admin","ttyname":"/dev/pts/0","submithost":"truenas.local","submitcwd":"/home/truenas_admin","runuid":0,"columns":209,"lines":88,"runargv":["/sbin/zpool","import","-f","Oxidizer"],"runenv":["PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin","TERM=xterm-256color","LC_ALL=C.UTF-8","MAIL=/var/mail/root","LOGNAME=root","USER=root","HOME=/root","SHELL=/usr/bin/zsh","SUDO_COMMAND=/bin/su","SUDO_USER=truenas_admin","SUDO_UID=950","SUDO_GID=950","SHLVL=1","PWD=/home/truenas_admin","OLDPWD=/home/truenas_admin","_=/sbin/zpool"]}}}
Nov 11 16:39:32 truenas dhclient[4160]: DHCPREQUEST for 192.168.5.100 on bond1 to 192.168.5.1 port 67
Nov 11 16:39:32 truenas dhclient[4160]: DHCPACK of 192.168.5.100 from 192.168.5.1
Nov 11 16:39:32 truenas dhclient[4160]: bound to 192.168.5.100 -- renewal in 3235 seconds.
Nov 11 16:40:03 truenas systemd[1]: Starting sysstat-collect.service - system activity accounting tool...
Nov 11 16:40:03 truenas systemd[1]: sysstat-collect.service: Deactivated successfully.
Nov 11 16:40:03 truenas systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
So basically it shows me nothing.
Doing an import to that TXG I get some errors in my journal:
Nov 11 17:07:28 truenas kernel: INFO: task IoThread:3055 blocked for more than 120 seconds.
Nov 11 17:07:28 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:07:28 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:07:28 truenas kernel: task:IoThread state:D stack:0 pid:3055 tgid:3005 ppid:1 flags:0x00000002
Nov 11 17:07:28 truenas kernel: Call Trace:
Nov 11 17:07:28 truenas kernel: <TASK>
Nov 11 17:07:28 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:07:28 truenas kernel: schedule+0x27/0xd0
Nov 11 17:07:28 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:07:28 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:07:28 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:07:28 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:07:28 truenas kernel: ? entity_namecheck+0xaa/0x250 [zfs]
Nov 11 17:07:28 truenas kernel: pool_status_check.constprop.0+0x5d/0xf0 [zfs]
Nov 11 17:07:28 truenas kernel: zfsdev_ioctl_common+0x46e/0x6c0 [zfs]
Nov 11 17:07:28 truenas kernel: ? __kvmalloc_node_noprof+0x40/0xd0
Nov 11 17:07:28 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:07:28 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:07:28 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:07:28 truenas kernel: ? exc_page_fault+0x76/0x190
Nov 11 17:07:28 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:07:28 truenas kernel: RIP: 0033:0x7f61b0954d1b
Nov 11 17:07:28 truenas kernel: RSP: 002b:00007f6171fce770 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:07:28 truenas kernel: RAX: ffffffffffffffda RBX: 00007f6171fce7f0 RCX: 00007f61b0954d1b
Nov 11 17:07:28 truenas kernel: RDX: 00007f6171fce7f0 RSI: 0000000000005a12 RDI: 0000000000000015
Nov 11 17:07:28 truenas kernel: RBP: 00007f6171fce7e0 R08: 00007f6164000180 R09: 00007f6164000180
Nov 11 17:07:28 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f6164001280
Nov 11 17:07:28 truenas kernel: R13: 00007f6164001280 R14: 00007f61645af6a0 R15: 0000000000000000
Nov 11 17:07:28 truenas kernel: </TASK>
Nov 11 17:07:28 truenas kernel: INFO: task middlewared (wo:3073 blocked for more than 120 seconds.
Nov 11 17:07:28 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:07:28 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:07:28 truenas kernel: task:middlewared (wo state:D stack:0 pid:3073 tgid:3073 ppid:3005 flags:0x00000002
Nov 11 17:07:28 truenas kernel: Call Trace:
Nov 11 17:07:28 truenas kernel: <TASK>
Nov 11 17:07:28 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:07:28 truenas kernel: schedule+0x27/0xd0
Nov 11 17:07:28 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:07:28 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:07:28 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:07:28 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:07:28 truenas kernel: ? __lruvec_stat_mod_folio+0xa4/0xd0
Nov 11 17:07:28 truenas kernel: spa_get_stats+0x4e/0x210 [zfs]
Nov 11 17:07:28 truenas kernel: zfs_ioc_pool_stats+0x40/0x90 [zfs]
Nov 11 17:07:28 truenas kernel: zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
Nov 11 17:07:28 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:07:28 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:07:28 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:07:28 truenas kernel: ? __count_memcg_events+0x53/0xf0
Nov 11 17:07:28 truenas kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 11 17:07:28 truenas kernel: ? handle_mm_fault+0xae/0x2d0
Nov 11 17:07:28 truenas kernel: ? do_user_addr_fault+0x379/0x670
Nov 11 17:07:28 truenas kernel: ? exc_page_fault+0x76/0x190
Nov 11 17:07:28 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:07:28 truenas kernel: RIP: 0033:0x7f4d53b3cd1b
Nov 11 17:07:28 truenas kernel: RSP: 002b:00007fff90bfa260 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:07:28 truenas kernel: RAX: ffffffffffffffda RBX: 00000000405c59c0 RCX: 00007f4d53b3cd1b
Nov 11 17:07:28 truenas kernel: RDX: 00007fff90bfa2e0 RSI: 0000000000005a05 RDI: 0000000000000013
Nov 11 17:07:28 truenas kernel: RBP: 00007fff90bfd8d0 R08: 000000004022ffa0 R09: 00007f4d53c11d50
Nov 11 17:07:28 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff90bfa2e0
Nov 11 17:07:28 truenas kernel: R13: 00000000405c59c0 R14: 00000000409761a0 R15: 00007fff90bfd8f4
Nov 11 17:07:28 truenas kernel: </TASK>
Nov 11 17:09:28 truenas kernel: INFO: task IoThread:3055 blocked for more than 241 seconds.
Nov 11 17:09:28 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:09:29 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:09:29 truenas kernel: task:IoThread state:D stack:0 pid:3055 tgid:3005 ppid:1 flags:0x00000002
Nov 11 17:09:29 truenas kernel: Call Trace:
Nov 11 17:09:29 truenas kernel: <TASK>
Nov 11 17:09:29 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:09:29 truenas kernel: schedule+0x27/0xd0
Nov 11 17:09:29 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:09:29 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:09:29 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:09:29 truenas kernel: ? entity_namecheck+0xaa/0x250 [zfs]
Nov 11 17:09:29 truenas kernel: pool_status_check.constprop.0+0x5d/0xf0 [zfs]
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl_common+0x46e/0x6c0 [zfs]
Nov 11 17:09:29 truenas kernel: ? __kvmalloc_node_noprof+0x40/0xd0
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:09:29 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:09:29 truenas kernel: ? exc_page_fault+0x76/0x190
Nov 11 17:09:29 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:09:29 truenas kernel: RIP: 0033:0x7f61b0954d1b
Nov 11 17:09:29 truenas kernel: RSP: 002b:00007f6171fce770 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:09:29 truenas kernel: RAX: ffffffffffffffda RBX: 00007f6171fce7f0 RCX: 00007f61b0954d1b
Nov 11 17:09:29 truenas kernel: RDX: 00007f6171fce7f0 RSI: 0000000000005a12 RDI: 0000000000000015
Nov 11 17:09:29 truenas kernel: RBP: 00007f6171fce7e0 R08: 00007f6164000180 R09: 00007f6164000180
Nov 11 17:09:29 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f6164001280
Nov 11 17:09:29 truenas kernel: R13: 00007f6164001280 R14: 00007f61645af6a0 R15: 0000000000000000
Nov 11 17:09:29 truenas kernel: </TASK>
Nov 11 17:09:29 truenas kernel: INFO: task IoThread:5231 blocked for more than 120 seconds.
Nov 11 17:09:29 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:09:29 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:09:29 truenas kernel: task:IoThread state:D stack:0 pid:5231 tgid:3005 ppid:1 flags:0x00000002
Nov 11 17:09:29 truenas kernel: Call Trace:
Nov 11 17:09:29 truenas kernel: <TASK>
Nov 11 17:09:29 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:09:29 truenas kernel: schedule+0x27/0xd0
Nov 11 17:09:29 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:09:29 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:09:29 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:09:29 truenas kernel: ? entity_namecheck+0xaa/0x250 [zfs]
Nov 11 17:09:29 truenas kernel: pool_status_check.constprop.0+0x5d/0xf0 [zfs]
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl_common+0x46e/0x6c0 [zfs]
Nov 11 17:09:29 truenas kernel: ? __kvmalloc_node_noprof+0x40/0xd0
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:09:29 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:09:29 truenas kernel: ? __pte_offset_map+0x1b/0x180
Nov 11 17:09:29 truenas kernel: ? __handle_mm_fault+0xaf8/0xf90
Nov 11 17:09:29 truenas kernel: ? __count_memcg_events+0x53/0xf0
Nov 11 17:09:29 truenas kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 11 17:09:29 truenas kernel: ? handle_mm_fault+0xae/0x2d0
Nov 11 17:09:29 truenas kernel: ? do_user_addr_fault+0x379/0x670
Nov 11 17:09:29 truenas kernel: ? exc_page_fault+0x76/0x190
Nov 11 17:09:29 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:09:29 truenas kernel: RIP: 0033:0x7f61b0954d1b
Nov 11 17:09:29 truenas kernel: RSP: 002b:00007f610a6f7400 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:09:29 truenas kernel: RAX: ffffffffffffffda RBX: 00007f610a6f7480 RCX: 00007f61b0954d1b
Nov 11 17:09:29 truenas kernel: RDX: 00007f610a6f7480 RSI: 0000000000005a12 RDI: 0000000000000051
Nov 11 17:09:29 truenas kernel: RBP: 00007f610a6f7470 R08: 00007f60f401fe10 R09: 00007f60f40002a0
Nov 11 17:09:29 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f60f4001280
Nov 11 17:09:29 truenas kernel: R13: 00007f60f4001280 R14: 00007f60f4009570 R15: 0000000000000000
Nov 11 17:09:29 truenas kernel: </TASK>
Nov 11 17:09:29 truenas kernel: INFO: task middlewared (wo:3073 blocked for more than 241 seconds.
Nov 11 17:09:29 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:09:29 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:09:29 truenas kernel: task:middlewared (wo state:D stack:0 pid:3073 tgid:3073 ppid:3005 flags:0x00000002
Nov 11 17:09:29 truenas kernel: Call Trace:
Nov 11 17:09:29 truenas kernel: <TASK>
Nov 11 17:09:29 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:09:29 truenas kernel: schedule+0x27/0xd0
Nov 11 17:09:29 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:09:29 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:09:29 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:09:29 truenas kernel: ? __lruvec_stat_mod_folio+0xa4/0xd0
Nov 11 17:09:29 truenas kernel: spa_get_stats+0x4e/0x210 [zfs]
Nov 11 17:09:29 truenas kernel: zfs_ioc_pool_stats+0x40/0x90 [zfs]
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:09:29 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:09:29 truenas kernel: ? __count_memcg_events+0x53/0xf0
Nov 11 17:09:29 truenas kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 11 17:09:29 truenas kernel: ? handle_mm_fault+0xae/0x2d0
Nov 11 17:09:29 truenas kernel: ? do_user_addr_fault+0x379/0x670
Nov 11 17:09:29 truenas kernel: ? exc_page_fault+0x76/0x190
Nov 11 17:09:29 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:09:29 truenas kernel: RIP: 0033:0x7f4d53b3cd1b
Nov 11 17:09:29 truenas kernel: RSP: 002b:00007fff90bfa260 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:09:29 truenas kernel: RAX: ffffffffffffffda RBX: 00000000405c59c0 RCX: 00007f4d53b3cd1b
Nov 11 17:09:29 truenas kernel: RDX: 00007fff90bfa2e0 RSI: 0000000000005a05 RDI: 0000000000000013
Nov 11 17:09:29 truenas kernel: RBP: 00007fff90bfd8d0 R08: 000000004022ffa0 R09: 00007f4d53c11d50
Nov 11 17:09:29 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff90bfa2e0
Nov 11 17:09:29 truenas kernel: R13: 00000000405c59c0 R14: 00000000409761a0 R15: 00007fff90bfd8f4
Nov 11 17:09:29 truenas kernel: </TASK>
Nov 11 17:09:29 truenas kernel: INFO: task middlewared (wo:8830 blocked for more than 120 seconds.
Nov 11 17:09:29 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:09:29 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:09:29 truenas kernel: task:middlewared (wo state:D stack:0 pid:8830 tgid:8830 ppid:3005 flags:0x00000002
Nov 11 17:09:29 truenas kernel: Call Trace:
Nov 11 17:09:29 truenas kernel: <TASK>
Nov 11 17:09:29 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:09:29 truenas kernel: schedule+0x27/0xd0
Nov 11 17:09:29 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:09:29 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:09:29 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:09:29 truenas kernel: ? __lruvec_stat_mod_folio+0xa4/0xd0
Nov 11 17:09:29 truenas kernel: spa_get_stats+0x4e/0x210 [zfs]
Nov 11 17:09:29 truenas kernel: zfs_ioc_pool_stats+0x40/0x90 [zfs]
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:09:29 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:09:29 truenas kernel: ? do_fault+0x281/0x4d0
Nov 11 17:09:29 truenas kernel: ? __handle_mm_fault+0x7bb/0xf90
Nov 11 17:09:29 truenas kernel: ? __count_memcg_events+0x53/0xf0
Nov 11 17:09:29 truenas kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 11 17:09:29 truenas kernel: ? handle_mm_fault+0xae/0x2d0
Nov 11 17:09:29 truenas kernel: ? do_user_addr_fault+0x379/0x670
Nov 11 17:09:29 truenas kernel: ? exc_page_fault+0x76/0x190
Nov 11 17:09:29 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:09:29 truenas kernel: RIP: 0033:0x7f9e7cfe8d1b
Nov 11 17:09:29 truenas kernel: RSP: 002b:00007fff1a4cac50 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:09:29 truenas kernel: RAX: ffffffffffffffda RBX: 00000000407dbbc0 RCX: 00007f9e7cfe8d1b
Nov 11 17:09:29 truenas kernel: RDX: 00007fff1a4cacd0 RSI: 0000000000005a05 RDI: 000000000000001b
Nov 11 17:09:29 truenas kernel: RBP: 00007fff1a4ce2c0 R08: 00000000407f21e0 R09: 00007f9e7d0bdd50
Nov 11 17:09:29 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff1a4cacd0
Nov 11 17:09:29 truenas kernel: R13: 00000000407dbbc0 R14: 0000000040b75d70 R15: 00007fff1a4ce2e4
Nov 11 17:09:29 truenas kernel: </TASK>
Nov 11 17:09:29 truenas kernel: INFO: task middlewared (wo:8880 blocked for more than 120 seconds.
Nov 11 17:09:29 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:09:29 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:09:29 truenas kernel: task:middlewared (wo state:D stack:0 pid:8880 tgid:8880 ppid:3005 flags:0x00000002
Nov 11 17:09:29 truenas kernel: Call Trace:
Nov 11 17:09:29 truenas kernel: <TASK>
Nov 11 17:09:29 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:09:29 truenas kernel: schedule+0x27/0xd0
Nov 11 17:09:29 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:09:29 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:09:29 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:09:29 truenas kernel: ? __lruvec_stat_mod_folio+0xa4/0xd0
Nov 11 17:09:29 truenas kernel: spa_get_stats+0x4e/0x210 [zfs]
Nov 11 17:09:29 truenas kernel: zfs_ioc_pool_stats+0x40/0x90 [zfs]
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:09:29 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:09:29 truenas kernel: ? __count_memcg_events+0x53/0xf0
Nov 11 17:09:29 truenas kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 11 17:09:29 truenas kernel: ? handle_mm_fault+0xae/0x2d0
Nov 11 17:09:29 truenas kernel: ? do_user_addr_fault+0x379/0x670
Nov 11 17:09:29 truenas kernel: ? exc_page_fault+0x76/0x190
Nov 11 17:09:29 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:09:29 truenas kernel: RIP: 0033:0x7f5a5c63fd1b
Nov 11 17:09:29 truenas kernel: RSP: 002b:00007ffe73fbb1f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:09:29 truenas kernel: RAX: ffffffffffffffda RBX: 0000000003406d20 RCX: 00007f5a5c63fd1b
Nov 11 17:09:29 truenas kernel: RDX: 00007ffe73fbb270 RSI: 0000000000005a05 RDI: 000000000000001b
Nov 11 17:09:29 truenas kernel: RBP: 00007ffe73fbe860 R08: 00000000034bad80 R09: 00007f5a5c714d50
Nov 11 17:09:29 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe73fbb270
Nov 11 17:09:29 truenas kernel: R13: 0000000003406d20 R14: 00000000037a2430 R15: 00007ffe73fbe884
Nov 11 17:09:29 truenas kernel: </TASK>
Nov 11 17:09:29 truenas kernel: INFO: task zfs:8980 blocked for more than 120 seconds.
Nov 11 17:09:29 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:09:29 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:09:29 truenas kernel: task:zfs state:D stack:0 pid:8980 tgid:8980 ppid:5299 flags:0x00000002
Nov 11 17:09:29 truenas kernel: Call Trace:
Nov 11 17:09:29 truenas kernel: <TASK>
Nov 11 17:09:29 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:09:29 truenas kernel: schedule+0x27/0xd0
Nov 11 17:09:29 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:09:29 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:09:29 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:09:29 truenas kernel: ? entity_namecheck+0xaa/0x250 [zfs]
Nov 11 17:09:29 truenas kernel: pool_status_check.constprop.0+0x5d/0xf0 [zfs]
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl_common+0x46e/0x6c0 [zfs]
Nov 11 17:09:29 truenas kernel: ? __kvmalloc_node_noprof+0x40/0xd0
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:09:29 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:09:29 truenas kernel: ? __count_memcg_events+0x53/0xf0
Nov 11 17:09:29 truenas kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 11 17:09:29 truenas kernel: ? handle_mm_fault+0xae/0x2d0
Nov 11 17:09:29 truenas kernel: ? do_user_addr_fault+0x379/0x670
Nov 11 17:09:29 truenas kernel: ? exc_page_fault+0x76/0x190
Nov 11 17:09:29 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:09:29 truenas kernel: RIP: 0033:0x7f8ff653dd1b
Nov 11 17:09:29 truenas kernel: RSP: 002b:00007ffc729c7be0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:09:29 truenas kernel: RAX: ffffffffffffffda RBX: 00007ffc729c7c60 RCX: 00007f8ff653dd1b
Nov 11 17:09:29 truenas kernel: RDX: 00007ffc729c7c60 RSI: 0000000000005a12 RDI: 0000000000000003
Nov 11 17:09:29 truenas kernel: RBP: 00007ffc729c7c50 R08: 0000000000060421 R09: 0000000000000000
Nov 11 17:09:29 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000055ca133dd2c0
Nov 11 17:09:29 truenas kernel: R13: 000055ca133dd2c0 R14: 000055ca133edfa0 R15: 0000000000000000
Nov 11 17:09:29 truenas kernel: </TASK>
Nov 11 17:10:04 truenas systemd[1]: Starting sysstat-collect.service - system activity accounting tool...
Nov 11 17:10:04 truenas systemd[1]: sysstat-collect.service: Deactivated successfully.
Nov 11 17:10:04 truenas systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
Nov 11 17:11:29 truenas kernel: INFO: task IoThread:3055 blocked for more than 362 seconds.
Nov 11 17:11:29 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:11:29 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:11:29 truenas kernel: task:IoThread state:D stack:0 pid:3055 tgid:3005 ppid:1 flags:0x00000002
Nov 11 17:11:29 truenas kernel: Call Trace:
Nov 11 17:11:29 truenas kernel: <TASK>
Nov 11 17:11:29 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:11:29 truenas kernel: schedule+0x27/0xd0
Nov 11 17:11:29 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:11:29 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:11:29 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:11:29 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:11:29 truenas kernel: ? entity_namecheck+0xaa/0x250 [zfs]
Nov 11 17:11:29 truenas kernel: pool_status_check.constprop.0+0x5d/0xf0 [zfs]
Nov 11 17:11:29 truenas kernel: zfsdev_ioctl_common+0x46e/0x6c0 [zfs]
Nov 11 17:11:29 truenas kernel: ? __kvmalloc_node_noprof+0x40/0xd0
Nov 11 17:11:29 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:11:29 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:11:29 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:11:29 truenas kernel: ? exc_page_fault+0x76/0x190
Nov 11 17:11:29 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:11:29 truenas kernel: RIP: 0033:0x7f61b0954d1b
Nov 11 17:11:29 truenas kernel: RSP: 002b:00007f6171fce770 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:11:29 truenas kernel: RAX: ffffffffffffffda RBX: 00007f6171fce7f0 RCX: 00007f61b0954d1b
Nov 11 17:11:29 truenas kernel: RDX: 00007f6171fce7f0 RSI: 0000000000005a12 RDI: 0000000000000015
Nov 11 17:11:29 truenas kernel: RBP: 00007f6171fce7e0 R08: 00007f6164000180 R09: 00007f6164000180
Nov 11 17:11:29 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f6164001280
Nov 11 17:11:29 truenas kernel: R13: 00007f6164001280 R14: 00007f61645af6a0 R15: 0000000000000000
Nov 11 17:11:29 truenas kernel: </TASK>
Nov 11 17:11:29 truenas kernel: INFO: task IoThread:3212 blocked for more than 120 seconds.
Nov 11 17:11:29 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:11:29 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:11:29 truenas kernel: task:IoThread state:D stack:0 pid:3212 tgid:3005 ppid:1 flags:0x00000002
Nov 11 17:11:29 truenas kernel: Call Trace:
Nov 11 17:11:29 truenas kernel: <TASK>
Nov 11 17:11:29 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:11:29 truenas kernel: schedule+0x27/0xd0
Nov 11 17:11:29 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:11:29 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:11:29 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:11:29 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:11:29 truenas kernel: ? entity_namecheck+0xaa/0x250 [zfs]
Nov 11 17:11:29 truenas kernel: pool_status_check.constprop.0+0x5d/0xf0 [zfs]
Nov 11 17:11:29 truenas kernel: zfsdev_ioctl_common+0x46e/0x6c0 [zfs]
Nov 11 17:11:29 truenas kernel: ? __kvmalloc_node_noprof+0x40/0xd0
Nov 11 17:11:29 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:11:29 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:11:29 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:11:29 truenas kernel: ? reweight_entity+0x1c1/0x260
Nov 11 17:11:29 truenas kernel: ? effective_cpu_util+0x2f/0xd0
Nov 11 17:11:29 truenas kernel: ? sugov_get_util+0x4d/0xa0
Nov 11 17:11:29 truenas kernel: ? sugov_update_single_freq+0xf4/0x230
Nov 11 17:11:29 truenas kernel: ? sched_balance_update_blocked_averages+0x54b/0x630
Nov 11 17:11:29 truenas kernel: ? sched_balance_domains+0xed/0x3a0
Nov 11 17:11:29 truenas kernel: ? handle_softirqs+0xd0/0x2c0
Nov 11 17:11:29 truenas kernel: ? __irq_exit_rcu+0x38/0xb0
Nov 11 17:11:29 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:11:29 truenas kernel: RIP: 0033:0x7f61b0954d1b
Nov 11 17:11:29 truenas kernel: RSP: 002b:00007f614cff4770 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:11:29 truenas kernel: RAX: ffffffffffffffda RBX: 00007f614cff47f0 RCX: 00007f61b0954d1b
Nov 11 17:11:29 truenas kernel: RDX: 00007f614cff47f0 RSI: 0000000000005a12 RDI: 0000000000000043
Nov 11 17:11:29 truenas kernel: RBP: 00007f614cff47e0 R08: 00007f6130056ee0 R09: 00007f6130000280
Nov 11 17:11:29 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f6130001280
Nov 11 17:11:29 truenas kernel: R13: 00007f6130001280 R14: 00007f613005c980 R15: 0000000000000000
Nov 11 17:11:29 truenas kernel: </TASK>
Nov 11 17:11:29 truenas kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
Relevant part:
Nov 11 17:09:29 truenas kernel: INFO: task zfs:8980 blocked for more than 120 seconds.
Nov 11 17:09:29 truenas kernel: Tainted: P OE 6.12.33-production+truenas #1
Nov 11 17:09:29 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 17:09:29 truenas kernel: task:zfs state:D stack:0 pid:8980 tgid:8980 ppid:5299 flags:0x00000002
Nov 11 17:09:29 truenas kernel: Call Trace:
Nov 11 17:09:29 truenas kernel: <TASK>
Nov 11 17:09:29 truenas kernel: __schedule+0x461/0xa10
Nov 11 17:09:29 truenas kernel: schedule+0x27/0xd0
Nov 11 17:09:29 truenas kernel: cv_wait_common+0xef/0x130 [spl]
Nov 11 17:09:29 truenas kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Nov 11 17:09:29 truenas kernel: spa_lookup+0x3e/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: spa_open_common+0x75/0x440 [zfs]
Nov 11 17:09:29 truenas kernel: ? entity_namecheck+0xaa/0x250 [zfs]
Nov 11 17:09:29 truenas kernel: pool_status_check.constprop.0+0x5d/0xf0 [zfs]
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl_common+0x46e/0x6c0 [zfs]
Nov 11 17:09:29 truenas kernel: ? __kvmalloc_node_noprof+0x40/0xd0
Nov 11 17:09:29 truenas kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Nov 11 17:09:29 truenas kernel: __x64_sys_ioctl+0x94/0xd0
Nov 11 17:09:29 truenas kernel: do_syscall_64+0x82/0x190
Nov 11 17:09:29 truenas kernel: ? __count_memcg_events+0x53/0xf0
Nov 11 17:09:29 truenas kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 11 17:09:29 truenas kernel: ? handle_mm_fault+0xae/0x2d0
Nov 11 17:09:29 truenas kernel: ? do_user_addr_fault+0x379/0x670
Nov 11 17:09:29 truenas kernel: ? exc_page_fault+0x76/0x190
Nov 11 17:09:29 truenas kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 11 17:09:29 truenas kernel: RIP: 0033:0x7f8ff653dd1b
Nov 11 17:09:29 truenas kernel: RSP: 002b:00007ffc729c7be0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 11 17:09:29 truenas kernel: RAX: ffffffffffffffda RBX: 00007ffc729c7c60 RCX: 00007f8ff653dd1b
Nov 11 17:09:29 truenas kernel: RDX: 00007ffc729c7c60 RSI: 0000000000005a12 RDI: 0000000000000003
Nov 11 17:09:29 truenas kernel: RBP: 00007ffc729c7c50 R08: 0000000000060421 R09: 0000000000000000
Nov 11 17:09:29 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000055ca133dd2c0
Nov 11 17:09:29 truenas kernel: R13: 000055ca133dd2c0 R14: 000055ca133edfa0 R15: 0000000000000000
Nov 11 17:09:29 truenas kernel: </TASK>
Does it eventually throw a full on PANIC or is it still thinking away here?
1 Like
Also bumping this again @itractus if you have time to fill me in on the hardware details here.
It’s still working on it, just started it 10 mins ago, cpu load is increasing so i’m optimisic.
1 Like
Check your drive stats more than CPU, see if your dedup or meta vdevs are steamrolling along at full throttle.
1 Like
Absolutely:
The nvme’s are on a carrier card on a bifurcated pcie X16 slot, the netapp chassis is connected over an lsi 3008 HBA . the other 4 SSD’s are in the front loader, so connected to the HPE raid card or sas thing built in.
lspcie:
root@truenas[/home/truenas_admin]# lspci
00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 01)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01)
00:01.1 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
00:02.1 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
00:02.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
00:02.3 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
00:03.1 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
00:03.3 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
00:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)
00:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01)
00:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01)
00:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01)
00:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01)
00:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01)
00:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01)
00:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01)
00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 01)
00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 01)
00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 01)
00:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC (rev 01)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.2 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #3 (rev d5)
00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 (rev d5)
00:1c.6 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 (rev d5)
00:1c.7 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #8 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 06)
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH (rev 01)
01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 06)
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 03)
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
02:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
03:00.0 Serial Attached SCSI controller: Hewlett-Packard Company Smart Array Gen9 Controllers (rev 01)
05:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
08:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD
0b:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD
7f:08.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01)
7f:08.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01)
7f:09.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01)
7f:09.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01)
7f:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01)
7f:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01)
7f:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01)
7f:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug (rev 01)
7f:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0c.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0c.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0c.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0c.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0d.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0d.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0f.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
7f:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 01)
7f:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 01)
7f:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 01)
7f:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 01)
7f:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 01)
7f:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 01)
7f:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 01)
7f:12.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 Debug (rev 01)
7f:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 01)
7f:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 01)
7f:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01)
7f:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01)
7f:13.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01)
7f:13.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01)
7f:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 01)
7f:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 01)
7f:14.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control (rev 01)
7f:14.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control (rev 01)
7f:14.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error (rev 01)
7f:14.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error (rev 01)
7f:14.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01)
7f:14.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01)
7f:14.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01)
7f:14.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01)
7f:15.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control (rev 01)
7f:15.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control (rev 01)
7f:15.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Error (rev 01)
7f:15.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Error (rev 01)
7f:16.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS (rev 01)
7f:16.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Broadcast (rev 01)
7f:16.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 01)
7f:17.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 0 Thermal Control (rev 01)
7f:17.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01)
7f:17.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01)
7f:17.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01)
7f:17.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01)
7f:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
7f:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
7f:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
7f:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
7f:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
7f:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
7f:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
80:00.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 0 (rev 01)
80:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01)
80:01.1 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01)
80:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
80:02.1 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
80:02.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
80:02.3 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
80:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
80:03.1 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
80:03.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
80:03.3 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
80:04.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 0 (rev 01)
80:04.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 1 (rev 01)
80:04.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 2 (rev 01)
80:04.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 3 (rev 01)
80:04.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 4 (rev 01)
80:04.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 5 (rev 01)
80:04.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 6 (rev 01)
80:04.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Crystal Beach DMA Channel 7 (rev 01)
80:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 01)
80:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 01)
80:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 01)
80:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC (rev 01)
82:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
82:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
ff:08.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01)
ff:08.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01)
ff:09.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01)
ff:09.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01)
ff:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01)
ff:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01)
ff:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01)
ff:0b.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug (rev 01)
ff:0c.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0c.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0c.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0c.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0c.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0c.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0c.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0c.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0d.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0d.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0f.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0f.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0f.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:0f.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent (rev 01)
ff:10.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 01)
ff:10.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent (rev 01)
ff:10.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 01)
ff:10.6 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 01)
ff:10.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox (rev 01)
ff:12.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 01)
ff:12.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 (rev 01)
ff:12.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 Debug (rev 01)
ff:13.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 01)
ff:13.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS (rev 01)
ff:13.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01)
ff:13.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01)
ff:13.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01)
ff:13.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder (rev 01)
ff:13.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast (rev 01)
ff:13.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 01)
ff:14.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control (rev 01)
ff:14.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control (rev 01)
ff:14.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error (rev 01)
ff:14.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error (rev 01)
ff:14.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01)
ff:14.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01)
ff:14.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01)
ff:14.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface (rev 01)
ff:15.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control (rev 01)
ff:15.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control (rev 01)
ff:15.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Error (rev 01)
ff:15.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Error (rev 01)
ff:16.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS (rev 01)
ff:16.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Broadcast (rev 01)
ff:16.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast (rev 01)
ff:17.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 0 Thermal Control (rev 01)
ff:17.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01)
ff:17.5 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01)
ff:17.6 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01)
ff:17.7 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface (rev 01)
ff:1e.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
ff:1e.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
ff:1e.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
ff:1e.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
ff:1e.4 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
ff:1f.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
ff:1f.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit (rev 01)
There is some usage on my meta vdev disks… minimal… some serious reads from one of the data vdev drives and some writes to another… I guess i’ll let it cook. Atleast it’s not crashing!
03:00.0 Serial Attached SCSI controller: Hewlett-Packard Company Smart Array Gen9 Controllers (rev 01)
That’s what I was worried about. Can you lspci -vv | grep 03:00.0 -A60 so I can see the exact VID/PID model and driver loaded?
If you know it offhand that works as well - it’s also the question of “are you using a virtual RAID0 disk set up via RAID BIOS, or is it in quasi-HBA imitation model”?
1 Like
I didn’t configure any virtual raid or pools on the storage controller, these are all bare drives connecting to truenas.
Here is the pci info:
root@truenas[/home/truenas_admin]# lspci -vv | grep 03:00.0 -A60
03:00.0 Serial Attached SCSI controller: Hewlett-Packard Company Smart Array Gen9 Controllers (rev 01)
DeviceName: Embedded RAID 1
Subsystem: Hewlett-Packard Company P440ar
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
NUMA node: 0
IOMMU group: 56
Region 0: Memory at 93400000 (64-bit, non-prefetchable) [size=1M]
Region 2: Memory at 93500000 (64-bit, non-prefetchable) [size=1K]
Region 4: I/O ports at 3000 [size=256]
Expansion ROM at 93580000 [virtual] [disabled] [size=512K]
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [b0] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003000
Capabilities: [c0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Via message, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp+ ExtTPHComp-
AtomicOpsCap: 32bit+ 64bit+ 128bitCAS+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [300 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Kernel driver in use: hpsa
From ILO:
Embedded Smart Array P440ar Controller 749796-001 N/A PDNLH0BRH8N3H9 B 6.60 OK
So up untill 20 mins ago, there was drive activity. Now i log back in and see the uptime is 20 mins, the pool is not imported and we’re back at the same point sadly…
So no luck yet.