在win10通过smb整理文件时进度卡死，然后重启（从UI操作）之后挂载不上pool/Progress stuck while organizing files via smb in win10, then after reboot (operating from UI) the mount won't go on pool

leng_sok · July 21, 2025, 6:00pm

Thanks for your reply

Today I imported the old system’s configuration file into 24.10.2.3 and added two 32G memory to start it up. The UI is still stuck and these two processes: pool.import_on_boot and pool.dataset.sync_db_keys are both 0%
It will still restart after filling up the memory.

This is the latest status of the dev member

root@truenas[~]# zdb -l /dev/sdc2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'sok1001'
    state: 0
    txg: 6814766
    pool_guid: 1221631789954295656
    errata: 0
    hostid: 1981761578
    hostname: 'truenas'
    top_guid: 1132576499012000637
    guid: 15237958642688602726
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 1132576499012000637
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 95992471879680
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 705601381597961486
            path: '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6'
            whole_disk: 0
            DTL: 37037
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 15682408288857988643
            path: '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab'
            whole_disk: 0
            DTL: 37036
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 16097378473779213230
            path: '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70'
            whole_disk: 0
            DTL: 37035
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 10339353618853963483
            path: '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb'
            whole_disk: 0
            DTL: 37034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 4278422679801059362
            path: '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d'
            whole_disk: 0
            DTL: 37033
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15237958642688602726
            path: '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464'
            whole_disk: 0
            DTL: 37032
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdd2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'sok1001'
    state: 0
    txg: 6814766
    pool_guid: 1221631789954295656
    errata: 0
    hostid: 1981761578
    hostname: 'truenas'
    top_guid: 1132576499012000637
    guid: 4278422679801059362
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 1132576499012000637
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 95992471879680
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 705601381597961486
            path: '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6'
            whole_disk: 0
            DTL: 37037
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 15682408288857988643
            path: '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab'
            whole_disk: 0
            DTL: 37036
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 16097378473779213230
            path: '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70'
            whole_disk: 0
            DTL: 37035
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 10339353618853963483
            path: '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb'
            whole_disk: 0
            DTL: 37034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 4278422679801059362
            path: '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d'
            whole_disk: 0
            DTL: 37033
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15237958642688602726
            path: '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464'
            whole_disk: 0
            DTL: 37032
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sde2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'sok1001'
    state: 0
    txg: 6814766
    pool_guid: 1221631789954295656
    errata: 0
    hostid: 1981761578
    hostname: 'truenas'
    top_guid: 1132576499012000637
    guid: 10339353618853963483
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 1132576499012000637
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 95992471879680
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 705601381597961486
            path: '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6'
            whole_disk: 0
            DTL: 37037
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 15682408288857988643
            path: '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab'
            whole_disk: 0
            DTL: 37036
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 16097378473779213230
            path: '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70'
            whole_disk: 0
            DTL: 37035
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 10339353618853963483
            path: '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb'
            whole_disk: 0
            DTL: 37034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 4278422679801059362
            path: '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d'
            whole_disk: 0
            DTL: 37033
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15237958642688602726
            path: '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464'
            whole_disk: 0
            DTL: 37032
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdf2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'sok1001'
    state: 0
    txg: 6814766
    pool_guid: 1221631789954295656
    errata: 0
    hostid: 1981761578
    hostname: 'truenas'
    top_guid: 1132576499012000637
    guid: 16097378473779213230
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 1132576499012000637
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 95992471879680
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 705601381597961486
            path: '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6'
            whole_disk: 0
            DTL: 37037
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 15682408288857988643
            path: '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab'
            whole_disk: 0
            DTL: 37036
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 16097378473779213230
            path: '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70'
            whole_disk: 0
            DTL: 37035
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 10339353618853963483
            path: '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb'
            whole_disk: 0
            DTL: 37034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 4278422679801059362
            path: '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d'
            whole_disk: 0
            DTL: 37033
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15237958642688602726
            path: '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464'
            whole_disk: 0
            DTL: 37032
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdg2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'sok1001'
    state: 0
    txg: 6814766
    pool_guid: 1221631789954295656
    errata: 0
    hostid: 1981761578
    hostname: 'truenas'
    top_guid: 1132576499012000637
    guid: 15682408288857988643
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 1132576499012000637
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 95992471879680
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 705601381597961486
            path: '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6'
            whole_disk: 0
            DTL: 37037
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 15682408288857988643
            path: '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab'
            whole_disk: 0
            DTL: 37036
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 16097378473779213230
            path: '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70'
            whole_disk: 0
            DTL: 37035
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 10339353618853963483
            path: '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb'
            whole_disk: 0
            DTL: 37034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 4278422679801059362
            path: '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d'
            whole_disk: 0
            DTL: 37033
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15237958642688602726
            path: '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464'
            whole_disk: 0
            DTL: 37032
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3 
root@truenas[~]# zdb -l /dev/sdh2
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'sok1001'
    state: 0
    txg: 6814766
    pool_guid: 1221631789954295656
    errata: 0
    hostid: 1981761578
    hostname: 'truenas'
    top_guid: 1132576499012000637
    guid: 705601381597961486
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 1132576499012000637
        nparity: 2
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 12
        asize: 95992471879680
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 705601381597961486
            path: '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6'
            whole_disk: 0
            DTL: 37037
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 15682408288857988643
            path: '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab'
            whole_disk: 0
            DTL: 37036
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 16097378473779213230
            path: '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70'
            whole_disk: 0
            DTL: 37035
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 10339353618853963483
            path: '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb'
            whole_disk: 0
            DTL: 37034
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 4278422679801059362
            path: '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d'
            whole_disk: 0
            DTL: 37033
            create_txg: 4
        children[5]:
            type: 'disk'
            id: 5
            guid: 15237958642688602726
            path: '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464'
            whole_disk: 0
            DTL: 37032
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3

Yes, I have turned on the data duplication function.
Yes, it uses the integrated SATA controller of the Xeon D processor itself, and the data cable is directly connected to the SATA port on the motherboard. But I have never encountered the situation where the hard disk is not recognized.

This system was sold to me as a complete set, so there should be no incompatible power supply or other hardware. (except the hard disk).

This is the log of the hard disks connected one by one after the system starts
don’t know if it’s useful.

Jul 22 00:48:29 truenas.local kernel: ata1: link is slow to respond, please be patient (ready=0)
Jul 22 00:48:39 truenas.local kernel: ata1: link is slow to respond, please be patient (ready=0)
Jul 22 00:48:41 truenas.local kernel: ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jul 22 00:48:41 truenas.local kernel: ata1.00: ATA-11: WDC  WUH721816ALE6L4, PCGNW232, max UDMA/133
Jul 22 00:48:41 truenas.local kernel: ata1.00: 31251759104 sectors, multi 0: LBA48 NCQ (depth 32), AA
Jul 22 00:48:41 truenas.local kernel: ata1.00: Features: NCQ-sndrcv NCQ-prio
Jul 22 00:48:41 truenas.local kernel: ata1.00: configured for UDMA/133
Jul 22 00:48:41 truenas.local kernel: scsi 0:0:0:0: Direct-Access     ATA      WDC  WUH721816AL W232 PQ: 0 ANSI: 5
Jul 22 00:48:41 truenas.local kernel: sd 0:0:0:0: Attached scsi generic sg4 type 0
Jul 22 00:48:41 truenas.local kernel: sd 0:0:0:0: [sdc] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
Jul 22 00:48:41 truenas.local kernel: sd 0:0:0:0: [sdc] 4096-byte physical blocks
Jul 22 00:48:41 truenas.local kernel: sd 0:0:0:0: [sdc] Write Protect is off
Jul 22 00:48:41 truenas.local kernel: sd 0:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jul 22 00:48:41 truenas.local kernel: sd 0:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes
Jul 22 00:48:41 truenas.local kernel:  sdc: sdc1 sdc2
Jul 22 00:48:41 truenas.local kernel: sd 0:0:0:0: [sdc] Attached SCSI disk
Jul 22 00:48:42 truenas.local systemd-journald[972]: Data hash table of /var/log/journal/acf7fd7a54ec4ad782fa59a63aaf6450/system.journal has a fill level at 75.0 (8534 of 11377 items, 6553600 file size, 767 bytes per hash table item), suggesting rotation.
Jul 22 00:48:42 truenas.local systemd-journald[972]: /var/log/journal/acf7fd7a54ec4ad782fa59a63aaf6450/system.journal: Journal header limits reached or header out-of-date, rotating.
Jul 22 00:48:43 truenas.local netdata[6982]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Jul 22 00:49:36 truenas.local kernel: ata2: link is slow to respond, please be patient (ready=0)
Jul 22 00:49:46 truenas.local kernel: ata2: link is slow to respond, please be patient (ready=0)
Jul 22 00:49:48 truenas.local kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jul 22 00:49:49 truenas.local kernel: ata2.00: ATA-11: WDC  WUH721816ALE6L4, PCGNW232, max UDMA/133
Jul 22 00:49:49 truenas.local kernel: ata2.00: 31251759104 sectors, multi 0: LBA48 NCQ (depth 32), AA
Jul 22 00:49:49 truenas.local kernel: ata2.00: Features: NCQ-sndrcv NCQ-prio
Jul 22 00:49:49 truenas.local kernel: ata2.00: configured for UDMA/133
Jul 22 00:49:49 truenas.local kernel: scsi 1:0:0:0: Direct-Access     ATA      WDC  WUH721816AL W232 PQ: 0 ANSI: 5
Jul 22 00:49:49 truenas.local kernel: sd 1:0:0:0: Attached scsi generic sg5 type 0
Jul 22 00:49:49 truenas.local kernel: sd 1:0:0:0: [sdd] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
Jul 22 00:49:49 truenas.local kernel: sd 1:0:0:0: [sdd] 4096-byte physical blocks
Jul 22 00:49:49 truenas.local kernel: sd 1:0:0:0: [sdd] Write Protect is off
Jul 22 00:49:49 truenas.local kernel: sd 1:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jul 22 00:49:49 truenas.local kernel: sd 1:0:0:0: [sdd] Preferred minimum I/O size 4096 bytes
Jul 22 00:49:49 truenas.local kernel:  sdd: sdd1 sdd2
Jul 22 00:49:49 truenas.local kernel: sd 1:0:0:0: [sdd] Attached SCSI disk
Jul 22 00:49:52 truenas.local netdata[7919]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Jul 22 00:50:07 truenas.local kernel: ata3: link is slow to respond, please be patient (ready=0)
Jul 22 00:50:17 truenas.local kernel: ata3: link is slow to respond, please be patient (ready=0)
Jul 22 00:50:19 truenas.local kernel: ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jul 22 00:50:20 truenas.local kernel: ata3.00: ATA-11: WDC  WUH721816ALE6L4, PCGNW232, max UDMA/133
Jul 22 00:50:20 truenas.local kernel: ata3.00: 31251759104 sectors, multi 0: LBA48 NCQ (depth 32), AA
Jul 22 00:50:20 truenas.local kernel: ata3.00: Features: NCQ-sndrcv NCQ-prio
Jul 22 00:50:20 truenas.local kernel: ata3.00: configured for UDMA/133
Jul 22 00:50:20 truenas.local kernel: scsi 2:0:0:0: Direct-Access     ATA      WDC  WUH721816AL W232 PQ: 0 ANSI: 5
Jul 22 00:50:20 truenas.local kernel: sd 2:0:0:0: Attached scsi generic sg6 type 0
Jul 22 00:50:20 truenas.local kernel: sd 2:0:0:0: [sde] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
Jul 22 00:50:20 truenas.local kernel: sd 2:0:0:0: [sde] 4096-byte physical blocks
Jul 22 00:50:20 truenas.local kernel: sd 2:0:0:0: [sde] Write Protect is off
Jul 22 00:50:20 truenas.local kernel: sd 2:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jul 22 00:50:20 truenas.local kernel: sd 2:0:0:0: [sde] Preferred minimum I/O size 4096 bytes
Jul 22 00:50:20 truenas.local kernel:  sde: sde1 sde2
Jul 22 00:50:20 truenas.local kernel: sd 2:0:0:0: [sde] Attached SCSI disk
Jul 22 00:50:23 truenas.local netdata[8572]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Jul 22 00:50:35 truenas.local kernel: ata4: link is slow to respond, please be patient (ready=0)
Jul 22 00:50:45 truenas.local kernel: ata4: link is slow to respond, please be patient (ready=0)
Jul 22 00:50:47 truenas.local kernel: ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jul 22 00:50:48 truenas.local kernel: ata4.00: ATA-11: WDC  WUH721816ALE6L4, PCGNW232, max UDMA/133
Jul 22 00:50:48 truenas.local kernel: ata4.00: 31251759104 sectors, multi 0: LBA48 NCQ (depth 32), AA
Jul 22 00:50:48 truenas.local kernel: ata4.00: Features: NCQ-sndrcv NCQ-prio
Jul 22 00:50:48 truenas.local kernel: ata4.00: configured for UDMA/133
Jul 22 00:50:48 truenas.local kernel: scsi 3:0:0:0: Direct-Access     ATA      WDC  WUH721816AL W232 PQ: 0 ANSI: 5
Jul 22 00:50:48 truenas.local kernel: sd 3:0:0:0: Attached scsi generic sg7 type 0
Jul 22 00:50:48 truenas.local kernel: sd 3:0:0:0: [sdf] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
Jul 22 00:50:48 truenas.local kernel: sd 3:0:0:0: [sdf] 4096-byte physical blocks
Jul 22 00:50:48 truenas.local kernel: sd 3:0:0:0: [sdf] Write Protect is off
Jul 22 00:50:48 truenas.local kernel: sd 3:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jul 22 00:50:48 truenas.local kernel: sd 3:0:0:0: [sdf] Preferred minimum I/O size 4096 bytes
Jul 22 00:50:48 truenas.local kernel:  sdf: sdf1 sdf2
Jul 22 00:50:48 truenas.local kernel: sd 3:0:0:0: [sdf] Attached SCSI disk
Jul 22 00:51:02 truenas.local netdata[9322]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Jul 22 00:51:14 truenas.local kernel: ata5: link is slow to respond, please be patient (ready=0)
Jul 22 00:51:24 truenas.local kernel: ata5: link is slow to respond, please be patient (ready=0)
Jul 22 00:51:26 truenas.local kernel: ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jul 22 00:51:27 truenas.local kernel: ata5.00: ATA-11: WDC  WUH721816ALE6L4, PCGNW232, max UDMA/133
Jul 22 00:51:27 truenas.local kernel: ata5.00: 31251759104 sectors, multi 0: LBA48 NCQ (depth 32), AA
Jul 22 00:51:27 truenas.local kernel: ata5.00: Features: NCQ-sndrcv NCQ-prio
Jul 22 00:51:27 truenas.local kernel: ata5.00: configured for UDMA/133
Jul 22 00:51:27 truenas.local kernel: scsi 4:0:0:0: Direct-Access     ATA      WDC  WUH721816AL W232 PQ: 0 ANSI: 5
Jul 22 00:51:27 truenas.local kernel: sd 4:0:0:0: Attached scsi generic sg8 type 0
Jul 22 00:51:27 truenas.local kernel: sd 4:0:0:0: [sdg] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
Jul 22 00:51:27 truenas.local kernel: sd 4:0:0:0: [sdg] 4096-byte physical blocks
Jul 22 00:51:27 truenas.local kernel: sd 4:0:0:0: [sdg] Write Protect is off
Jul 22 00:51:27 truenas.local kernel: sd 4:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jul 22 00:51:27 truenas.local kernel: sd 4:0:0:0: [sdg] Preferred minimum I/O size 4096 bytes
Jul 22 00:51:27 truenas.local kernel:  sdg: sdg1 sdg2
Jul 22 00:51:27 truenas.local kernel: sd 4:0:0:0: [sdg] Attached SCSI disk
Jul 22 00:51:39 truenas.local netdata[10370]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Jul 22 00:52:05 truenas.local kernel: ata6: link is slow to respond, please be patient (ready=0)
Jul 22 00:52:15 truenas.local kernel: ata6: link is slow to respond, please be patient (ready=0)
Jul 22 00:52:17 truenas.local kernel: ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jul 22 00:52:18 truenas.local kernel: ata6.00: ATA-11: WDC  WUH721816ALE6L4, PCGNW232, max UDMA/133
Jul 22 00:52:18 truenas.local kernel: ata6.00: 31251759104 sectors, multi 0: LBA48 NCQ (depth 32), AA
Jul 22 00:52:18 truenas.local kernel: ata6.00: Features: NCQ-sndrcv NCQ-prio
Jul 22 00:52:18 truenas.local kernel: ata6.00: configured for UDMA/133
Jul 22 00:52:18 truenas.local kernel: scsi 5:0:0:0: Direct-Access     ATA      WDC  WUH721816AL W232 PQ: 0 ANSI: 5
Jul 22 00:52:18 truenas.local kernel: sd 5:0:0:0: Attached scsi generic sg9 type 0
Jul 22 00:52:18 truenas.local kernel: sd 5:0:0:0: [sdh] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
Jul 22 00:52:18 truenas.local kernel: sd 5:0:0:0: [sdh] 4096-byte physical blocks
Jul 22 00:52:18 truenas.local kernel: sd 5:0:0:0: [sdh] Write Protect is off
Jul 22 00:52:18 truenas.local kernel: sd 5:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jul 22 00:52:18 truenas.local kernel: sd 5:0:0:0: [sdh] Preferred minimum I/O size 4096 bytes
Jul 22 00:52:18 truenas.local kernel:  sdh: sdh1 sdh2
Jul 22 00:52:18 truenas.local kernel: sd 5:0:0:0: [sdh] Attached SCSI disk
Jul 22 00:52:20 truenas.local netdata[11630]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.

So there is a bug in the sata controller of this motherboard?
What do you think I should do?
Replace the new motherboard like X10SDV-4C-TLN2F or have a HBA card?
or try to remove all drive from backplane and test agian with directly connect first?

etorix · July 21, 2025, 6:14pm

leng_sok:

root@truenas[~]# tail -n 200 /proc/spl/kstat/zfs/dbgmsg
1752994679   ffff9211a09fe500 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7024602, spa local date, vdev_id 0, ms_id 95, weight 700000000000001, selected txg 7024484 (604206 ms ago), alloc_txg 0, loaded 752661 ms ago, max_size 536100864

The first log excerpt even shows txgs 7024602 and 7024484. It’s like the labels were frozen days ago.

Deduplication on the whole 50 TB pool? Ouch! You’d need about 250 GB RAM with that (at standard recordsize) for smooth operation.

leng_sok · July 21, 2025, 6:17pm

But this motherboard only supports 128G at most, and I have already filled it up.
I read an old article that said 1TB needs 1G memory…

“local date” should be another pool
It is a normal pool, only have one ssd drive

root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sdb3      ONLINE       0     0     0

errors: No known data errors

  pool: local date
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:04:42 with 0 errors on Sun Jun  8 00:04:44 2025
config:

        NAME                                    STATE     READ WRITE CKSUM
        local date                              ONLINE       0     0     0
          e3a88fea-cb0d-40f1-94de-4b4c5dc36166  ONLINE       0     0     0

errors: No known data errors

etorix · July 21, 2025, 7:26pm

That’s the guidance for regular storage (and it relaxes as storage grows).
With dedup, the guidance is 5 GB per TB for smooth operation (with RAM only). You may do with less, but it will get terribly slow under load—which is probably how the pool got stuck when doing lots of change.

If you can recover the pool, I suggest to either

add a L2ARC;
add two or three SSDs in mirror as dedup vdev; or
remove dedup.

But first you need to recover.

HoneyBadger · July 21, 2025, 8:20pm

As pointed out by @etorix this is going to cause major issues with RAM pressure. At an estimated 80% full on a 6wZ2 pool of 16T disks, that’s about 38T of data before dedup - at the default 128K recordsize that gives about a 92G deduplication table, and it could be larger if some of the files are smaller than that. Maxing out the RAM at 128G may be sufficient to allow the pool to import eventually - but it may be that it’s tripping a deadman timer before being able to reach that point.

1753036266   ffff921294e5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 6823468, spa sok1001, vdev_id 0, ms_id 2660, smp_length 41280, unflushed_allocs 2153103360, unflushed_frees 442368, freed 0, defer 0 + 0, unloaded time 43408883 ms, loading_time 32 ms, ms_max_size 771137536, max size error 771100672, old_weight 740000000000001, new_weight 740000000000001
1753036266   ffff921294e5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 6823468, spa sok1001, vdev_id 0, ms_id 3287, smp_length 149240, unflushed_allocs 104779776, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 43408915 ms, loading_time 61 ms, ms_max_size 696844288, max size error 696844288, old_weight 740000000000001, new_weight 740000000000001
1753036266   ffff921294e5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 6823468, spa sok1001, vdev_id 0, ms_id 2570, smp_length 295088, unflushed_allocs 369057792, unflushed_frees 147456, freed 0, defer 0 + 0, unloaded time 43408977 ms, loading_time 94 ms, ms_max_size 452542464, max size error 452395008, old_weight 700000000000001, new_weight 700000000000001
1753036266   ffff921294e5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 6823468, spa sok1001, vdev_id 0, ms_id 2594, smp_length 1432, unflushed_allocs 948154368, unflushed_frees 1000980480, freed 0, defer 0 + 0, unloaded time 43409071 ms, loading_time 46 ms, ms_max_size 385769472, max size error 384417792, old_weight 700000000000001, new_weight 700000000000001
1753036266   ffff921294e5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 6823468, spa sok1001, vdev_id 0, ms_id 2635, smp_length 216216, unflushed_allocs 2454712320, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 43409119 ms, loading_time 63 ms, ms_max_size 491069440, max size error 491069440, old_weight 700000000000001, new_weight 700000000000001
1753036266   ffff921294e5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 6823468, spa sok1001, vdev_id 0, ms_id 3127, smp_length 9288, unflushed_allocs 2342596608, unflushed_frees 494702592, freed 0, defer 0 + 0, unloaded time 43409182 ms, loading_time 75 ms, ms_max_size 201265152, max size error 201142272, old_weight 6c0000000000001, new_weight 6c0000000000001
1753036266   ffff921294e5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 6823468, spa sok1001, vdev_id 0, ms_id 3253, smp_length 5920, unflushed_allocs 668098560, unflushed_frees 360062976, freed 0, defer 0 + 0, unloaded time 43409258 ms, loading_time 32 ms, ms_max_size 70078464, max size error 69009408, old_weight 680000000000001, new_weight 680000000000001

There’s ~9 billion unflushed_allocs and ~1.8 billion unflushed_frees in this chunk.

Were you doing a lot of new copies, in-place moves, or deletes?

Regardless.

We may be able to get read-only access by importing the pool in a read-only state at an earlier transaction group, where it isn’t trying to play “catch-up” with the pending txgs - it might mean discarding a lot of changes though, and you would still require the space needed afterwards to copy your data off (and presumably disable deduplication)

@leng_sok with the greater 128G RAM capacity what does the dbgmsg output look like?

leng_sok · July 21, 2025, 9:56pm

Yes, I am moving about 3T of media files from folder to another folder, they are all in the sok1001 pool

1753119846   ffff980748af3280 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753119846   ffff980748af3280 spa.c:6916:spa_tryimport(): spa_tryimport: importing boot-pool
1753119846   ffff980748af3280 spa_misc.c:429:spa_load_note(): spa_load($import-ffff980748af3280-boot-pool, config trusted): LOADING
1753119846   ffff9807f0916500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdb3': probe done, cant_read=0 cant_write=1
1753119846   ffff980748af3280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdb3': best uberblock found for spa $import-ffff980748af3280-boot-pool. txg 24771
1753119846   ffff980748af3280 spa_misc.c:429:spa_load_note(): spa_load($import-ffff980748af3280-boot-pool, config untrusted): using uberblock with txg=24771
1753119846   ffff980748af3280 vdev.c:2514:vdev_update_path(): vdev_copy_path: vdev 16950950020456470637: vdev_path changed from '/dev/sdh3' to '/dev/sdb3'
1753119846   ffff9807f0916500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdb3': probe done, cant_read=0 cant_write=1
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading checkpoint txg
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading indirect vdev metadata
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Checking feature flags
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading special MOS directories
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading properties
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading AUX vdevs
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading vdev metadata
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading dedup tables
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading BRT
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Verifying Log Devices
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Verifying pool data
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Calculating deflated space
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Starting import
1753119846   ffff980748af3280 spa.c:9209:spa_async_request(): spa=$import-ffff980748af3280-boot-pool async request task=2048
1753119846   ffff980748af3280 spa_misc.c:429:spa_load_note(): spa_load($import-ffff980748af3280-boot-pool, config trusted): LOADED
1753119846   ffff980748af3280 spa_misc.c:429:spa_load_note(): spa_load($import-ffff980748af3280-boot-pool, config trusted): UNLOADING
1753119846   ffff980748af3280 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753119846   ffff980748af3280 spa.c:6761:spa_import(): spa_import: importing boot-pool
1753119846   ffff980748af3280 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): LOADING
1753119846   ffff98086078cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdb3': probe done, cant_read=0 cant_write=1
1753119846   ffff980748af3280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdb3': best uberblock found for spa boot-pool. txg 24771
1753119846   ffff980748af3280 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config untrusted): using uberblock with txg=24771
1753119846   ffff980748af3280 vdev.c:2514:vdev_update_path(): vdev_copy_path: vdev 16950950020456470637: vdev_path changed from '/dev/sdh3' to '/dev/sdb3'
1753119846   ffff98086078cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/sdb3': probe done, cant_read=0 cant_write=0
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading checkpoint txg
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading indirect vdev metadata
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Checking feature flags
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading special MOS directories
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading properties
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading AUX vdevs
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading vdev metadata
1753119846   ffff980748af3280 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): Read 28 log space maps (28 total blocks - blksz = 131072 bytes) in 25 ms
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading dedup tables
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Loading BRT
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Verifying Log Devices
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Verifying pool data
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Calculating deflated space
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Starting import
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Claiming ZIL blocks
1753119846   ffff980748af3280 mmp.c:239:mmp_thread_start(): MMP thread started pool 'boot-pool' gethrtime 25600980738
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Syncing ZIL claims
1753119846   ffff980860b38000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24772, spa boot-pool, vdev_id 0, ms_id 98, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25601 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753119846   ffff980860b38000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24772, spa boot-pool, vdev_id 0, ms_id 0, smp_length 56, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25601 ms, loading_time 1 ms, ms_max_size 525938688, max size error 525938688, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861f8cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24772, spa boot-pool, vdev_id 0, ms_id 2, smp_length 2024, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25602 ms, loading_time 3 ms, ms_max_size 528277504, max size error 528277504, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861f8cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24772, spa boot-pool, vdev_id 0, ms_id 4, smp_length 1848, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25605 ms, loading_time 2 ms, ms_max_size 526270464, max size error 526270464, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861f8cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24772, spa boot-pool, vdev_id 0, ms_id 6, smp_length 2208, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25620 ms, loading_time 1 ms, ms_max_size 517255168, max size error 517255168, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861f88000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24772, spa boot-pool, vdev_id 0, ms_id 8, smp_length 2088, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25623 ms, loading_time 2 ms, ms_max_size 521072640, max size error 521072640, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861f88000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24772, spa boot-pool, vdev_id 0, ms_id 9, smp_length 1664, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25625 ms, loading_time 2 ms, ms_max_size 535412736, max size error 535412736, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861f8cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24772, spa boot-pool, vdev_id 0, ms_id 11, smp_length 5416, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25647 ms, loading_time 1 ms, ms_max_size 536809472, max size error 536809472, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861f8cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24772, spa boot-pool, vdev_id 0, ms_id 14, smp_length 1096, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25648 ms, loading_time 1 ms, ms_max_size 298160128, max size error 298160128, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Updating configs
1753119846   ffff980748af3280 spa.c:9209:spa_async_request(): spa=boot-pool async request task=1
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Starting resilvers
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Restarting device removals
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Cleaning up inconsistent objsets
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Cleaning up temporary userrefs
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Restarting initialize
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Restarting TRIM
1753119846   ffff980748af3280 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Finished importing
1753119846   ffff980748af3280 spa.c:9209:spa_async_request(): spa=boot-pool async request task=2048
1753119846   ffff980748af3280 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): LOADED
1753119846   ffff980861f8b280 spa_history.c:306:spa_history_log_sync(): txg 24773 open pool version 5000; software version zfs-2.1.99-2804-gd8c2ec4e9; uts (none) 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Fri Jul 11 12:56:00 UTC 2025 x86_64
1753119846   ffff9806d8360000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24774, spa boot-pool, vdev_id 0, ms_id 15, smp_length 40, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25880 ms, loading_time 1 ms, ms_max_size 536776704, max size error 536776704, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980748af3280 spa.c:9209:spa_async_request(): spa=boot-pool async request task=32
1753119846   ffff9806d8360000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24774, spa boot-pool, vdev_id 0, ms_id 17, smp_length 1056, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25882 ms, loading_time 1 ms, ms_max_size 375443456, max size error 375443456, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861dd6500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24775, spa boot-pool, vdev_id 0, ms_id 18, smp_length 808, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25882 ms, loading_time 1 ms, ms_max_size 455548928, max size error 455548928, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff98086270e500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24775, spa boot-pool, vdev_id 0, ms_id 23, smp_length 37520, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25883 ms, loading_time 4 ms, ms_max_size 458506240, max size error 458506240, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861dd9940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24775, spa boot-pool, vdev_id 0, ms_id 21, smp_length 2432, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25883 ms, loading_time 4 ms, ms_max_size 533430272, max size error 533430272, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861dde500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24775, spa boot-pool, vdev_id 0, ms_id 19, smp_length 40, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25883 ms, loading_time 5 ms, ms_max_size 536690688, max size error 536690688, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff98086270b280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24775, spa boot-pool, vdev_id 0, ms_id 24, smp_length 43960, unflushed_allocs 0, unflushed_frees 4096, freed 0, defer 0 + 0, unloaded time 25883 ms, loading_time 5 ms, ms_max_size 469372928, max size error 469368832, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861ddb280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24775, spa boot-pool, vdev_id 0, ms_id 20, smp_length 800, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25883 ms, loading_time 6 ms, ms_max_size 389197824, max size error 389197824, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861dd3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24775, spa boot-pool, vdev_id 0, ms_id 87, smp_length 50032, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25882 ms, loading_time 10 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753119846   ffff980861dd8000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24775, spa boot-pool, vdev_id 0, ms_id 22, smp_length 44624, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 25883 ms, loading_time 11 ms, ms_max_size 407572480, max size error 407572480, old_weight 700000000000001, new_weight 700000000000001
1753119846   ffff980861f8b280 spa_history.c:306:spa_history_log_sync(): txg 24775 import pool version 5000; software version zfs-2.1.99-2804-gd8c2ec4e9; uts (none) 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Fri Jul 11 12:56:00 UTC 2025 x86_64
1753119849   ffff980861f8b280 spa_history.c:293:spa_history_log_sync(): command: zpool import -N -f boot-pool
1753119882   ffff980860bfcbc0 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753119882   ffff980860bfcbc0 spa.c:6916:spa_tryimport(): spa_tryimport: importing local date
1753119882   ffff980860bfcbc0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff980860bfcbc0-local date, config trusted): LOADING
1753119882   ffff98086271b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': probe done, cant_read=0 cant_write=1
1753119882   ffff980860bfcbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': best uberblock found for spa $import-ffff980860bfcbc0-local date. txg 7034558
1753119882   ffff980860bfcbc0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff980860bfcbc0-local date, config untrusted): using uberblock with txg=7034558
1753119882   ffff98086271b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': probe done, cant_read=0 cant_write=1
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading checkpoint txg
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading indirect vdev metadata
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Checking feature flags
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading special MOS directories
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading properties
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading AUX vdevs
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading vdev metadata
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading dedup tables
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading BRT
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Verifying Log Devices
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Verifying pool data
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Calculating deflated space
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Starting import
1753119882   ffff980860bfcbc0 spa.c:9209:spa_async_request(): spa=$import-ffff980860bfcbc0-local date async request task=2048
1753119882   ffff980860bfcbc0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff980860bfcbc0-local date, config trusted): LOADED
1753119882   ffff980860bfcbc0 spa_misc.c:429:spa_load_note(): spa_load($import-ffff980860bfcbc0-local date, config trusted): UNLOADING
1753119882   ffff980860bfcbc0 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753119882   ffff980860bfcbc0 spa.c:6761:spa_import(): spa_import: importing local date
1753119882   ffff980860bfcbc0 spa_misc.c:429:spa_load_note(): spa_load(local date, config trusted): LOADING
1753119882   ffff9807f4d83280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': probe done, cant_read=0 cant_write=1
1753119882   ffff980860bfcbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': best uberblock found for spa local date. txg 7034558
1753119882   ffff980860bfcbc0 spa_misc.c:429:spa_load_note(): spa_load(local date, config untrusted): using uberblock with txg=7034558
1753119882   ffff9807f4d83280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': probe done, cant_read=0 cant_write=0
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading checkpoint txg
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading indirect vdev metadata
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Checking feature flags
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading special MOS directories
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading properties
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading AUX vdevs
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading vdev metadata
1753119882   ffff980860bfcbc0 spa_misc.c:429:spa_load_note(): spa_load(local date, config trusted): Read 26 log space maps (26 total blocks - blksz = 131072 bytes) in 3 ms
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading dedup tables
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading BRT
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Verifying Log Devices
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Verifying pool data
1753119882   ffff980860bfcbc0 spa_misc.c:429:spa_load_note(): spa_load(local date, config trusted): spa_load_verify found 0 metadata errors and 1 data errors
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Calculating deflated space
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Starting import
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Claiming ZIL blocks
1753119882   ffff980860bfcbc0 mmp.c:239:mmp_thread_start(): MMP thread started pool 'local date' gethrtime 61862623979
1753119882   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Syncing ZIL claims
1753119882   ffff9807f4d8e500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034559, spa local date, vdev_id 0, ms_id 29, smp_length 152624, unflushed_allocs 245760, unflushed_frees 540672, freed 0, defer 0 + 0, unloaded time 61862 ms, loading_time 8 ms, ms_max_size 1084186624, max size error 1084116992, old_weight 780000000000001, new_weight 780000000000001
1753119882   ffff9807f4d8e500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034559, spa local date, vdev_id 0, ms_id 99, smp_length 315016, unflushed_allocs 708608, unflushed_frees 397312, freed 0, defer 0 + 0, unloaded time 61871 ms, loading_time 15 ms, ms_max_size 1103282176, max size error 1103233024, old_weight 780000000000001, new_weight 780000000000001
1753119882   ffff9807f4d8e500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034559, spa local date, vdev_id 0, ms_id 106, smp_length 520896, unflushed_allocs 258048, unflushed_frees 2019328, freed 0, defer 0 + 0, unloaded time 61886 ms, loading_time 22 ms, ms_max_size 1083199488, max size error 1083138048, old_weight 780000000000001, new_weight 780000000000001
1753119882   ffff9807f4d8e500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034559, spa local date, vdev_id 0, ms_id 103, smp_length 342528, unflushed_allocs 352256, unflushed_frees 2531328, freed 0, defer 0 + 0, unloaded time 61909 ms, loading_time 13 ms, ms_max_size 774074368, max size error 772927488, old_weight 740000000000004, new_weight 740000000000004
1753119882   ffff980782d5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034559, spa local date, vdev_id 0, ms_id 104, smp_length 314352, unflushed_allocs 212992, unflushed_frees 1130496, freed 0, defer 0 + 0, unloaded time 61923 ms, loading_time 22 ms, ms_max_size 717287424, max size error 717197312, old_weight 740000000000004, new_weight 740000000000004
1753119882   ffff980782d5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034559, spa local date, vdev_id 0, ms_id 28, smp_length 223144, unflushed_allocs 417792, unflushed_frees 385024, freed 0, defer 0 + 0, unloaded time 61945 ms, loading_time 24 ms, ms_max_size 869285888, max size error 869212160, old_weight 740000000000003, new_weight 740000000000003
1753119882   ffff980782d5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034559, spa local date, vdev_id 0, ms_id 107, smp_length 282608, unflushed_allocs 520192, unflushed_frees 417792, freed 0, defer 0 + 0, unloaded time 61970 ms, loading_time 18 ms, ms_max_size 1042108416, max size error 1042046976, old_weight 740000000000003, new_weight 740000000000003
1753119883   ffff980782d5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034559, spa local date, vdev_id 0, ms_id 101, smp_length 349648, unflushed_allocs 339968, unflushed_frees 581632, freed 12288, defer 0 + 0, unloaded time 61991 ms, loading_time 23 ms, ms_max_size 897818624, max size error 897736704, old_weight 740000000000002, new_weight 740000000000002
1753119883   ffff980782d5cbc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034559, spa local date, vdev_id 0, ms_id 105, smp_length 133032, unflushed_allocs 4198400, unflushed_frees 376832, freed 16384, defer 0 + 0, unloaded time 62014 ms, loading_time 8 ms, ms_max_size 780910592, max size error 780836864, old_weight 740000000000002, new_weight 740000000000002
1753119883   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Updating configs
1753119883   ffff980860bfcbc0 spa.c:9209:spa_async_request(): spa=local date async request task=1
1753119883   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Starting resilvers
1753119883   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Restarting device removals
1753119883   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Cleaning up inconsistent objsets
1753119883   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Cleaning up temporary userrefs
1753119883   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Restarting initialize
1753119883   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Restarting TRIM
1753119883   ffff980860bfcbc0 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Finished importing
1753119883   ffff980860bfcbc0 spa.c:9209:spa_async_request(): spa=local date async request task=2048
1753119883   ffff980860bfcbc0 spa_misc.c:429:spa_load_note(): spa_load(local date, config trusted): LOADED
1753119883   ffff980712c83280 spa_history.c:306:spa_history_log_sync(): txg 7034560 open pool version 5000; software version zfs-2.1.99-2804-gd8c2ec4e9; uts truenas 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Fri Jul 11 12:56:00 UTC 2025 x86_64
1753119883   ffff9807f4deb280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034560, spa local date, vdev_id 0, ms_id 94, smp_length 85168, unflushed_allocs 413696, unflushed_frees 348160, freed 0, defer 69632 + 114688, unloaded time 62063 ms, loading_time 10 ms, ms_max_size 539324416, max size error 539262976, old_weight 740000000000001, new_weight 740000000000001
1753119883   ffff980712004bc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034561, spa local date, vdev_id 0, ms_id 95, smp_length 26272, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 62079 ms, loading_time 2 ms, ms_max_size 536100864, max size error 536100864, old_weight 700000000000001, new_weight 700000000000001
1753119883   ffff980713623280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034561, spa local date, vdev_id 0, ms_id 100, smp_length 93088, unflushed_allocs 0, unflushed_frees 8192, freed 0, defer 0 + 0, unloaded time 62079 ms, loading_time 6 ms, ms_max_size 546504704, max size error 546496512, old_weight 740000000000001, new_weight 740000000000001
1753119883   ffff980712001940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034561, spa local date, vdev_id 0, ms_id 91, smp_length 163952, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 62079 ms, loading_time 13 ms, ms_max_size 419549184, max size error 419549184, old_weight 700000000000002, new_weight 700000000000002
1753119883   ffff980860bfcbc0 spa.c:9209:spa_async_request(): spa=local date async request task=32
1753119883   ffff980712c83280 spa_history.c:306:spa_history_log_sync(): txg 7034562 import pool version 5000; software version zfs-2.1.99-2804-gd8c2ec4e9; uts truenas 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Fri Jul 11 12:56:00 UTC 2025 x86_64
1753119883   ffff980861d1e500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034562, spa local date, vdev_id 0, ms_id 98, smp_length 431384, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 62079 ms, loading_time 16 ms, ms_max_size 552349696, max size error 552349696, old_weight 740000000000001, new_weight 740000000000001
1753119883   ffff980712000000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034562, spa local date, vdev_id 0, ms_id 93, smp_length 161760, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 62079 ms, loading_time 16 ms, ms_max_size 387538944, max size error 387538944, old_weight 700000000000001, new_weight 700000000000001
1753119883   ffff980712003280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034562, spa local date, vdev_id 0, ms_id 92, smp_length 369912, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 62079 ms, loading_time 19 ms, ms_max_size 403533824, max size error 403533824, old_weight 700000000000003, new_weight 700000000000003
1753119883   ffff980713621940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034562, spa local date, vdev_id 0, ms_id 49, smp_length 521560, unflushed_allocs 86016, unflushed_frees 131072, freed 0, defer 0 + 0, unloaded time 62079 ms, loading_time 20 ms, ms_max_size 8541392896, max size error 8541261824, old_weight 800000000000001, new_weight 800000000000001
1753119883   ffff9807f4deb280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034562, spa local date, vdev_id 0, ms_id 97, smp_length 441048, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 62074 ms, loading_time 25 ms, ms_max_size 1069182976, max size error 1069182976, old_weight 740000000000001, new_weight 740000000000001
1753119883   ffff980712006500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034562, spa local date, vdev_id 0, ms_id 96, smp_length 464880, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 62079 ms, loading_time 25 ms, ms_max_size 530911232, max size error 530911232, old_weight 700000000000001, new_weight 700000000000001
1753119888   ffff980712c83280 spa_history.c:293:spa_history_log_sync(): command: zpool import 13277177274598996053 -R /mnt -m -N -f -o cachefile=/data/zfs/zpool.cache
1753119891   ffff98086270b280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 24785, spa boot-pool, vdev_id 0, ms_id 88, smp_length 128, unflushed_allocs 0, unflushed_frees 81920, freed 0, defer 16384 + 8192, unloaded time 70557 ms, loading_time 2 ms, ms_max_size 532234240, max size error 532201472, old_weight 740000000000001, new_weight 740000000000001
1753120155   ffff9806d156e500 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753120155   ffff9806d156e500 spa.c:6916:spa_tryimport(): spa_tryimport: importing sok1001
1753120155   ffff9806d156e500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9806d156e500-sok1001, config trusted): LOADING
1753120155   ffff9806dfd4e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120156   ffff9809ec6f6500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': open error=2 timeout=1003374501/1000000000
1753120156   ffff9808d4170000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': open error=2 timeout=1011338746/1000000000
1753120156   ffff9808d4174bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': open error=2 timeout=1011361985/1000000000
1753120156   ffff9809ec6f1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': open error=2 timeout=1011334486/1000000000
1753120156   ffff9809ec6f3280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': open error=2 timeout=1011345880/1000000000
1753120156   ffff9806d156e500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9806d156e500-sok1001, config untrusted): vdev tree has 1 missing top-level vdevs.
1753120156   ffff9806d156e500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9806d156e500-sok1001, config untrusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
1753120156   ffff9806d156e500 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9806d156e500-sok1001, config untrusted): FAILED: unable to open vdev tree [error=2]
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 1221631789954295656, path: N/A, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 1132576499012000637, path: N/A, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 705601381597961486, path: /dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 15682408288857988643, path: /dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 16097378473779213230, path: /dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 10339353618853963483, path: /dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 4278422679801059362, path: /dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 15237958642688602726, path: /dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464, healthy
1753120156   ffff9806d156e500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9806d156e500-sok1001, config untrusted): UNLOADING
1753120190   ffff9807f5cf9940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753120190   ffff9807f5cf9940 spa.c:6916:spa_tryimport(): spa_tryimport: importing sok1001
1753120190   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config trusted): LOADING
1753120190   ffff9808d6b31940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753120190   ffff9808d6b31940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120191   ffff9808d402b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': open error=2 timeout=1003181995/1000000000
1753120191   ffff9808d66a1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': open error=2 timeout=1007158049/1000000000
1753120191   ffff9807f5f0cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': open error=2 timeout=1011195551/1000000000
1753120191   ffff9808d66a4bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': open error=2 timeout=1015161894/1000000000
1753120191   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config untrusted): vdev tree has 1 missing top-level vdevs.
1753120191   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config untrusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
1753120191   ffff9807f5cf9940 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9807f5cf9940-sok1001, config untrusted): FAILED: unable to open vdev tree [error=2]
1753120191   ffff9807f5cf9940 vdev.c:219:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 1221631789954295656, path: N/A, can't open
1753120191   ffff9807f5cf9940 vdev.c:219:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 1132576499012000637, path: N/A, can't open
1753120191   ffff9807f5cf9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 705601381597961486, path: /dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6, can't open
1753120191   ffff9807f5cf9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 15682408288857988643, path: /dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab, can't open
1753120191   ffff9807f5cf9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 16097378473779213230, path: /dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70, can't open
1753120191   ffff9807f5cf9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 10339353618853963483, path: /dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb, can't open
1753120191   ffff9807f5cf9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 4278422679801059362, path: /dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d, healthy
1753120191   ffff9807f5cf9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 15237958642688602726, path: /dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464, healthy
1753120191   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config untrusted): UNLOADING
1753120221   ffff9808624f9940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753120221   ffff9808624f9940 spa.c:6916:spa_tryimport(): spa_tryimport: importing sok1001
1753120221   ffff9808624f9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9808624f9940-sok1001, config trusted): LOADING
1753120221   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753120221   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753120221   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120222   ffff980749d0cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': open error=2 timeout=1003651170/1000000000
1753120222   ffff980749d0e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': open error=2 timeout=1007642670/1000000000
1753120222   ffff980749d08000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': open error=2 timeout=1011685349/1000000000
1753120222   ffff9808624f9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9808624f9940-sok1001, config untrusted): vdev tree has 1 missing top-level vdevs.
1753120222   ffff9808624f9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9808624f9940-sok1001, config untrusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
1753120222   ffff9808624f9940 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9808624f9940-sok1001, config untrusted): FAILED: unable to open vdev tree [error=2]
1753120222   ffff9808624f9940 vdev.c:219:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 1221631789954295656, path: N/A, can't open
1753120222   ffff9808624f9940 vdev.c:219:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 1132576499012000637, path: N/A, can't open
1753120222   ffff9808624f9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 705601381597961486, path: /dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6, can't open
1753120222   ffff9808624f9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 15682408288857988643, path: /dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab, can't open
1753120222   ffff9808624f9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 16097378473779213230, path: /dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70, can't open
1753120222   ffff9808624f9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 10339353618853963483, path: /dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb, healthy
1753120222   ffff9808624f9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 4278422679801059362, path: /dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d, healthy
1753120222   ffff9808624f9940 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 15237958642688602726, path: /dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464, healthy
1753120222   ffff9808624f9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9808624f9940-sok1001, config untrusted): UNLOADING
1753120260   ffff9808624f9940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753120260   ffff9808624f9940 spa.c:6916:spa_tryimport(): spa_tryimport: importing sok1001
1753120260   ffff9808624f9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9808624f9940-sok1001, config trusted): LOADING
1753120260   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753120260   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753120260   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120260   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753120261   ffff9807b9d51940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': open error=2 timeout=1010206270/1000000000
1753120261   ffff9808d684e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': open error=2 timeout=1014160021/1000000000
1753120261   ffff9808624f9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': best uberblock found for spa $import-ffff9808624f9940-sok1001. txg 6823467
1753120261   ffff9808624f9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9808624f9940-sok1001, config untrusted): using uberblock with txg=6823467
1753120261   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753120261   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753120261   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120261   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753120262   ffff980861ddcbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': open error=2 timeout=1005724707/1000000000
1753120262   ffff9807f7950000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': open error=2 timeout=1013733422/1000000000
1753120262   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading checkpoint txg
1753120262   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading indirect vdev metadata
1753120262   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Checking feature flags
1753120262   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading special MOS directories
1753120262   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading properties
1753120262   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading AUX vdevs
1753120262   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading vdev metadata
1753120263   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading dedup tables
1753120263   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading BRT
1753120263   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Verifying Log Devices
1753120263   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Verifying pool data
1753120263   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Calculating deflated space
1753120263   ffff9808624f9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Starting import
1753120263   ffff9808624f9940 spa.c:9209:spa_async_request(): spa=$import-ffff9808624f9940-sok1001 async request task=2048
1753120263   ffff9808624f9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9808624f9940-sok1001, config trusted): LOADED
1753120263   ffff9808624f9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9808624f9940-sok1001, config trusted): UNLOADING
1753120294   ffff9807f5cf9940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753120294   ffff9807f5cf9940 spa.c:6916:spa_tryimport(): spa_tryimport: importing sok1001
1753120294   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config trusted): LOADING
1753120294   ffff9806d156b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': probe done, cant_read=0 cant_write=1
1753120294   ffff9806d156b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753120294   ffff9806d156b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120294   ffff9806d156b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753120294   ffff9806d156b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753120295   ffff980748fdb280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': open error=2 timeout=1010023658/1000000000
1753120295   ffff9807f5cf9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': best uberblock found for spa $import-ffff9807f5cf9940-sok1001. txg 6823467
1753120295   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config untrusted): using uberblock with txg=6823467
1753120295   ffff9806d156b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120295   ffff9806d156b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753120295   ffff9806d156b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753120295   ffff9806d156b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753120295   ffff9806d156b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': probe done, cant_read=0 cant_write=1
1753120296   ffff9809ecc48000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': open error=2 timeout=1007083458/1000000000
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading checkpoint txg
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading indirect vdev metadata
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Checking feature flags
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading special MOS directories
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading properties
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading AUX vdevs
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading vdev metadata
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading dedup tables
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading BRT
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Verifying Log Devices
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Verifying pool data
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Calculating deflated space
1753120296   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Starting import
1753120296   ffff9807f5cf9940 spa.c:9209:spa_async_request(): spa=$import-ffff9807f5cf9940-sok1001 async request task=2048
1753120296   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config trusted): LOADED
1753120296   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config trusted): UNLOADING
1753120339   ffff9807f5cf9940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753120339   ffff9807f5cf9940 spa.c:6916:spa_tryimport(): spa_tryimport: importing sok1001
1753120339   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config trusted): LOADING
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': probe done, cant_read=0 cant_write=1
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': probe done, cant_read=0 cant_write=1
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753120339   ffff9807f5cf9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': best uberblock found for spa $import-ffff9807f5cf9940-sok1001. txg 6823467
1753120339   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config untrusted): using uberblock with txg=6823467
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': probe done, cant_read=0 cant_write=1
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': probe done, cant_read=0 cant_write=1
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753120339   ffff980712443280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading checkpoint txg
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading indirect vdev metadata
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Checking feature flags
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading special MOS directories
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading properties
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading AUX vdevs
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading vdev metadata
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading dedup tables
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading BRT
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Verifying Log Devices
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Verifying pool data
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Calculating deflated space
1753120339   ffff9807f5cf9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Starting import
1753120339   ffff9807f5cf9940 spa.c:9209:spa_async_request(): spa=$import-ffff9807f5cf9940-sok1001 async request task=2048
1753120339   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config trusted): LOADED
1753120339   ffff9807f5cf9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9807f5cf9940-sok1001, config trusted): UNLOADING
1753120490   ffff980861f8b280 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 24896, spa boot-pool, vdev_id 0, ms_id 24, weight 700000000000001, selected txg 24784 (605026 ms ago), alloc_txg 0, loaded 643877 ms ago, max_size 469372928

Does dbgmsg have records before the restart? I don’t know if these are
This log looks like it should be a log after a reboot.

128g was not imported successfully either. I thought it could run longer, but it ended quickly in the second half and I was scared by the restart beep.

leng_sok · July 21, 2025, 10:14pm

I didn’t see the second half, this is the first half
It has been here for a long time, and no new content has been added for several hours
The last time I checked was around the sixth hour, when the memory usage was 40g, but it restarted not long after.

root@truenas[~]# tail -n 200 /proc/spl/kstat/zfs/dbgmsg
1753084897   ffff969789c09940 spa.c:9209:spa_async_request(): spa=boot-pool async request task=1
1753084897   ffff969789c09940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Starting resilvers
1753084897   ffff969789c09940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Restarting device removals
1753084897   ffff969789c09940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Cleaning up inconsistent objsets
1753084897   ffff969789c09940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Cleaning up temporary userrefs
1753084897   ffff969789c09940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Restarting initialize
1753084897   ffff969789c09940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Restarting TRIM
1753084897   ffff969789c09940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'boot-pool' Finished importing
1753084897   ffff969789c09940 spa.c:9209:spa_async_request(): spa=boot-pool async request task=2048
1753084897   ffff969789c09940 spa_misc.c:429:spa_load_note(): spa_load(boot-pool, config trusted): LOADED
1753084897   ffff96948bc96500 spa_history.c:306:spa_history_log_sync(): txg 18424 open pool version 5000; software version zfs-2.1.99-2804-gd8c2ec4e9; uts (none) 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Fri Jul 11 12:56:00 UTC 2025 x86_64
1753084898   ffff969491aa3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 18424, spa boot-pool, vdev_id 0, ms_id 87, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 28183 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753084898   ffff969491aa3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 18424, spa boot-pool, vdev_id 0, ms_id 88, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 28183 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753084898   ffff969491aa3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 18424, spa boot-pool, vdev_id 0, ms_id 89, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 28183 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753084898   ffff969491aa3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 18424, spa boot-pool, vdev_id 0, ms_id 76, smp_length 0, unflushed_allocs 1822720, unflushed_frees 0, freed 0, defer 0 + 36864, unloaded time 28183 ms, loading_time 0 ms, ms_max_size 523849728, max size error 523849728, old_weight 700000000000001, new_weight 700000000000001
1753084898   ffff969491aa3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 18424, spa boot-pool, vdev_id 0, ms_id 90, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 28183 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753084898   ffff969491aa3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 18424, spa boot-pool, vdev_id 0, ms_id 91, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 28183 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753084898   ffff969491aa3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 18424, spa boot-pool, vdev_id 0, ms_id 92, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 28183 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753084898   ffff969491aa3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 18424, spa boot-pool, vdev_id 0, ms_id 93, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 28183 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753084898   ffff969491aa3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 18424, spa boot-pool, vdev_id 0, ms_id 94, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 28183 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753084898   ffff969491aa3280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 18424, spa boot-pool, vdev_id 0, ms_id 95, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 28183 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753084898   ffff969789c09940 spa.c:9209:spa_async_request(): spa=boot-pool async request task=32
1753084898   ffff96948bc96500 spa_history.c:306:spa_history_log_sync(): txg 18426 import pool version 5000; software version zfs-2.1.99-2804-gd8c2ec4e9; uts (none) 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Fri Jul 11 12:56:00 UTC 2025 x86_64
1753084901   ffff96948bc96500 spa_history.c:293:spa_history_log_sync(): command: zpool import -N -f boot-pool
1753084937   ffff96948bc96500 spa_history.c:297:spa_history_log_sync(): txg 18436 set boot-pool/ROOT/24.10.2.3/audit (id 173) org.freenas:refquota_warning=80
1753084941   ffff96945b8b9940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753084941   ffff96945b8b9940 spa.c:6916:spa_tryimport(): spa_tryimport: importing local date
1753084941   ffff96945b8b9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96945b8b9940-local date, config trusted): LOADING
1753084941   ffff969471d1b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': probe done, cant_read=0 cant_write=1
1753084941   ffff96945b8b9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': best uberblock found for spa $import-ffff96945b8b9940-local date. txg 7034291
1753084941   ffff96945b8b9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96945b8b9940-local date, config untrusted): using uberblock with txg=7034291
1753084941   ffff969471d1b280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': probe done, cant_read=0 cant_write=1
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading checkpoint txg
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading indirect vdev metadata
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Checking feature flags
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading special MOS directories
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading properties
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading AUX vdevs
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading vdev metadata
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading dedup tables
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading BRT
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Verifying Log Devices
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Verifying pool data
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Calculating deflated space
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Starting import
1753084941   ffff96945b8b9940 spa.c:9209:spa_async_request(): spa=$import-ffff96945b8b9940-local date async request task=2048
1753084941   ffff96945b8b9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96945b8b9940-local date, config trusted): LOADED
1753084941   ffff96945b8b9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96945b8b9940-local date, config trusted): UNLOADING
1753084941   ffff96945b8b9940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753084941   ffff96945b8b9940 spa.c:6761:spa_import(): spa_import: importing local date
1753084941   ffff96945b8b9940 spa_misc.c:429:spa_load_note(): spa_load(local date, config trusted): LOADING
1753084941   ffff96947e2d6500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': probe done, cant_read=0 cant_write=1
1753084941   ffff96945b8b9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': best uberblock found for spa local date. txg 7034291
1753084941   ffff96945b8b9940 spa_misc.c:429:spa_load_note(): spa_load(local date, config untrusted): using uberblock with txg=7034291
1753084941   ffff96947e2d6500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e3a88fea-cb0d-40f1-94de-4b4c5dc36166': probe done, cant_read=0 cant_write=0
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading checkpoint txg
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading indirect vdev metadata
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Checking feature flags
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading special MOS directories
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading properties
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading AUX vdevs
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading vdev metadata
1753084941   ffff96945b8b9940 spa_misc.c:429:spa_load_note(): spa_load(local date, config trusted): Read 12 log space maps (12 total blocks - blksz = 131072 bytes) in 1 ms
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading dedup tables
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Loading BRT
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Verifying Log Devices
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Verifying pool data
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Calculating deflated space
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Starting import
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Claiming ZIL blocks
1753084941   ffff96945b8b9940 mmp.c:239:mmp_thread_start(): MMP thread started pool 'local date' gethrtime 71744807511
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Syncing ZIL claims
1753084941   ffff96947e2d8000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034292, spa local date, vdev_id 0, ms_id 29, smp_length 150008, unflushed_allocs 36864, unflushed_frees 4096, freed 0, defer 0 + 0, unloaded time 71744 ms, loading_time 8 ms, ms_max_size 1084186624, max size error 1084182528, old_weight 780000000000001, new_weight 780000000000001
1753084941   ffff96947e2d8000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034292, spa local date, vdev_id 0, ms_id 99, smp_length 313336, unflushed_allocs 151552, unflushed_frees 135168, freed 0, defer 0 + 0, unloaded time 71753 ms, loading_time 15 ms, ms_max_size 1103282176, max size error 1103216640, old_weight 780000000000001, new_weight 780000000000001
1753084941   ffff96947e2d8000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034292, spa local date, vdev_id 0, ms_id 106, smp_length 517576, unflushed_allocs 188416, unflushed_frees 90112, freed 0, defer 0 + 0, unloaded time 71768 ms, loading_time 24 ms, ms_max_size 1083199488, max size error 1083162624, old_weight 780000000000001, new_weight 780000000000001
1753084941   ffff96947e2d8000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034292, spa local date, vdev_id 0, ms_id 103, smp_length 341528, unflushed_allocs 999424, unflushed_frees 106496, freed 0, defer 0 + 0, unloaded time 71793 ms, loading_time 16 ms, ms_max_size 774074368, max size error 774037504, old_weight 740000000000004, new_weight 740000000000004
1753084941   ffff96947e2d8000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034292, spa local date, vdev_id 0, ms_id 104, smp_length 313120, unflushed_allocs 106496, unflushed_frees 135168, freed 0, defer 0 + 0, unloaded time 71809 ms, loading_time 15 ms, ms_max_size 717287424, max size error 717213696, old_weight 740000000000004, new_weight 740000000000004
1753084941   ffff96947e2d8000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034292, spa local date, vdev_id 0, ms_id 28, smp_length 221976, unflushed_allocs 225280, unflushed_frees 253952, freed 0, defer 0 + 0, unloaded time 71825 ms, loading_time 12 ms, ms_max_size 869285888, max size error 869212160, old_weight 740000000000003, new_weight 740000000000003
1753084941   ffff96947e2d8000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034292, spa local date, vdev_id 0, ms_id 107, smp_length 281792, unflushed_allocs 172032, unflushed_frees 110592, freed 0, defer 0 + 0, unloaded time 71839 ms, loading_time 12 ms, ms_max_size 1042108416, max size error 1042051072, old_weight 740000000000003, new_weight 740000000000003
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Updating configs
1753084941   ffff96945b8b9940 spa.c:9209:spa_async_request(): spa=local date async request task=1
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Starting resilvers
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Restarting device removals
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Cleaning up inconsistent objsets
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Cleaning up temporary userrefs
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Restarting initialize
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Restarting TRIM
1753084941   ffff96945b8b9940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'local date' Finished importing
1753084941   ffff96945b8b9940 spa.c:9209:spa_async_request(): spa=local date async request task=2048
1753084941   ffff96945b8b9940 spa_misc.c:429:spa_load_note(): spa_load(local date, config trusted): LOADED
1753084941   ffff9694637a4bc0 spa_history.c:306:spa_history_log_sync(): txg 7034293 open pool version 5000; software version zfs-2.1.99-2804-gd8c2ec4e9; uts truenas 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Fri Jul 11 12:56:00 UTC 2025 x86_64
1753084941   ffff969480873280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034293, spa local date, vdev_id 0, ms_id 101, smp_length 348744, unflushed_allocs 69632, unflushed_frees 303104, freed 0, defer 8192 + 0, unloaded time 71879 ms, loading_time 17 ms, ms_max_size 897818624, max size error 897736704, old_weight 740000000000002, new_weight 740000000000002
1753084941   ffff969480873280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034293, spa local date, vdev_id 0, ms_id 105, smp_length 131872, unflushed_allocs 430080, unflushed_frees 180224, freed 4096, defer 73728 + 0, unloaded time 71896 ms, loading_time 6 ms, ms_max_size 780910592, max size error 780836864, old_weight 740000000000002, new_weight 740000000000002
1753084941   ffff96947e5e9940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034293, spa local date, vdev_id 0, ms_id 94, smp_length 84264, unflushed_allocs 65536, unflushed_frees 294912, freed 0, defer 163840 + 69632, unloaded time 71917 ms, loading_time 11 ms, ms_max_size 539324416, max size error 539267072, old_weight 740000000000001, new_weight 740000000000001
1753084941   ffff96945b714bc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034294, spa local date, vdev_id 0, ms_id 95, smp_length 26272, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71935 ms, loading_time 2 ms, ms_max_size 536100864, max size error 536100864, old_weight 700000000000001, new_weight 700000000000001
1753084941   ffff96945b713280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034294, spa local date, vdev_id 0, ms_id 100, smp_length 93088, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71935 ms, loading_time 6 ms, ms_max_size 546504704, max size error 546504704, old_weight 740000000000001, new_weight 740000000000001
1753084941   ffff96945b716500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034294, spa local date, vdev_id 0, ms_id 91, smp_length 163952, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71935 ms, loading_time 9 ms, ms_max_size 419549184, max size error 419549184, old_weight 700000000000002, new_weight 700000000000002
1753084941   ffff96945b710000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034294, spa local date, vdev_id 0, ms_id 93, smp_length 161760, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71935 ms, loading_time 10 ms, ms_max_size 387538944, max size error 387538944, old_weight 700000000000001, new_weight 700000000000001
1753084941   ffff96945b720000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034294, spa local date, vdev_id 0, ms_id 49, smp_length 521528, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71935 ms, loading_time 13 ms, ms_max_size 8541392896, max size error 8541392896, old_weight 800000000000001, new_weight 800000000000001
1753084941   ffff96947e5e9940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034294, spa local date, vdev_id 0, ms_id 97, smp_length 441048, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71929 ms, loading_time 23 ms, ms_max_size 1069182976, max size error 1069182976, old_weight 740000000000001, new_weight 740000000000001
1753084941   ffff96945b711940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034294, spa local date, vdev_id 0, ms_id 92, smp_length 369912, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71935 ms, loading_time 22 ms, ms_max_size 403533824, max size error 403533824, old_weight 700000000000003, new_weight 700000000000003
1753084941   ffff96945b726500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034294, spa local date, vdev_id 0, ms_id 98, smp_length 431384, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71935 ms, loading_time 23 ms, ms_max_size 552349696, max size error 552349696, old_weight 740000000000001, new_weight 740000000000001
1753084941   ffff96945b8b9940 spa.c:9209:spa_async_request(): spa=local date async request task=32
1753084941   ffff9694637a4bc0 spa_history.c:306:spa_history_log_sync(): txg 7034295 import pool version 5000; software version zfs-2.1.99-2804-gd8c2ec4e9; uts truenas 6.6.44-production+truenas #1 SMP PREEMPT_DYNAMIC Fri Jul 11 12:56:00 UTC 2025 x86_64
1753084941   ffff96945b70e500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034295, spa local date, vdev_id 0, ms_id 96, smp_length 464880, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71935 ms, loading_time 24 ms, ms_max_size 530911232, max size error 530911232, old_weight 700000000000001, new_weight 700000000000001
1753084947   ffff9694637a4bc0 spa_history.c:293:spa_history_log_sync(): command: zpool import 13277177274598996053 -R /mnt -m -N -f -o cachefile=/data/zfs/zpool.cache
1753084947   ffff96947b691940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753084947   ffff96947b691940 spa.c:6916:spa_tryimport(): spa_tryimport: importing sok1001
1753084947   ffff96947b691940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96947b691940-sok1001, config trusted): LOADING
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': probe done, cant_read=0 cant_write=1
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': probe done, cant_read=0 cant_write=1
1753084947   ffff96947b691940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': best uberblock found for spa $import-ffff96947b691940-sok1001. txg 6823467
1753084947   ffff96947b691940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96947b691940-sok1001, config untrusted): using uberblock with txg=6823467
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': probe done, cant_read=0 cant_write=1
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': probe done, cant_read=0 cant_write=1
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753084947   ffff96945bc28000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753084947   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading checkpoint txg
1753084947   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading indirect vdev metadata
1753084947   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Checking feature flags
1753084947   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading special MOS directories
1753084947   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading properties
1753084947   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading AUX vdevs
1753084947   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading vdev metadata
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading dedup tables
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading BRT
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Verifying Log Devices
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Verifying pool data
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Calculating deflated space
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Starting import
1753084948   ffff96947b691940 spa.c:9209:spa_async_request(): spa=$import-ffff96947b691940-sok1001 async request task=2048
1753084948   ffff96947b691940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96947b691940-sok1001, config trusted): LOADED
1753084948   ffff96947b691940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff96947b691940-sok1001, config trusted): UNLOADING
1753084948   ffff96947b691940 metaslab.c:1683:spa_set_allocator(): spa allocator: dynamic
1753084948   ffff96947b691940 spa.c:6761:spa_import(): spa_import: importing sok1001
1753084948   ffff96947b691940 spa_misc.c:429:spa_load_note(): spa_load(sok1001, config trusted): LOADING
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': probe done, cant_read=0 cant_write=1
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': probe done, cant_read=0 cant_write=1
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753084948   ffff96947b691940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': best uberblock found for spa sok1001. txg 6823467
1753084948   ffff96947b691940 spa_misc.c:429:spa_load_note(): spa_load(sok1001, config untrusted): using uberblock with txg=6823467
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': probe done, cant_read=0 cant_write=0
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=0
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=0
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=0
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': probe done, cant_read=0 cant_write=0
1753084948   ffff96946a07cbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=0
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading checkpoint txg
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading indirect vdev metadata
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Checking feature flags
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading special MOS directories
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading properties
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading AUX vdevs
1753084948   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading vdev metadata
1753084957   ffff96947b691940 spa_misc.c:429:spa_load_note(): spa_load(sok1001, config trusted): Read 698 log space maps (2350 total blocks - blksz = 131072 bytes) in 8147 ms
1753084957   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading dedup tables
1753084957   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Loading BRT
1753084957   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Verifying Log Devices
1753084957   ffff96945bfa4bc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 0, spa sok1001, vdev_id 0, ms_id 4650, smp_length 2451488, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 87550 ms, loading_time 275 ms, ms_max_size 16236728320, max size error 16236728320, old_weight 740000000000001, new_weight 840000000000001
1753084966   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Verifying pool data
1753084992   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Calculating deflated space
1753084992   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Starting import
1753084992   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Claiming ZIL blocks
1753085004   ffff96947b691940 mmp.c:239:mmp_thread_start(): MMP thread started pool 'sok1001' gethrtime 134480110784
1753085004   ffff96947b691940 spa_misc.c:2381:spa_import_progress_set_notes_impl(): 'sok1001' Syncing ZIL claims
1753085561   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034420, spa local date, vdev_id 0, ms_id 96, weight 700000000000001, selected txg 7034302 (604159 ms ago), alloc_txg 0, loaded 619716 ms ago, max_size 530911232
1753085561   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034420, spa local date, vdev_id 0, ms_id 97, weight 740000000000001, selected txg 7034302 (604159 ms ago), alloc_txg 0, loaded 619723 ms ago, max_size 1069182976
1753085561   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034420, spa local date, vdev_id 0, ms_id 98, weight 740000000000001, selected txg 7034302 (604159 ms ago), alloc_txg 0, loaded 619718 ms ago, max_size 552349696
1753085561   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034420, spa local date, vdev_id 0, ms_id 99, weight 780000000000001, selected txg 7034302 (604159 ms ago), alloc_txg 7034295, loaded 619907 ms ago, max_size 1103282176
1753085561   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034420, spa local date, vdev_id 0, ms_id 100, weight 740000000000001, selected txg 7034302 (604159 ms ago), alloc_txg 0, loaded 619734 ms ago, max_size 546504704
1753085561   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034420, spa local date, vdev_id 0, ms_id 91, weight 700000000000002, selected txg 7034302 (604159 ms ago), alloc_txg 0, loaded 619731 ms ago, max_size 419549184
1753085561   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034420, spa local date, vdev_id 0, ms_id 94, weight 740000000000001, selected txg 7034302 (604159 ms ago), alloc_txg 7034300, loaded 619747 ms ago, max_size 539324416
1753085561   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034420, spa local date, vdev_id 0, ms_id 95, weight 700000000000001, selected txg 7034302 (604159 ms ago), alloc_txg 0, loaded 619738 ms ago, max_size 536100864
1753085863   ffff969480870000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034479, spa local date, vdev_id 0, ms_id 99, smp_length 313512, unflushed_allocs 8192, unflushed_frees 126976, freed 0, defer 0 + 0, unloaded time 302083 ms, loading_time 16 ms, ms_max_size 1103282176, max size error 0, old_weight 780000000000001, new_weight 780000000000001
1753085863   ffff96945b726500 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034479, spa local date, vdev_id 0, ms_id 95, smp_length 26272, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 302108 ms, loading_time 1 ms, ms_max_size 536100864, max size error 0, old_weight 700000000000001, new_weight 700000000000001
1753085863   ffff96945b720000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034479, spa local date, vdev_id 0, ms_id 94, smp_length 84264, unflushed_allocs 274432, unflushed_frees 425984, freed 0, defer 0 + 106496, unloaded time 302108 ms, loading_time 4 ms, ms_max_size 539324416, max size error 0, old_weight 740000000000001, new_weight 740000000000001
1753085863   ffff96947e5e9940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034479, spa local date, vdev_id 0, ms_id 100, smp_length 93088, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 302108 ms, loading_time 10 ms, ms_max_size 546504704, max size error 0, old_weight 740000000000001, new_weight 740000000000001
1753085863   ffff9694582e8000 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034479, spa local date, vdev_id 0, ms_id 96, smp_length 464880, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 302108 ms, loading_time 26 ms, ms_max_size 530911232, max size error 0, old_weight 700000000000001, new_weight 700000000000001
1753085863   ffff96945b713280 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034479, spa local date, vdev_id 0, ms_id 91, smp_length 163952, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 302108 ms, loading_time 28 ms, ms_max_size 419549184, max size error 0, old_weight 700000000000002, new_weight 700000000000002
1753085863   ffff969462bf4bc0 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034479, spa local date, vdev_id 0, ms_id 98, smp_length 431384, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 302108 ms, loading_time 45 ms, ms_max_size 552349696, max size error 0, old_weight 740000000000001, new_weight 740000000000001
1753085863   ffff96945b711940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 7034479, spa local date, vdev_id 0, ms_id 97, smp_length 441048, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 302108 ms, loading_time 52 ms, ms_max_size 1069182976, max size error 0, old_weight 740000000000001, new_weight 740000000000001
1753086835   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034672, spa local date, vdev_id 0, ms_id 96, weight 700000000000001, selected txg 7034551 (603445 ms ago), alloc_txg 0, loaded 971896 ms ago, max_size 530911232
1753086835   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034672, spa local date, vdev_id 0, ms_id 97, weight 740000000000001, selected txg 7034551 (603445 ms ago), alloc_txg 0, loaded 971871 ms ago, max_size 1069182976
1753086835   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034672, spa local date, vdev_id 0, ms_id 98, weight 740000000000001, selected txg 7034551 (603445 ms ago), alloc_txg 0, loaded 971877 ms ago, max_size 552349696
1753086835   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034672, spa local date, vdev_id 0, ms_id 100, weight 740000000000001, selected txg 7034551 (603445 ms ago), alloc_txg 0, loaded 971912 ms ago, max_size 546504704
1753086835   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034672, spa local date, vdev_id 0, ms_id 107, weight 740000000000003, selected txg 7034551 (603445 ms ago), alloc_txg 7034513, loaded 1893856 ms ago, max_size 1042108416
1753086835   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034672, spa local date, vdev_id 0, ms_id 91, weight 700000000000002, selected txg 7034551 (603445 ms ago), alloc_txg 0, loaded 971895 ms ago, max_size 419549184
1753086835   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034672, spa local date, vdev_id 0, ms_id 94, weight 740000000000001, selected txg 7034551 (603445 ms ago), alloc_txg 7034549, loaded 971918 ms ago, max_size 539324416
1753086835   ffff9694637a4bc0 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 7034672, spa local date, vdev_id 0, ms_id 95, weight 700000000000001, selected txg 7034551 (603445 ms ago), alloc_txg 0, loaded 971921 ms ago, max_size 536100864
1753108892   ffff96947e171940 metaslab.c:2491:metaslab_load_impl(): metaslab_load: txg 22864, spa boot-pool, vdev_id 0, ms_id 96, smp_length 0, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 24024439 ms, loading_time 0 ms, ms_max_size 536870912, max size error 536870912, old_weight 740000000000001, new_weight 740000000000001
1753109497   ffff96948bc96500 metaslab.c:2604:metaslab_unload(): metaslab_unload: txg 22972, spa boot-pool, vdev_id 0, ms_id 86, weight 700000000000001, selected txg 22864 (604524 ms ago), alloc_txg 22864, loaded 24602052 ms ago, max_size 286900224

Need to run agian?

HoneyBadger · July 22, 2025, 2:22pm

One question I do not see an answer to yet - there was discussion of “upgrading to” 24.04.2.5 and later versions. What was the version of TrueNAS in use when the system initially stopped? Was it a 24.04 version, or an earlier release?

Was this a client-side drag-and-drop move from Windows 10 over SMB, or a manual “copy, then delete source” two-step process?

It should be able to be “let run” and it will eventually import the pool, provided that the DDT did not overflow RAM - if it has, then the problem is you do not have enough space to hold the necessary metadata in memory, and the comparisons of existing data against the DDT and/or updates to the DDT will take an enormous amount of time. For example your “slow spa_sync” calls showed the process starting “3003 seconds ago” at the tail end of one log - that is over 50 minutes ago, to process a transaction - and it hasn’t completed.

This section of your log also concerns me:

1753120155   ffff9806d156e500 spa.c:6916:spa_tryimport(): spa_tryimport: importing sok1001
1753120155   ffff9806d156e500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9806d156e500-sok1001, config trusted): LOADING
1753120155   ffff9806dfd4e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120156   ffff9809ec6f6500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': open error=2 timeout=1003374501/1000000000
1753120156   ffff9808d4170000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': open error=2 timeout=1011338746/1000000000
1753120156   ffff9808d4174bc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': open error=2 timeout=1011361985/1000000000
1753120156   ffff9809ec6f1940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': open error=2 timeout=1011334486/1000000000
1753120156   ffff9809ec6f3280 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': open error=2 timeout=1011345880/1000000000
1753120156   ffff9806d156e500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9806d156e500-sok1001, config untrusted): vdev tree has 1 missing top-level vdevs.
1753120156   ffff9806d156e500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9806d156e500-sok1001, config untrusted): current settings allow for maximum 0 missing top-level vdevs at this stage.
1753120156   ffff9806d156e500 spa_misc.c:415:spa_load_failed(): spa_load($import-ffff9806d156e500-sok1001, config untrusted): FAILED: unable to open vdev tree [error=2]
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 1221631789954295656, path: N/A, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 1132576499012000637, path: N/A, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 705601381597961486, path: /dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 15682408288857988643, path: /dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 16097378473779213230, path: /dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 10339353618853963483, path: /dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 4: disk, guid: 4278422679801059362, path: /dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d, can't open
1753120156   ffff9806d156e500 vdev.c:219:vdev_dbgmsg_print_tree():       vdev 5: disk, guid: 15237958642688602726, path: /dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464, healthy
1753120156   ffff9806d156e500 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9806d156e500-sok1001, config untrusted): UNLOADING

During the initial boot/loading process your pool disks are coming online one-by-one with enough delay in them to trigger an I/O timeout to the pool import process, until such point as you have 4/6 members online which triggers an import.

1753120260   ffff9808624f9940 spa.c:6916:spa_tryimport(): spa_tryimport: importing sok1001
1753120260   ffff9808624f9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9808624f9940-sok1001, config trusted): LOADING
1753120260   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753120260   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753120260   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120260   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753120261   ffff9807b9d51940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': open error=2 timeout=1010206270/1000000000
1753120261   ffff9808d684e500 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': open error=2 timeout=1014160021/1000000000
1753120261   ffff9808624f9940 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': best uberblock found for spa $import-ffff9808624f9940-sok1001. txg 6823467
1753120261   ffff9808624f9940 spa_misc.c:429:spa_load_note(): spa_load($import-ffff9808624f9940-sok1001, config untrusted): using uberblock with txg=6823467
1753120261   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/1cc4581f-326e-4645-b0ab-f6b3ba059dcb': probe done, cant_read=0 cant_write=1
1753120261   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/66b0bdd6-58b1-4959-ae1a-ba84f87d5e70': probe done, cant_read=0 cant_write=1
1753120261   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/a41d1487-ea1e-4b5e-8d3f-e2667151e464': probe done, cant_read=0 cant_write=1
1753120261   ffff980748c38000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/20db8e19-f082-4140-af01-acb3932aa38d': probe done, cant_read=0 cant_write=1
1753120262   ffff980861ddcbc0 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/05802c1a-a4d3-4120-9f49-3c18de9340e6': open error=2 timeout=1005724707/1000000000
1753120262   ffff9807f7950000 vdev.c:167:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/ee782483-dfd4-49f1-a537-845bdcd3deab': open error=2 timeout=1013733422/1000000000

Hardware with proper staggered spinup support will delay the process until all members are prepared:

(dmesg)
[    8.610523] sd 0:0:1:0: [sdb] Spinning up disk...
[   15.472511] .
[   15.637085] ready

(dbgmsg)
1753192731   ffff8ad901dc0000 spa.c:6928:spa_tryimport(): spa_tryimport: importing hdd_pool
1753192731   ffff8ad901dc0000 spa_misc.c:429:spa_load_note(): spa_load($import-ffff8ad901dc0000-hdd_pool, config trusted): LOADING
1753192731   ffff8ad901dce600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e9a5ef9b-82b3-11eb-88d8-50e5498532d1': probe done, cant_read=0 cant_write=1
1753192731   ffff8ad901dce600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e9bd2d78-82b3-11eb-88d8-50e5498532d1': probe done, cant_read=0 cant_write=1
1753192731   ffff8ad901dce600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e99a2709-82b3-11eb-88d8-50e5498532d1': probe done, cant_read=0 cant_write=1
1753192731   ffff8ad901dce600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e97a6a85-82b3-11eb-88d8-50e5498532d1': probe done, cant_read=0 cant_write=1
1753192731   ffff8ad901dce600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/28cd9500-e532-45d4-91b9-3dfe7ec9dd3b': probe done, cant_read=0 cant_write=1
1753192731   ffff8ad901dce600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e9b1198e-82b3-11eb-88d8-50e5498532d1': probe done, cant_read=0 cant_write=1
1753192731   ffff8ad901dce600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e9d3fa22-82b3-11eb-88d8-50e5498532d1': probe done, cant_read=0 cant_write=1
1753192731   ffff8ad901dce600 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/45c979b1-f8b4-4a1e-832e-8db34049d1a7': probe done, cant_read=0 cant_write=1
1753192731   ffff8ad901dc0000 vdev.c:183:vdev_dbgmsg(): disk vdev '/dev/disk/by-partuuid/e97a6a85-82b3-11eb-88d8-50e5498532d1': best uberblock found for spa $import-ffff8ad901dc0000-hdd_pool. txg 54558

leng_sok · July 22, 2025, 2:36pm

24.04.1

Cut from the original folder and paste it to the target folder
On Windows, ctrl+x and ctrl+v

You explained it in detail, thank you, I learned a lot

So, is the memory gradually filling up because of the deduplication table?
If so, is there a command that can make the system write this table to the high-speed device I prepared during import?
I got this device a while ago

HoneyBadger · July 22, 2025, 6:16pm

24.04.1 should still have been leveraging block cloning where possible, which means you hopefully were not doing full copies and going through the DDT for the moves.

The memory is gradually filling up as it is trying to “flush” the previously queued transactions to disk, and update the deduplication table. In order to know where it is updating, it has to read it into memory - but if there is not enough memory to hold the whole table at once, it will end up having to unload some portions - only to potentially have to read them back from disk again for a later chunk of data being compared. The deduplication table on-disk is made of small records, which are difficult for spinning disks to process - hence why the special or dedup vdev types are sometimes used.

This would only be possible if the Optane device was in place before the deduplicated data was written. “Moving” the tables after their creation is not possible - new entries could live there, but the previous existing data would still be in place on the spinning HDDs.

leng_sok · July 22, 2025, 8:21pm

Sorry, forgive my poor English, I don’t quite understand
“should still have been leveraging block cloning”
Does it mean that it should be copied to the destination first, and then deleted at the original location to complete the move. Instead of moving directly？

Yes I know. it is not about adding Optane to the pool right now。
What I want to ask is: when executing import, can Optane share the memory pressure?
For example,if Optane is formatted as a Swap partition, will Optane share the data that would otherwise fill up the memory when import?
Is there a way to temporarily use a hard disk as memory in linux?
I only know of the reverse method of using it, such as ramdisk.it is for windows.
Or something similar that can be implemented in zfs system？

SmallBarky · July 23, 2025, 5:12pm

The current versions of TrueNAS Scale don’t use swap.

leng_sok · July 23, 2025, 5:42pm

If swap can alleviate this problem

Is it possible to import the pool in a linux system with zfs support manually installed and then import it back to truenas?

SmallBarky · July 23, 2025, 6:48pm

You would need a system with more RAM. I don’t know if Linux with ZFS systems use Swap or they have turned it off due to problems also.

Is your system still trying to make progress? I don’t know what to look at using the command line to tell. I guess you could look at TOP or HTOP and the input / output statistics

leng_sok · July 24, 2025, 8:56am

I haven’t tried it yet, I’m waiting for lsi3008 and a new backplane

If this idea is worth trying, I’d like to get some suggestions, such as the version, etc.

etorix · July 25, 2025, 7:07am

@HoneyBadger is there a way to re-enable swap on TrueNAS SCALE? Preferably with the Optane drive as swap device?
Or should OP rather try to import his pool, at least read-only, on a Linux distro with ZFS (and lots of swap space)?

HoneyBadger · July 25, 2025, 7:08pm

I don’t know that swap would help here I believe ZFS ARC is designed to avoid the traditional Linux swap filesystem - it’s designed to be among other things a fast memory cache in front of disk, so swapping said cache back to disk is antithetical to purpose. I imagine once memory gets full, ZFS is going to unload metadata on its own.

Because this was on TrueNAS 24.04, it means that even if you did the “copy first, and manually delete” it would have used the lightweight “block cloning” functionality instead of a full copy (which would engage the deduplication tables) - I am trying to figure out which scenario would have provoked the problem most significantly.

To summarize - you are still experiencing system reboots even with 128GB?

etorix · July 25, 2025, 9:14pm

It seems that TrueNAS still hangs while trying to import, so this is where having a large swap to hold the DDT might help. Might.

leng_sok · July 26, 2025, 5:47am

yes

If this is an artificial choice that makes the system prefer to operate metadata unloading and reloading in memory rather than using swap, the metadata operation mechanism should be reliable.

So, does the inability to complete metadata operations and cause a reboot mean that hardware problems are more likely to occur, rather than just insufficient memory?

What else can I try?

HoneyBadger · July 29, 2025, 3:17pm

The metadata is loaded from disk into RAM - so to exhaust RAM and move it back to swap and then have to pull it back into RAM from swap would mean ZFS is unaware of it, rather than consciously unloading the data and requesting it from stable state on the pool instead. I don’t know if there is a way to override this behavior.

My thought is that the system has crossed over the threshold of RAM capacity for deduplication, and as such has a lot of “pending work” to complete - but the problem is that there is an internal timer that is triggering based on a belief that the system is non-responsive. zfs_deadman_failmode defaults to wait on TrueNAS, so in theory it should wait forever and eventually crawl its way to a successful convergence and import.

What does the output of sudo ipmitool mc watchdog get result in? If your hardware has its own timer that is triggering the reboot, we may need to disable that.

Topic		Replies	Views
Corrupted Pool TrueNAS General SCALE , pool , corruption	25	324	December 30, 2024
Discs exported? How do I access the data on them? TrueNAS General	15	262	March 8, 2025
Possibly lost pool, all advice appreciated TrueNAS General SCALE , ZFS	15	269	April 25, 2025
Scrub task stuck TrueNAS General SCALE , Hardware , ZFS	33	1348	April 29, 2024
25.04.02 - "Services" consuming ram causing OOM killer to kill processes TrueNAS General	5	209	August 7, 2025

在win10通过smb整理文件时进度卡死，然后重启（从UI操作）之后挂载不上pool/Progress stuck while organizing files via smb in win10, then after reboot (operating from UI) the mount won't go on pool

Related topics