Please help me Identify ZFS Errors

ok found where the scrub started in the log

Ive opened in VI so i can search in it easier.

log during scrub

1784900725 ffff99fa6406b300 spa_history.c:294:spa_history_log_sync(): command: zpool scrub HDDs
1784900725 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900725 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900725 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900725 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900725 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900725 ffff99fa6406b300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for HDDs txg 9869354; bm=423/0/0/1024
1784900730 ffff99fa6406b300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 423 (HDDs/ix-apps/docker) with min=1 max=9869352; suspending=1
1784900730 ffff99fa6406b300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 12016118 blocks of HDDs in 4988ms (1 os’s, 9634654 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1784900730 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900730 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900730 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900730 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900730 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900730 ffff99fa6406b300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for HDDs txg 9869355; bm=423/2683552/0/0
1784900731 ffff99fa6406b300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 423 (HDDs/ix-apps/docker) with min=1 max=9869352; suspending=0
1784900731 ffff99fa6406b300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 444 (HDDs/Photos) with min=1 max=9869352; suspending=0
1784900735 ffff99fa6406b300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 527 (HDDs/Movies) with min=1 max=9869352; suspending=1
1784900735 ffff99fa6406b300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 4480320 blocks of HDDs in 4994ms (3 os’s, 1099446 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1784900735 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900735 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900735 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900735 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900735 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900735 ffff99fa6406b300 dsl_scan.c:4569:dsl_scan_sync(): doing scan sync for HDDs txg 9869356; bm=527/190/0/6501
1784900739 ffff99fa6406b300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 527 (HDDs/Movies) with min=1 max=9869352; suspending=0
1784900740 ffff99fa6406b300 dsl_scan.c:2861:dsl_scan_visitds(): scanned dataset 534 (HDDs/Shows) with min=1 max=9869352; suspending=1
1784900744 ffff99fa6406b300 dsl_scan.c:4600:dsl_scan_sync(): scan visited 2671436 blocks of HDDs in 8607ms (2 os’s, 42269 holes, 0 < mintxg, 0 in ddt, 0 > maxtxg)
1784900744 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900744 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900744 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900744 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900744 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900744 ffff99fa6406b300 dsl_scan.c:4529:dsl_scan_sync(): begin scan clearing for HDDs
1784900749 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 35945 blocks for HDDs (1 segs) in 5094ms (avg_block_size = 155644, avg_seg_size = 5615476736)
1784900749 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900749 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900749 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900749 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900749 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900754 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 38248 blocks for HDDs (2 segs) in 5069ms (avg_block_size = 155653, avg_seg_size = 2986696704)
1784900755 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900755 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900755 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900755 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900755 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0

1784900760 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 41003 blocks for HDDs (1 segs) in 5081ms (avg_block_size = 155641, avg_seg_size = 6399918080)
1784900760 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900760 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900760 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900760 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900760 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900765 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 40476 blocks for HDDs (2 segs) in 5074ms (avg_block_size = 155606, avg_seg_size = 3157880832)
1784900765 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900765 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900765 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900765 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900765 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900770 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 38263 blocks for HDDs (1 segs) in 5085ms (avg_block_size = 155637, avg_seg_size = 5958942720)
1784900770 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900770 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900770 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900770 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900770 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900775 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 37502 blocks for HDDs (2 segs) in 4929ms (avg_block_size = 155625, avg_seg_size = 2921701376)
1784900776 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900776 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900776 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900776 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900776 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900781 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 37647 blocks for HDDs (2 segs) in 5196ms (avg_block_size = 155648, avg_seg_size = 2934349824)
1784900781 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900781 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900781 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900781 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900781 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900786 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 37990 blocks for HDDs (1 segs) in 5087ms (avg_block_size = 155162, avg_seg_size = 5900599296)
1784900786 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900786 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900786 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900786 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900786 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900791 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 36575 blocks for HDDs (2 segs) in 5086ms (avg_block_size = 155058, avg_seg_size = 2837356544)
1784900792 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900792 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900792 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900792 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900792 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0

1784900797 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 38298 blocks for HDDs (1 segs) in 5087ms (avg_block_size = 155091, avg_seg_size = 5941075968)
1784900797 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900797 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900797 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900797 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900797 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900802 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 37799 blocks for HDDs (2 segs) in 5089ms (avg_block_size = 154962, avg_seg_size = 2929766400)
1784900802 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900802 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900802 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900802 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900802 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900807 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 38024 blocks for HDDs (1 segs) in 5086ms (avg_block_size = 155241, avg_seg_size = 5906472960)
1784900808 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900808 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900808 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900808 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900808 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900813 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 36717 blocks for HDDs (2 segs) in 4973ms (avg_block_size = 154998, avg_seg_size = 2847686656)
1784900813 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900813 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900813 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900813 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900813 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900818 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 36283 blocks for HDDs (1 segs) in 5124ms (avg_block_size = 155155, avg_seg_size = 5631156224)
1784900818 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900818 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900818 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900818 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900818 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900823 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 36566 blocks for HDDs (2 segs) in 5091ms (avg_block_size = 155101, avg_seg_size = 2837217280)
1784900823 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900823 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900823 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900823 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900823 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900828 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 35668 blocks for HDDs (1 segs) in 5090ms (avg_block_size = 155078, avg_seg_size = 5531590656)
1784900829 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900829 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900829 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900829 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900829 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0

1784900834 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 38332 blocks for HDDs (2 segs) in 5084ms (avg_block_size = 154860, avg_seg_size = 2969800704)
1784900834 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900834 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900834 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900834 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900834 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900839 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 37225 blocks for HDDs (1 segs) in 5085ms (avg_block_size = 154353, avg_seg_size = 5748711424)
1784900839 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900839 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900839 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900839 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900839 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900844 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 38042 blocks for HDDs (1 segs) in 5088ms (avg_block_size = 153266, avg_seg_size = 5830909952)
1784900845 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900845 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900845 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900845 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900845 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900850 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 36775 blocks for HDDs (2 segs) in 4955ms (avg_block_size = 155248, avg_seg_size = 2856488960)
1784900850 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900850 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900850 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900850 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900850 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900855 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 37085 blocks for HDDs (1 segs) in 5140ms (avg_block_size = 155171, avg_seg_size = 5754642432)
1784900855 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900855 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900855 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900855 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900855 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900860 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 37501 blocks for HDDs (2 segs) in 5089ms (avg_block_size = 154991, avg_seg_size = 2908790784)
1784900860 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900860 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900860 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900860 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900860 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900865 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 36680 blocks for HDDs (1 segs) in 5087ms (avg_block_size = 154993, avg_seg_size = 5686411264)
1784900866 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900866 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900866 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900866 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900866 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0

1784900871 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 37872 blocks for HDDs (1 segs) in 5089ms (avg_block_size = 154763, avg_seg_size = 5862522880)
1784900871 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900871 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900871 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900871 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900871 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900876 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 36541 blocks for HDDs (2 segs) in 5090ms (avg_block_size = 154849, avg_seg_size = 2832261120)
1784900876 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900876 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900876 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900876 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900876 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900881 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 36903 blocks for HDDs (1 segs) in 5088ms (avg_block_size = 155098, avg_seg_size = 5728124928)
1784900882 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900882 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900882 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900882 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900882 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900887 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 33870 blocks for HDDs (2 segs) in 4981ms (avg_block_size = 155277, avg_seg_size = 2633400320)
1784900887 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900887 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900887 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900887 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900887 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900892 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 34116 blocks for HDDs (1 segs) in 5162ms (avg_block_size = 155292, avg_seg_size = 5297700864)
1784900892 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900892 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900892 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900892 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900892 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900897 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 32110 blocks for HDDs (2 segs) in 5100ms (avg_block_size = 154725, avg_seg_size = 2484826112)
1784900897 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900897 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900897 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900897 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900897 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900903 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 31230 blocks for HDDs (1 segs) in 5105ms (avg_block_size = 155206, avg_seg_size = 4849958912)
1784900903 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900903 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900903 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900903 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900903 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0
1784900908 ffff99fa6406b300 dsl_scan.c:4641:dsl_scan_sync(): scan issued 34965 blocks for HDDs (2 segs) in 5095ms (avg_block_size = 155032, avg_seg_size = 2710503424)
1784900908 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 0: traversing from min_txg=1 bookmark -1/0/0/1024
1784900908 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 1: traversing from min_txg=-1 bookmark 0/0/0/0
1784900908 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 2: traversing from min_txg=-1 bookmark 0/0/0/0
1784900908 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 3: traversing from min_txg=-1 bookmark 0/0/0/0
1784900908 ffff99fa6406b300 bptree.c:226:bptree_iterate(): bptree index 4: traversing from min_txg=-1 bookmark 0/0/0/0

@HoneyBadger do you see anything out of the ordinary?

@Stephen_Roberts I’m curious. Does this reveal there is leakage in the pool?

zpool get leaked,freeing HDDs

It should be 0 if there is no leakage. “freeing” should also be 0 by now, assuming that you haven’t been deleting large amounts of data just now.


What is the value of this parameter? 1 or 0? If it’s 0, ZFS might try over and over again to reclaim space that is likely forever lost (“leaked”). If so, the earlier get leaked command might show nothing.

cat /sys/module/zfs/parameters/zfs_free_leak_on_eio

You might consider setting this parameter to 1 for ZFS to ignore the leakage and move on. I would ask someone more versed in this if it’s safe to do. It likely is safe, especially if the free space can never be reclaimed anyways.

Didn’t even know this was a thing

This is really beyond my expertise, which is why I’m hoping @HoneyBadger might be able to provide some insight.

I have a theory on what might be going on. It could be wrong, which wouldn’t surprise me.

Before he replaced his bad/failing RAM, an attempt was made to free up space on the pool after deleting some data. Something went wrong, perhaps from the bad RAM. Now there is some space still being referenced as “allocated”, even though the data has since been deleted. Because this is a permanent corruption, ZFS is unable to safely make that space usable again. The pool can still work normally, but the “allocated” space can never be freed. This space can be marked as “leaked”.

I’m not sure if this is really what is happening or if it’s related to his persistent “2 errors” after every clear and scrub. I find the timing of the bad RAM and its replacement interesting.

@Stephen_Roberts, until someone more experienced chimes in, your output might clue us what is going on.



If this is 0
:backhand_index_pointing_down:
cat /sys/module/zfs/parameters/zfs_free_leak_on_eio



and this has a value above 0 for “freeing”
:backhand_index_pointing_down:
zpool get leaked,freeing HDDs



then the combination of previously having bad RAM + recurring consistent checksum errors means that the “solution” might be to allow ZFS to permanently “leak” this space.

Same here 10 days ago…

Nothing unusual from the log there, that’s what I’d expect from a zpool scrub in action largely.

Did we get anything from the check for leaked,freeing ?

Sorry i have been out for a few hours. Here is the output of my Leaked command

zpool get leaked,freeing HDDs
NAME PROPERTY VALUE SOURCE
HDDs leaked 0 -
HDDs freeing 942M -

cat /sys/module/zfs/parameters/zfs_free_leak_on_eio
0

I’m going to guess that the value for “freeing” never meaningfully drops below that number? You might end up having to settle for permanently leaking 942 MB in this pool.[1] (I’m assuming you haven’t been deleting large amounts of data during this time?)

I’ll leave it there, since this is going beyond my comfort level. The “ad” known as HoneyBadger may better understand what’s going on.

As for what caused this, who knows? I doubt it will clear the checksum errors if the corruption is at lower level pool metadata.


  1. It’s possible to enable the ZFS parameter, reimport the pool, allow it to permanently leak the space after failing to free it, disable the parameter again, and finally reimport the pool again. Your pool will forever lose this “leaked” space. ↩︎

if i had enough spare room i would move all the data off and rebuild but with current HDD prices that is not really at option at the moment.

Ill leave it overnight and check that number again in the morning see if it ever changes from 942M

A healthy system should eventually reach 0, especially if you aren’t actively deleting files or destroying datasets and snapshots. Reclaiming previously used space in ZFS is not instantaneous, but it shouldn’t take forever.

I’m glad that you replaced RAM that was failing memtests, which coincided with earlier issues. The problem is that passing memtests should be confirmed before you commit to a ZFS pool. It’s like doing HDD burn-ins, but for your RAM sticks. :grinning_face:

The ram had been working ok for quite a long time. It is old but in use hardware that the server runs on and ran ok for many months before i started having trouble.

The new Ram i did a 24 hour memtest with and it passed with no errors.

I recently also had checksum errors on both drives of a mirror and two corrupted files due to this, which was incredibly weird (different drive models, how could they possible fail at the same time at the same sector?).

In the end it turned out that one of the two 16GB SO-DIMM DDR4 modules was having incredibly sporadic memory errors. Not the kind of crazy errors that would make the system crash or behave weirdly, but just enough to occasionally damage a file. ZFS caught this way before I would have otherwise noticed and by running memtest for a long time (just 2 wrong bits in 1h12m of memtest86 v11.7!) I was able to isolate the defective module and replace it.