I was exporting a 49 disk (557 TiB) pool from the GUI. It stopped at 80% complete for about 10 minutes. Trying to use zfs or zpool in the shell while it was exporting just hung. Initially I saw a lot of disk activity, but that stopped after a minute or two. Anyone ever experienced this? It does show up as importable but I’m not going to try it until I move it to a new system.
Have this in dmesg:
[2529050.520261] INFO: task middlewared (wo:3507597 blocked for more than 120 seconds.
[2529050.520821] Tainted: P OE 6.12.15-production+truenas #1
[2529050.521286] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2529050.521766] task:middlewared (wo state:D stack:0 pid:3507597 tgid:3507597 ppid:11215 flags:0x00004002
[2529050.521781] Call Trace:
[2529050.521786] <TASK>
[2529050.521799] __schedule+0x461/0xa10
[2529050.521821] schedule+0x27/0xd0
[2529050.521830] io_schedule+0x46/0x70
[2529050.521842] cv_wait_common+0xa9/0x130 [spl]
[2529050.521878] ? __pfx_autoremove_wake_function+0x10/0x10
[2529050.521895] txg_wait_synced_impl+0xc0/0x110 [zfs]
[2529050.522451] txg_wait_synced+0x10/0x40 [zfs]
[2529050.523087] spa_export_common+0x69c/0x6b0 [zfs]
[2529050.523679] zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
[2529050.524221] zfsdev_ioctl+0x53/0xe0 [zfs]
[2529050.524639] __x64_sys_ioctl+0x94/0xd0
[2529050.524652] do_syscall_64+0x82/0x190
[2529050.524666] ? srso_return_thunk+0x5/0x5f
[2529050.524675] ? __count_memcg_events+0x53/0xf0
[2529050.524686] ? srso_return_thunk+0x5/0x5f
[2529050.524693] ? count_memcg_events.constprop.0+0x1a/0x30
[2529050.524702] ? srso_return_thunk+0x5/0x5f
[2529050.524708] ? handle_mm_fault+0xae/0x2d0
[2529050.524720] ? srso_return_thunk+0x5/0x5f
[2529050.524726] ? do_user_addr_fault+0x379/0x670
[2529050.524740] ? srso_return_thunk+0x5/0x5f
[2529050.524746] ? exc_page_fault+0x76/0x190
[2529050.524756] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2529050.524766] RIP: 0033:0x7f2cc6334cdb
[2529050.524773] RSP: 002b:00007ffc51e257a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[2529050.524782] RAX: ffffffffffffffda RBX: 0000000018cadd00 RCX: 00007f2cc6334cdb
[2529050.524788] RDX: 00007ffc51e25810 RSI: 0000000000005a03 RDI: 000000000000001a
[2529050.524793] RBP: 00007ffc51e28e00 R08: 0000000000000007 R09: 0000000000000000
[2529050.524798] R10: a302aa9724c8736d R11: 0000000000000246 R12: 00007ffc51e25810
[2529050.524802] R13: 0000000018cadd10 R14: 0000000000000000 R15: 0000000000000001
[2529050.524819] </TASK>
[2529050.524851] INFO: task middlewared (wo:3509128 blocked for more than 120 seconds.
[2529050.525321] Tainted: P OE 6.12.15-production+truenas #1
[2529050.525801] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2529050.526326] task:middlewared (wo state:D stack:0 pid:3509128 tgid:3509128 ppid:11181 flags:0x00000002
[2529050.526338] Call Trace:
[2529050.526342] <TASK>
[2529050.526348] __schedule+0x461/0xa10
[2529050.526363] schedule+0x27/0xd0
[2529050.526371] cv_wait_common+0xef/0x130 [spl]
[2529050.526400] ? __pfx_autoremove_wake_function+0x10/0x10
[2529050.526411] spa_lookup+0x3e/0xe0 [zfs]
[2529050.526875] spa_open_common+0x75/0x440 [zfs]
[2529050.527417] ? srso_return_thunk+0x5/0x5f
[2529050.527426] ? __lruvec_stat_mod_folio+0xa4/0xd0
[2529050.527439] spa_get_stats+0x4e/0x210 [zfs]
[2529050.527914] zfs_ioc_pool_stats+0x40/0x90 [zfs]
[2529050.528545] zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
[2529050.529132] zfsdev_ioctl+0x53/0xe0 [zfs]
[2529050.529719] __x64_sys_ioctl+0x94/0xd0
[2529050.529732] do_syscall_64+0x82/0x190
[2529050.529742] ? srso_return_thunk+0x5/0x5f
[2529050.529750] ? audit_reset_context+0x232/0x300
[2529050.529763] ? srso_return_thunk+0x5/0x5f
[2529050.529771] ? syscall_exit_to_user_mode_prepare+0x148/0x170
[2529050.529783] ? srso_return_thunk+0x5/0x5f
[2529050.529791] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529050.529800] ? srso_return_thunk+0x5/0x5f
[2529050.529808] ? do_syscall_64+0x8e/0x190
[2529050.529816] ? srso_return_thunk+0x5/0x5f
[2529050.529824] ? do_syscall_64+0x8e/0x190
[2529050.529831] ? srso_return_thunk+0x5/0x5f
[2529050.529839] ? srso_return_thunk+0x5/0x5f
[2529050.529847] ? cp_new_stat+0x134/0x170
[2529050.529859] ? srso_return_thunk+0x5/0x5f
[2529050.529867] ? audit_filter_rules.constprop.0+0x140/0x1120
[2529050.529884] ? srso_return_thunk+0x5/0x5f
[2529050.529892] ? __audit_filter_op+0xaf/0x110
[2529050.529902] ? srso_return_thunk+0x5/0x5f
[2529050.529913] ? srso_return_thunk+0x5/0x5f
[2529050.529920] ? audit_reset_context+0x232/0x300
[2529050.529931] ? srso_return_thunk+0x5/0x5f
[2529050.529939] ? syscall_exit_to_user_mode_prepare+0x148/0x170
[2529050.529950] ? srso_return_thunk+0x5/0x5f
[2529050.529958] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529050.529967] ? srso_return_thunk+0x5/0x5f
[2529050.529976] ? do_syscall_64+0x8e/0x190
[2529050.529983] ? srso_return_thunk+0x5/0x5f
[2529050.529990] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529050.529999] ? srso_return_thunk+0x5/0x5f
[2529050.530007] ? do_syscall_64+0x8e/0x190
[2529050.530016] ? srso_return_thunk+0x5/0x5f
[2529050.530024] ? do_syscall_64+0x8e/0x190
[2529050.530031] ? srso_return_thunk+0x5/0x5f
[2529050.530038] ? do_syscall_64+0x8e/0x190
[2529050.530045] ? exc_page_fault+0x76/0x190
[2529050.530055] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2529050.530064] RIP: 0033:0x7f30fda6fcdb
[2529050.530071] RSP: 002b:00007ffcbc041f10 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[2529050.530079] RAX: ffffffffffffffda RBX: 00000000407124d0 RCX: 00007f30fda6fcdb
[2529050.530085] RDX: 00007ffcbc041f90 RSI: 0000000000005a05 RDI: 000000000000001a
[2529050.530090] RBP: 00007ffcbc045580 R08: 00007f30fdb45430 R09: 00000000409516c0
[2529050.530095] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcbc041f90
[2529050.530099] R13: 00000000407124d0 R14: 000000004091d5b0 R15: 00007ffcbc045594
[2529050.530114] </TASK>
[2529171.352016] INFO: task middlewared (wo:3507597 blocked for more than 241 seconds.
[2529171.352650] Tainted: P OE 6.12.15-production+truenas #1
[2529171.353176] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2529171.353696] task:middlewared (wo state:D stack:0 pid:3507597 tgid:3507597 ppid:11215 flags:0x00004002
[2529171.353709] Call Trace:
[2529171.353714] <TASK>
[2529171.353725] __schedule+0x461/0xa10
[2529171.353746] schedule+0x27/0xd0
[2529171.353754] io_schedule+0x46/0x70
[2529171.353764] cv_wait_common+0xa9/0x130 [spl]
[2529171.353796] ? __pfx_autoremove_wake_function+0x10/0x10
[2529171.353812] txg_wait_synced_impl+0xc0/0x110 [zfs]
[2529171.354289] txg_wait_synced+0x10/0x40 [zfs]
[2529171.354743] spa_export_common+0x69c/0x6b0 [zfs]
[2529171.355234] zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
[2529171.355684] zfsdev_ioctl+0x53/0xe0 [zfs]
[2529171.356127] __x64_sys_ioctl+0x94/0xd0
[2529171.356140] do_syscall_64+0x82/0x190
[2529171.356154] ? srso_return_thunk+0x5/0x5f
[2529171.356164] ? __count_memcg_events+0x53/0xf0
[2529171.356174] ? srso_return_thunk+0x5/0x5f
[2529171.356181] ? count_memcg_events.constprop.0+0x1a/0x30
[2529171.356190] ? srso_return_thunk+0x5/0x5f
[2529171.356197] ? handle_mm_fault+0xae/0x2d0
[2529171.356208] ? srso_return_thunk+0x5/0x5f
[2529171.356215] ? do_user_addr_fault+0x379/0x670
[2529171.356229] ? srso_return_thunk+0x5/0x5f
[2529171.356235] ? exc_page_fault+0x76/0x190
[2529171.356245] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2529171.356255] RIP: 0033:0x7f2cc6334cdb
[2529171.356262] RSP: 002b:00007ffc51e257a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[2529171.356271] RAX: ffffffffffffffda RBX: 0000000018cadd00 RCX: 00007f2cc6334cdb
[2529171.356277] RDX: 00007ffc51e25810 RSI: 0000000000005a03 RDI: 000000000000001a
[2529171.356282] RBP: 00007ffc51e28e00 R08: 0000000000000007 R09: 0000000000000000
[2529171.356287] R10: a302aa9724c8736d R11: 0000000000000246 R12: 00007ffc51e25810
[2529171.356292] R13: 0000000018cadd10 R14: 0000000000000000 R15: 0000000000000001
[2529171.356309] </TASK>
[2529171.356324] INFO: task middlewared (wo:3509128 blocked for more than 241 seconds.
[2529171.356860] Tainted: P OE 6.12.15-production+truenas #1
[2529171.357394] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2529171.357943] task:middlewared (wo state:D stack:0 pid:3509128 tgid:3509128 ppid:11181 flags:0x00000002
[2529171.357953] Call Trace:
[2529171.357957] <TASK>
[2529171.357964] __schedule+0x461/0xa10
[2529171.357979] schedule+0x27/0xd0
[2529171.357986] cv_wait_common+0xef/0x130 [spl]
[2529171.358016] ? __pfx_autoremove_wake_function+0x10/0x10
[2529171.358027] spa_lookup+0x3e/0xe0 [zfs]
[2529171.358490] spa_open_common+0x75/0x440 [zfs]
[2529171.358971] ? srso_return_thunk+0x5/0x5f
[2529171.358979] ? __lruvec_stat_mod_folio+0xa4/0xd0
[2529171.358992] spa_get_stats+0x4e/0x210 [zfs]
[2529171.359460] zfs_ioc_pool_stats+0x40/0x90 [zfs]
[2529171.359901] zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
[2529171.360345] zfsdev_ioctl+0x53/0xe0 [zfs]
[2529171.360762] __x64_sys_ioctl+0x94/0xd0
[2529171.360772] do_syscall_64+0x82/0x190
[2529171.360781] ? srso_return_thunk+0x5/0x5f
[2529171.360788] ? audit_reset_context+0x232/0x300
[2529171.360800] ? srso_return_thunk+0x5/0x5f
[2529171.360806] ? syscall_exit_to_user_mode_prepare+0x148/0x170
[2529171.360818] ? srso_return_thunk+0x5/0x5f
[2529171.360824] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529171.360832] ? srso_return_thunk+0x5/0x5f
[2529171.360839] ? do_syscall_64+0x8e/0x190
[2529171.360845] ? srso_return_thunk+0x5/0x5f
[2529171.360852] ? do_syscall_64+0x8e/0x190
[2529171.360858] ? srso_return_thunk+0x5/0x5f
[2529171.360865] ? srso_return_thunk+0x5/0x5f
[2529171.360872] ? cp_new_stat+0x134/0x170
[2529171.360883] ? srso_return_thunk+0x5/0x5f
[2529171.360889] ? audit_filter_rules.constprop.0+0x140/0x1120
[2529171.360904] ? srso_return_thunk+0x5/0x5f
[2529171.360910] ? __audit_filter_op+0xaf/0x110
[2529171.360919] ? srso_return_thunk+0x5/0x5f
[2529171.360928] ? srso_return_thunk+0x5/0x5f
[2529171.360934] ? audit_reset_context+0x232/0x300
[2529171.360944] ? srso_return_thunk+0x5/0x5f
[2529171.360950] ? syscall_exit_to_user_mode_prepare+0x148/0x170
[2529171.360959] ? srso_return_thunk+0x5/0x5f
[2529171.360966] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529171.360973] ? srso_return_thunk+0x5/0x5f
[2529171.360980] ? do_syscall_64+0x8e/0x190
[2529171.360986] ? srso_return_thunk+0x5/0x5f
[2529171.360993] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529171.361000] ? srso_return_thunk+0x5/0x5f
[2529171.361007] ? do_syscall_64+0x8e/0x190
[2529171.361014] ? srso_return_thunk+0x5/0x5f
[2529171.361020] ? do_syscall_64+0x8e/0x190
[2529171.361026] ? srso_return_thunk+0x5/0x5f
[2529171.361033] ? do_syscall_64+0x8e/0x190
[2529171.361039] ? exc_page_fault+0x76/0x190
[2529171.361047] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2529171.361054] RIP: 0033:0x7f30fda6fcdb
[2529171.361061] RSP: 002b:00007ffcbc041f10 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[2529171.361068] RAX: ffffffffffffffda RBX: 00000000407124d0 RCX: 00007f30fda6fcdb
[2529171.361074] RDX: 00007ffcbc041f90 RSI: 0000000000005a05 RDI: 000000000000001a
[2529171.361078] RBP: 00007ffcbc045580 R08: 00007f30fdb45430 R09: 00000000409516c0
[2529171.361083] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcbc041f90
[2529171.361088] R13: 00000000407124d0 R14: 000000004091d5b0 R15: 00007ffcbc045594
[2529171.361103] </TASK>
[2529292.175874] INFO: task middlewared (wo:3507597 blocked for more than 362 seconds.
[2529292.176577] Tainted: P OE 6.12.15-production+truenas #1
[2529292.177132] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2529292.177671] task:middlewared (wo state:D stack:0 pid:3507597 tgid:3507597 ppid:11215 flags:0x00004002
[2529292.177684] Call Trace:
[2529292.177689] <TASK>
[2529292.177700] __schedule+0x461/0xa10
[2529292.177721] schedule+0x27/0xd0
[2529292.177728] io_schedule+0x46/0x70
[2529292.177739] cv_wait_common+0xa9/0x130 [spl]
[2529292.177781] ? __pfx_autoremove_wake_function+0x10/0x10
[2529292.177797] txg_wait_synced_impl+0xc0/0x110 [zfs]
[2529292.178278] txg_wait_synced+0x10/0x40 [zfs]
[2529292.178750] spa_export_common+0x69c/0x6b0 [zfs]
[2529292.179217] zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
[2529292.179669] zfsdev_ioctl+0x53/0xe0 [zfs]
[2529292.180087] __x64_sys_ioctl+0x94/0xd0
[2529292.180100] do_syscall_64+0x82/0x190
[2529292.180113] ? srso_return_thunk+0x5/0x5f
[2529292.180122] ? __count_memcg_events+0x53/0xf0
[2529292.180133] ? srso_return_thunk+0x5/0x5f
[2529292.180140] ? count_memcg_events.constprop.0+0x1a/0x30
[2529292.180149] ? srso_return_thunk+0x5/0x5f
[2529292.180155] ? handle_mm_fault+0xae/0x2d0
[2529292.180166] ? srso_return_thunk+0x5/0x5f
[2529292.180173] ? do_user_addr_fault+0x379/0x670
[2529292.180186] ? srso_return_thunk+0x5/0x5f
[2529292.180193] ? exc_page_fault+0x76/0x190
[2529292.180202] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2529292.180213] RIP: 0033:0x7f2cc6334cdb
[2529292.180220] RSP: 002b:00007ffc51e257a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[2529292.180229] RAX: ffffffffffffffda RBX: 0000000018cadd00 RCX: 00007f2cc6334cdb
[2529292.180234] RDX: 00007ffc51e25810 RSI: 0000000000005a03 RDI: 000000000000001a
[2529292.180239] RBP: 00007ffc51e28e00 R08: 0000000000000007 R09: 0000000000000000
[2529292.180244] R10: a302aa9724c8736d R11: 0000000000000246 R12: 00007ffc51e25810
[2529292.180249] R13: 0000000018cadd10 R14: 0000000000000000 R15: 0000000000000001
[2529292.180266] </TASK>
[2529292.180279] INFO: task middlewared (wo:3509128 blocked for more than 362 seconds.
[2529292.180833] Tainted: P OE 6.12.15-production+truenas #1
[2529292.181395] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2529292.181964] task:middlewared (wo state:D stack:0 pid:3509128 tgid:3509128 ppid:11181 flags:0x00000002
[2529292.181974] Call Trace:
[2529292.181979] <TASK>
[2529292.181987] __schedule+0x461/0xa10
[2529292.182003] schedule+0x27/0xd0
[2529292.182011] cv_wait_common+0xef/0x130 [spl]
[2529292.182040] ? __pfx_autoremove_wake_function+0x10/0x10
[2529292.182051] spa_lookup+0x3e/0xe0 [zfs]
[2529292.182515] spa_open_common+0x75/0x440 [zfs]
[2529292.182997] ? srso_return_thunk+0x5/0x5f
[2529292.183005] ? __lruvec_stat_mod_folio+0xa4/0xd0
[2529292.183017] spa_get_stats+0x4e/0x210 [zfs]
[2529292.183486] zfs_ioc_pool_stats+0x40/0x90 [zfs]
[2529292.183926] zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
[2529292.184373] zfsdev_ioctl+0x53/0xe0 [zfs]
[2529292.184793] __x64_sys_ioctl+0x94/0xd0
[2529292.184803] do_syscall_64+0x82/0x190
[2529292.184812] ? srso_return_thunk+0x5/0x5f
[2529292.184819] ? audit_reset_context+0x232/0x300
[2529292.184831] ? srso_return_thunk+0x5/0x5f
[2529292.184838] ? syscall_exit_to_user_mode_prepare+0x148/0x170
[2529292.184849] ? srso_return_thunk+0x5/0x5f
[2529292.184855] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529292.184863] ? srso_return_thunk+0x5/0x5f
[2529292.184870] ? do_syscall_64+0x8e/0x190
[2529292.184876] ? srso_return_thunk+0x5/0x5f
[2529292.184883] ? do_syscall_64+0x8e/0x190
[2529292.184889] ? srso_return_thunk+0x5/0x5f
[2529292.184895] ? srso_return_thunk+0x5/0x5f
[2529292.184902] ? cp_new_stat+0x134/0x170
[2529292.184912] ? srso_return_thunk+0x5/0x5f
[2529292.184919] ? audit_filter_rules.constprop.0+0x140/0x1120
[2529292.184933] ? srso_return_thunk+0x5/0x5f
[2529292.184940] ? __audit_filter_op+0xaf/0x110
[2529292.184948] ? srso_return_thunk+0x5/0x5f
[2529292.184957] ? srso_return_thunk+0x5/0x5f
[2529292.184964] ? audit_reset_context+0x232/0x300
[2529292.184973] ? srso_return_thunk+0x5/0x5f
[2529292.184980] ? syscall_exit_to_user_mode_prepare+0x148/0x170
[2529292.184988] ? srso_return_thunk+0x5/0x5f
[2529292.184995] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529292.185003] ? srso_return_thunk+0x5/0x5f
[2529292.185009] ? do_syscall_64+0x8e/0x190
[2529292.185015] ? srso_return_thunk+0x5/0x5f
[2529292.185022] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529292.185029] ? srso_return_thunk+0x5/0x5f
[2529292.185036] ? do_syscall_64+0x8e/0x190
[2529292.185043] ? srso_return_thunk+0x5/0x5f
[2529292.185050] ? do_syscall_64+0x8e/0x190
[2529292.185056] ? srso_return_thunk+0x5/0x5f
[2529292.185063] ? do_syscall_64+0x8e/0x190
[2529292.185069] ? exc_page_fault+0x76/0x190
[2529292.185077] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2529292.185084] RIP: 0033:0x7f30fda6fcdb
[2529292.185090] RSP: 002b:00007ffcbc041f10 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[2529292.185098] RAX: ffffffffffffffda RBX: 00000000407124d0 RCX: 00007f30fda6fcdb
[2529292.185103] RDX: 00007ffcbc041f90 RSI: 0000000000005a05 RDI: 000000000000001a
[2529292.185108] RBP: 00007ffcbc045580 R08: 00007f30fdb45430 R09: 00000000409516c0
[2529292.185112] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcbc041f90
[2529292.185117] R13: 00000000407124d0 R14: 000000004091d5b0 R15: 00007ffcbc045594
[2529292.185132] </TASK>
[2529413.003618] INFO: task middlewared (wo:3507597 blocked for more than 483 seconds.
[2529413.004277] Tainted: P OE 6.12.15-production+truenas #1
[2529413.004837] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2529413.005395] task:middlewared (wo state:D stack:0 pid:3507597 tgid:3507597 ppid:11215 flags:0x00004002
[2529413.005407] Call Trace:
[2529413.005412] <TASK>
[2529413.005423] __schedule+0x461/0xa10
[2529413.005443] schedule+0x27/0xd0
[2529413.005450] io_schedule+0x46/0x70
[2529413.005460] cv_wait_common+0xa9/0x130 [spl]
[2529413.005492] ? __pfx_autoremove_wake_function+0x10/0x10
[2529413.005507] txg_wait_synced_impl+0xc0/0x110 [zfs]
[2529413.005982] txg_wait_synced+0x10/0x40 [zfs]
[2529413.006462] spa_export_common+0x69c/0x6b0 [zfs]
[2529413.006982] zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
[2529413.007435] zfsdev_ioctl+0x53/0xe0 [zfs]
[2529413.007877] __x64_sys_ioctl+0x94/0xd0
[2529413.007889] do_syscall_64+0x82/0x190
[2529413.007903] ? srso_return_thunk+0x5/0x5f
[2529413.007911] ? __count_memcg_events+0x53/0xf0
[2529413.007922] ? srso_return_thunk+0x5/0x5f
[2529413.007928] ? count_memcg_events.constprop.0+0x1a/0x30
[2529413.007937] ? srso_return_thunk+0x5/0x5f
[2529413.007944] ? handle_mm_fault+0xae/0x2d0
[2529413.007955] ? srso_return_thunk+0x5/0x5f
[2529413.007962] ? do_user_addr_fault+0x379/0x670
[2529413.007975] ? srso_return_thunk+0x5/0x5f
[2529413.007982] ? exc_page_fault+0x76/0x190
[2529413.007991] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2529413.008001] RIP: 0033:0x7f2cc6334cdb
[2529413.008008] RSP: 002b:00007ffc51e257a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[2529413.008017] RAX: ffffffffffffffda RBX: 0000000018cadd00 RCX: 00007f2cc6334cdb
[2529413.008022] RDX: 00007ffc51e25810 RSI: 0000000000005a03 RDI: 000000000000001a
[2529413.008027] RBP: 00007ffc51e28e00 R08: 0000000000000007 R09: 0000000000000000
[2529413.008032] R10: a302aa9724c8736d R11: 0000000000000246 R12: 00007ffc51e25810
[2529413.008037] R13: 0000000018cadd10 R14: 0000000000000000 R15: 0000000000000001
[2529413.008055] </TASK>
[2529413.008068] INFO: task middlewared (wo:3509128 blocked for more than 483 seconds.
[2529413.008716] Tainted: P OE 6.12.15-production+truenas #1
[2529413.009443] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2529413.010150] task:middlewared (wo state:D stack:0 pid:3509128 tgid:3509128 ppid:11181 flags:0x00000002
[2529413.010163] Call Trace:
[2529413.010167] <TASK>
[2529413.010175] __schedule+0x461/0xa10
[2529413.010192] schedule+0x27/0xd0
[2529413.010200] cv_wait_common+0xef/0x130 [spl]
[2529413.010239] ? __pfx_autoremove_wake_function+0x10/0x10
[2529413.010254] spa_lookup+0x3e/0xe0 [zfs]
[2529413.010820] spa_open_common+0x75/0x440 [zfs]
[2529413.011289] ? srso_return_thunk+0x5/0x5f
[2529413.011297] ? __lruvec_stat_mod_folio+0xa4/0xd0
[2529413.011309] spa_get_stats+0x4e/0x210 [zfs]
[2529413.011778] zfs_ioc_pool_stats+0x40/0x90 [zfs]
[2529413.012220] zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
[2529413.012663] zfsdev_ioctl+0x53/0xe0 [zfs]
[2529413.013076] __x64_sys_ioctl+0x94/0xd0
[2529413.013087] do_syscall_64+0x82/0x190
[2529413.013096] ? srso_return_thunk+0x5/0x5f
[2529413.013103] ? audit_reset_context+0x232/0x300
[2529413.013114] ? srso_return_thunk+0x5/0x5f
[2529413.013121] ? syscall_exit_to_user_mode_prepare+0x148/0x170
[2529413.013132] ? srso_return_thunk+0x5/0x5f
[2529413.013139] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529413.013147] ? srso_return_thunk+0x5/0x5f
[2529413.013154] ? do_syscall_64+0x8e/0x190
[2529413.013160] ? srso_return_thunk+0x5/0x5f
[2529413.013167] ? do_syscall_64+0x8e/0x190
[2529413.013173] ? srso_return_thunk+0x5/0x5f
[2529413.013180] ? srso_return_thunk+0x5/0x5f
[2529413.013186] ? cp_new_stat+0x134/0x170
[2529413.013197] ? srso_return_thunk+0x5/0x5f
[2529413.013203] ? audit_filter_rules.constprop.0+0x140/0x1120
[2529413.013217] ? srso_return_thunk+0x5/0x5f
[2529413.013224] ? __audit_filter_op+0xaf/0x110
[2529413.013232] ? srso_return_thunk+0x5/0x5f
[2529413.013242] ? srso_return_thunk+0x5/0x5f
[2529413.013248] ? audit_reset_context+0x232/0x300
[2529413.013257] ? srso_return_thunk+0x5/0x5f
[2529413.013264] ? syscall_exit_to_user_mode_prepare+0x148/0x170
[2529413.013273] ? srso_return_thunk+0x5/0x5f
[2529413.013280] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529413.013287] ? srso_return_thunk+0x5/0x5f
[2529413.013294] ? do_syscall_64+0x8e/0x190
[2529413.013300] ? srso_return_thunk+0x5/0x5f
[2529413.013307] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529413.013314] ? srso_return_thunk+0x5/0x5f
[2529413.013321] ? do_syscall_64+0x8e/0x190
[2529413.013328] ? srso_return_thunk+0x5/0x5f
[2529413.013335] ? do_syscall_64+0x8e/0x190
[2529413.013341] ? srso_return_thunk+0x5/0x5f
[2529413.013348] ? do_syscall_64+0x8e/0x190
[2529413.013354] ? exc_page_fault+0x76/0x190
[2529413.013362] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2529413.013370] RIP: 0033:0x7f30fda6fcdb
[2529413.013377] RSP: 002b:00007ffcbc041f10 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[2529413.013385] RAX: ffffffffffffffda RBX: 00000000407124d0 RCX: 00007f30fda6fcdb
[2529413.013390] RDX: 00007ffcbc041f90 RSI: 0000000000005a05 RDI: 000000000000001a
[2529413.013395] RBP: 00007ffcbc045580 R08: 00007f30fdb45430 R09: 00000000409516c0
[2529413.013400] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcbc041f90
[2529413.013404] R13: 00000000407124d0 R14: 000000004091d5b0 R15: 00007ffcbc045594
[2529413.013419] </TASK>
[2529533.831632] INFO: task middlewared (wo:3507597 blocked for more than 604 seconds.
[2529533.832353] Tainted: P OE 6.12.15-production+truenas #1
[2529533.832926] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2529533.833492] task:middlewared (wo state:D stack:0 pid:3507597 tgid:3507597 ppid:11215 flags:0x00004002
[2529533.833506] Call Trace:
[2529533.833511] <TASK>
[2529533.833523] __schedule+0x461/0xa10
[2529533.833546] schedule+0x27/0xd0
[2529533.833554] io_schedule+0x46/0x70
[2529533.833565] cv_wait_common+0xa9/0x130 [spl]
[2529533.833597] ? __pfx_autoremove_wake_function+0x10/0x10
[2529533.833613] txg_wait_synced_impl+0xc0/0x110 [zfs]
[2529533.834093] txg_wait_synced+0x10/0x40 [zfs]
[2529533.834571] spa_export_common+0x69c/0x6b0 [zfs]
[2529533.835038] zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
[2529533.835491] zfsdev_ioctl+0x53/0xe0 [zfs]
[2529533.835915] __x64_sys_ioctl+0x94/0xd0
[2529533.835928] do_syscall_64+0x82/0x190
[2529533.835942] ? srso_return_thunk+0x5/0x5f
[2529533.835951] ? __count_memcg_events+0x53/0xf0
[2529533.835962] ? srso_return_thunk+0x5/0x5f
[2529533.835969] ? count_memcg_events.constprop.0+0x1a/0x30
[2529533.835978] ? srso_return_thunk+0x5/0x5f
[2529533.835985] ? handle_mm_fault+0xae/0x2d0
[2529533.835996] ? srso_return_thunk+0x5/0x5f
[2529533.836003] ? do_user_addr_fault+0x379/0x670
[2529533.836016] ? srso_return_thunk+0x5/0x5f
[2529533.836023] ? exc_page_fault+0x76/0x190
[2529533.836033] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2529533.836043] RIP: 0033:0x7f2cc6334cdb
[2529533.836050] RSP: 002b:00007ffc51e257a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[2529533.836059] RAX: ffffffffffffffda RBX: 0000000018cadd00 RCX: 00007f2cc6334cdb
[2529533.836065] RDX: 00007ffc51e25810 RSI: 0000000000005a03 RDI: 000000000000001a
[2529533.836070] RBP: 00007ffc51e28e00 R08: 0000000000000007 R09: 0000000000000000
[2529533.836075] R10: a302aa9724c8736d R11: 0000000000000246 R12: 00007ffc51e25810
[2529533.836080] R13: 0000000018cadd10 R14: 0000000000000000 R15: 0000000000000001
[2529533.836096] </TASK>
[2529533.836108] INFO: task middlewared (wo:3509128 blocked for more than 604 seconds.
[2529533.836681] Tainted: P OE 6.12.15-production+truenas #1
[2529533.837258] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[2529533.837843] task:middlewared (wo state:D stack:0 pid:3509128 tgid:3509128 ppid:11181 flags:0x00000002
[2529533.837854] Call Trace:
[2529533.837858] <TASK>
[2529533.837864] __schedule+0x461/0xa10
[2529533.837879] schedule+0x27/0xd0
[2529533.837887] cv_wait_common+0xef/0x130 [spl]
[2529533.837916] ? __pfx_autoremove_wake_function+0x10/0x10
[2529533.837927] spa_lookup+0x3e/0xe0 [zfs]
[2529533.838403] spa_open_common+0x75/0x440 [zfs]
[2529533.838872] ? srso_return_thunk+0x5/0x5f
[2529533.838880] ? __lruvec_stat_mod_folio+0xa4/0xd0
[2529533.838893] spa_get_stats+0x4e/0x210 [zfs]
[2529533.839361] zfs_ioc_pool_stats+0x40/0x90 [zfs]
[2529533.839802] zfsdev_ioctl_common+0x5b9/0x6c0 [zfs]
[2529533.840245] zfsdev_ioctl+0x53/0xe0 [zfs]
[2529533.840669] __x64_sys_ioctl+0x94/0xd0
[2529533.840679] do_syscall_64+0x82/0x190
[2529533.840688] ? srso_return_thunk+0x5/0x5f
[2529533.840695] ? audit_reset_context+0x232/0x300
[2529533.840707] ? srso_return_thunk+0x5/0x5f
[2529533.840714] ? syscall_exit_to_user_mode_prepare+0x148/0x170
[2529533.840725] ? srso_return_thunk+0x5/0x5f
[2529533.840732] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529533.840740] ? srso_return_thunk+0x5/0x5f
[2529533.840747] ? do_syscall_64+0x8e/0x190
[2529533.840753] ? srso_return_thunk+0x5/0x5f
[2529533.840760] ? do_syscall_64+0x8e/0x190
[2529533.840766] ? srso_return_thunk+0x5/0x5f
[2529533.840773] ? srso_return_thunk+0x5/0x5f
[2529533.840780] ? cp_new_stat+0x134/0x170
[2529533.840790] ? srso_return_thunk+0x5/0x5f
[2529533.840797] ? audit_filter_rules.constprop.0+0x140/0x1120
[2529533.840811] ? srso_return_thunk+0x5/0x5f
[2529533.840818] ? __audit_filter_op+0xaf/0x110
[2529533.840826] ? srso_return_thunk+0x5/0x5f
[2529533.840836] ? srso_return_thunk+0x5/0x5f
[2529533.840842] ? audit_reset_context+0x232/0x300
[2529533.840851] ? srso_return_thunk+0x5/0x5f
[2529533.840858] ? syscall_exit_to_user_mode_prepare+0x148/0x170
[2529533.840867] ? srso_return_thunk+0x5/0x5f
[2529533.840874] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529533.840881] ? srso_return_thunk+0x5/0x5f
[2529533.840888] ? do_syscall_64+0x8e/0x190
[2529533.840894] ? srso_return_thunk+0x5/0x5f
[2529533.840901] ? syscall_exit_to_user_mode+0x10/0x1f0
[2529533.840908] ? srso_return_thunk+0x5/0x5f
[2529533.840915] ? do_syscall_64+0x8e/0x190
[2529533.840922] ? srso_return_thunk+0x5/0x5f
[2529533.840928] ? do_syscall_64+0x8e/0x190
[2529533.840934] ? srso_return_thunk+0x5/0x5f
[2529533.840941] ? do_syscall_64+0x8e/0x190
[2529533.840947] ? exc_page_fault+0x76/0x190
[2529533.840955] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[2529533.840963] RIP: 0033:0x7f30fda6fcdb
[2529533.840969] RSP: 002b:00007ffcbc041f10 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[2529533.840977] RAX: ffffffffffffffda RBX: 00000000407124d0 RCX: 00007f30fda6fcdb
[2529533.840982] RDX: 00007ffcbc041f90 RSI: 0000000000005a05 RDI: 000000000000001a
[2529533.840986] RBP: 00007ffcbc045580 R08: 00007f30fdb45430 R09: 00000000409516c0
[2529533.840991] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcbc041f90
[2529533.840996] R13: 00000000407124d0 R14: 000000004091d5b0 R15: 00007ffcbc045594
[2529533.841011] </TASK>
[2529533.841014] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings