Snapshot Mount Fixes

kvedder · September 28, 2026, 3:48pm

Hello,

(Cant post Links yet)

We are dealing with an issue caused by Github OpenZFS Issue 18689.

It seems to have been fixed in Github openzfs/zfs/pull/17941.

Which the fix(es) may have been backported in Github truenas/zfs/pull/321 .

Can anyone confirm if these fixes made it into 25.10 or the upcoming version 26? It looks like they were tagged for 25.10.2.2.

Thanks,

Ken

kvedder · September 28, 2026, 4:02pm

github.com/openzfs/zfs

zfsctl_snapshot_unmount_delay: deadlock between zfs_snapshot_lock reader and snapentry_expire taskq

opened 03:31PM - 19 Jun 26 UTC

closed 04:40PM - 06 Aug 26 UTC

adamboutcher

Type: Defect

### System information Type | Version/Name --- | --- Distribution Name | Rocky… Linux Distribution Version | 10.2 Kernel Version | 6.12.0-211.22.1.el10_2.x86_64 #1 SMP PREEMPT_DYNAMIC Architecture | x86_64 OpenZFS Version | 2.2.10-1 ### Describe the problem you're observing Accessing snapshots via NFS (v4) using crossmnt hangs. Notably I am using Bacula to read the snapshot files over NFS. Trying to work the issue with Claude (gasp AI!) as the issue is now beyond my general Linux/ZFS knowledge, they seem to think it is related to commit https://github.com/openzfs/zfs/commit/fd7265c646f40e364396af5014bbb83e809e124a and `zfsctl_snapshot_unmount_delay` having `READER` but `zfsctl_destroy` having `WRITER`. Claude Log: ``` So the deadlock is: - Thread A (mountd/Bacula stat) holds zfs_snapshot_lock as READER, calls unmount_cancel, which calls taskq_cancel_id — this blocks waiting for the taskq item to complete - The taskq item (snapentry_expire) runs and tries to call zfsctl_destroy, which tries to take zfs_snapshot_lock as WRITER - Writer can't proceed while reader holds the lock, reader is blocked waiting for the taskq — deadlock ``` ### Describe how to reproduce the problem Mount a snapshot via NFS (exported with /pool ) eg `mount -t nfs server:/pool/.zfs/snapshot/mysnapshot /snapshot` run backup and it fails. I have set `zfs_expire_snapshot` to `7884000` so the snapshots dont try to umount during a backup. ### Include any warning/errors/backtraces from the system logs Kernel Log: ``` INFO: task rpc.mountd:3445 blocked for more than 122 seconds. [184197.084131] Tainted: P S OE ------ --- 6.12.0-211.22.1.el10_2.x86_64 #1 [184197.084138] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [184197.084144] task:rpc.mountd state:D stack:0 pid:3445 tgid:3445 ppid:1 task_flags:0x480140 flags:0x00000002 [184197.084147] Call Trace: [184197.084149] <TASK> [184197.084151] __schedule+0x2a5/0x630 [184197.084156] schedule+0x27/0x80 [184197.084158] schedule_preempt_disabled+0x15/0x30 [184197.084159] rwsem_down_write_slowpath+0x1d8/0x510 [184197.084163] ? avl_find+0x54/0x90 [zfs] [184197.084280] down_write+0x5a/0x60 [184197.084282] zfsctl_snapshot_unmount_cancel+0x1a/0x80 [zfs] [184197.084392] zfsctl_snapshot_unmount_delay+0x42/0xb0 [zfs] [184197.084491] zfs_getattr_fast+0x141/0x270 [zfs] [184197.084588] zpl_getattr+0x51/0xf0 [zfs] [184197.084666] vfs_getattr_nosec+0xbe/0x100 [184197.084669] vfs_statx+0x11a/0x160 [184197.084673] vfs_fstatat+0x6b/0xa0 [184197.084675] __do_sys_newfstatat+0x3c/0x80 [184197.084677] do_syscall_64+0x7d/0x160 [184197.084680] ? syscall_exit_to_user_mode+0x32/0x190 [184197.084681] ? do_syscall_64+0x89/0x160 [184197.084683] ? do_syscall_64+0x89/0x160 [184197.084684] ? do_syscall_64+0x89/0x160 [184197.084685] ? __memcg_slab_free_hook+0x100/0x150 [184197.084688] ? __x64_sys_close+0x3c/0x80 [184197.084690] ? kmem_cache_free+0x3ed/0x450 [184197.084692] ? syscall_exit_work+0xf3/0x120 [184197.084693] ? syscall_exit_to_user_mode+0x32/0x190 [184197.084695] ? do_syscall_64+0x89/0x160 [184197.084696] ? do_syscall_64+0x89/0x160 [184197.084697] ? do_syscall_64+0x89/0x160 [184197.084698] entry_SYSCALL_64_after_hwframe+0x76/0x7e [184197.084702] RIP: 0033:0x7f2810d07c4e [184197.084726] RSP: 002b:00007fff11308808 EFLAGS: 00000246 ORIG_RAX: 0000000000000106 [184197.084728] RAX: ffffffffffffffda RBX: 0000000000000015 RCX: 00007f2810d07c4e [184197.084729] RDX: 00007fff11308950 RSI: 000055e0a3e1ac34 RDI: 00000000ffffff9c [184197.084730] RBP: 00007fff11310990 R08: 0000000000000003 R09: 0000000000000000 [184197.084731] R10: 0000000000000000 R11: 0000000000000246 R12: 000055e0a3e1ac34 [184197.084732] R13: 000055e0a3e19988 R14: 000055e0a3e19970 R15: 00007fff113088d0 [184197.084733] </TASK> ``` I apologise if this is incorrect but I've hit a brick wall with this and a task that used to work on an old CentOS7 configuration.

github.com/openzfs/zfs

Fix snapshot automount expiry cancellation deadlock (#17941)

master ← ixhamza:SEE-365-snapshot-deadlock

opened 12:13PM - 17 Nov 25 UTC

ixhamza

+71 -48

### Motivation and Context This fixes a deadlock that occurs when snapshot expi…ry tasks are cancelled while locks are held. The deadlock causes the system to hang with multiple threads blocked indefinitely, requiring a system restart. The issue manifests under heavy snapshot automount load combined with memory pressure triggering ARC pruning. ### Description The deadlock occurs when the snapshot expiry task, ARC memory reclamation, and lock acquisition form a circular dependency. The sequence is: 1. `snapentry_expire` task spawns an umount process via `call_usermodehelper()` and waits for completion 2. Concurrently, memory pressure triggers `arc_prune` which acquires locks (z_teardown_lock) 3. `arc_prune` calls `zfs_exit_fs()` → `zfsctl_snapshot_unmount_delay()` to reschedule snapshot expiry 4. `zfsctl_snapshot_unmount_delay()` attempts to cancel the running expiry task with `taskq_cancel_id()` 5. Old `taskq_cancel_id()` blocks waiting for the task to complete (holds lock while waiting) 6. The umount process spawned in step 1 blocks trying to acquire locks held by `arc_prune` 7. Circular dependency: expiry task waits for umount → umount waits for arc_prune → arc_prune waits for expiry task The fix adds a boolean `wait` parameter to `taskq_cancel_id()`: - `wait=B_TRUE`: Block until task completes (default behavior for all callers) - `wait=B_FALSE`: Return EBUSY immediately if task is running (non-blocking) The `zfs_exit_fs()` path now uses non-blocking cancellation (`wait=B_FALSE`), breaking the deadlock by returning immediately when the expiry task is already running. Additional changes include removing the per-entry `se_taskqid_lock` (all taskqid operations now use global `zfs_snapshot_lock` as WRITER), and adding an `se_in_umount` flag to prevent recursive waits when `zfsctl_destroy()` is called during unmount. **Hung Task Stack Trace:** ``` [ 363.714403] INFO: task spl_delay_taskq:182 blocked for more than 120 seconds. [ 363.717021] Not tainted 6.12.43-production+ #658 [ 363.718977] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.723435] task:spl_delay_taskq state:D stack:0 pid:182 tgid:182 ppid:2 flags:0x00004000 [ 363.726758] Call Trace: [ 363.728106] <TASK> [ 363.728906] __schedule+0x40b/0x13b0 [ 363.730275] ? _raw_spin_unlock_irqrestore+0x27/0x50 [ 363.732264] ? _raw_spin_unlock_irqrestore+0x27/0x50 [ 363.734081] ? try_to_wake_up+0xec/0x760 [ 363.735543] schedule+0x2a/0x100 [ 363.737732] schedule_timeout+0x14c/0x160 [ 363.739305] ? _raw_spin_unlock+0x19/0x40 [ 363.740977] ? __queue_work.part.0+0xf1/0x3e0 [ 363.742622] ? preempt_count_add+0x7b/0xc0 [ 363.744391] wait_for_completion_state+0x135/0x1e0 [ 363.746124] call_usermodehelper_exec+0x174/0x1b0 [ 363.747942] call_usermodehelper+0x93/0xb0 [ 363.749654] zfsctl_snapshot_unmount+0xf3/0x240 [ 363.751478] snapentry_expire+0x7a/0x180 [ 363.753263] taskq_thread+0x284/0x5d0 [ 363.754641] ? __pfx_default_wake_function+0x10/0x10 [ 363.756703] ? __pfx_taskq_thread+0x10/0x10 [ 363.758220] kthread+0xf3/0x120 [ 363.759537] ? __pfx_kthread+0x10/0x10 [ 363.761427] ret_from_fork+0x3d/0x60 [ 363.762816] ? __pfx_kthread+0x10/0x10 [ 363.765424] ret_from_fork_asm+0x1a/0x30 [ 363.767505] </TASK> [ 363.768697] INFO: task arc_prune:189 blocked for more than 120 seconds. [ 363.771110] Not tainted 6.12.43-production+ #658 [ 363.773377] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.776205] task:arc_prune state:D stack:0 pid:189 tgid:189 ppid:2 flags:0x00004000 [ 363.780464] Call Trace: [ 363.781598] <TASK> [ 363.782449] __schedule+0x40b/0x13b0 [ 363.784040] ? __lruvec_stat_mod_folio+0xbd/0xe0 [ 363.785724] schedule+0x2a/0x100 [ 363.786978] schedule_preempt_disabled+0x18/0x30 [ 363.788985] rwsem_down_read_slowpath+0x24e/0x480 [ 363.790726] down_read+0x4b/0xc0 [ 363.792269] zfsctl_snapshot_unmount_delay+0x23/0xe0 [ 363.794082] zfs_exit_fs+0x85/0x90 [ 363.795393] zfs_exit+0x12/0x30 [ 363.798031] zfs_prune+0xb9/0x2d0 [ 363.799474] zpl_prune_sb+0x90/0xa0 [ 363.802208] ? __pfx_zpl_prune_sb+0x10/0x10 [ 363.803754] arc_prune_task+0x22/0x40 [ 363.806804] taskq_thread+0x284/0x5d0 [ 363.809803] ? __pfx_default_wake_function+0x10/0x10 [ 363.813805] ? __pfx_taskq_thread+0x10/0x10 [ 363.817002] kthread+0xf3/0x120 [ 363.818191] ? __pfx_kthread+0x10/0x10 [ 363.819600] ret_from_fork+0x3d/0x60 [ 363.821363] ? __pfx_kthread+0x10/0x10 [ 363.823222] ret_from_fork_asm+0x1a/0x30 [ 363.825182] </TASK> [ 363.880214] INFO: task spl_delay_taskq:4477 blocked for more than 120 seconds. [ 363.882827] Not tainted 6.12.43-production+ #658 [ 363.884875] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.887698] task:spl_delay_taskq state:D stack:0 pid:4477 tgid:4477 ppid:2 flags:0x00004000 [ 363.891378] Call Trace: [ 363.892475] <TASK> [ 363.893290] __schedule+0x40b/0x13b0 [ 363.894648] ? dequeue_entities+0x52c/0x6e0 [ 363.896465] ? psi_group_change+0x126/0x340 [ 363.897992] ? kvm_sched_clock_read+0x11/0x20 [ 363.899622] schedule+0x2a/0x100 [ 363.901044] schedule_preempt_disabled+0x18/0x30 [ 363.902777] rwsem_down_write_slowpath+0x239/0x5b0 [ 363.905349] ? __pv_queued_spin_lock_slowpath+0xa0/0x380 [ 363.907299] down_write+0x62/0x80 [ 363.908683] snapentry_expire+0x35/0x180 [ 363.910141] taskq_thread+0x284/0x5d0 [ 363.911515] ? __pfx_default_wake_function+0x10/0x10 [ 363.913767] ? __pfx_taskq_thread+0x10/0x10 [ 363.915376] kthread+0xf3/0x120 [ 363.916721] ? __pfx_kthread+0x10/0x10 [ 363.918094] ret_from_fork+0x3d/0x60 [ 363.919428] ? __pfx_kthread+0x10/0x10 [ 363.921025] ret_from_fork_asm+0x1a/0x30 [ 363.922524] </TASK> [ 363.923765] INFO: task spl_delay_taskq:4483 blocked for more than 121 seconds. [ 363.926382] Not tainted 6.12.43-production+ #658 [ 363.928677] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 363.931518] task:spl_delay_taskq state:D stack:0 pid:4483 tgid:4483 ppid:2 flags:0x00004000 [ 363.935072] Call Trace: [ 363.936243] <TASK> [ 363.937038] __schedule+0x40b/0x13b0 [ 363.938397] ? _raw_spin_unlock_irqrestore+0x27/0x50 [ 363.940394] ? dl_server_stop+0x2f/0x40 [ 363.941788] schedule+0x2a/0x100 [ 363.943024] schedule_preempt_disabled+0x18/0x30 [ 363.945136] rwsem_down_read_slowpath+0x24e/0x480 [ 363.946850] down_read+0x4b/0xc0 [ 363.948342] zfsctl_snapshot_unmount+0x71/0x240 [ 363.949988] snapentry_expire+0x7a/0x180 [ 363.951465] taskq_thread+0x284/0x5d0 [ 363.953075] ? __pfx_default_wake_function+0x10/0x10 [ 363.954900] ? __pfx_taskq_thread+0x10/0x10 [ 363.957267] kthread+0xf3/0x120 [ 363.958974] ? __pfx_kthread+0x10/0x10 [ 363.961105] ret_from_fork+0x3d/0x60 [ 363.963014] ? __pfx_kthread+0x10/0x10 [ 363.965210] ret_from_fork_asm+0x1a/0x30 [ 363.966702] </TASK> ``` ### How Has This Been Tested? **Reproduction script:** ``` zpool create -f testpool mirror /dev/sdc /dev/sdd -O mountpoint=none mkdir -p /run/testfs zfs create -o mountpoint=/run/testfs -o snapdir=visible testpool/testfs echo 1 > /sys/module/zfs/parameters/zfs_expire_snapshot echo 524288 > /sys/module/zfs/parameters/zfs_arc_dnode_limit for i in {1..1000}; do zfs snapshot testpool/testfs@snap$i; done export SLEEP_AMOUNT=1 for group in {0..9}; do for proc in {0..9}; do start=$((group * 100 + proc * 10 + 1)) end=$((start + 9)) bash -c "for attempt in {1..43200}; do for i in {$start..$end}; do ls /run/testfs/.zfs/snapshot/snapi/ >/dev/null 2>&1 & done; sleep SLEEP_AMOUNT; echo attempt; done" & done sleep 1 done ``` **Results:** - **Without fix:** Deadlock occurs within ~5 minutes with reproduction script, system hangs with hung task warnings - **With fix:** No deadlock after 24+ hours of continuous testing with reproduction script - Verified task cancellation works correctly with both blocking and non-blocking modes ### Types of changes - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Performance enhancement (non-breaking change which improves efficiency) - [ ] Code cleanup (non-breaking change which makes code smaller or more readable) - [ ] Quality assurance (non-breaking change which makes the code more robust against bugs) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Library ABI change (libzfs, libzfs\_core, libnvpair, libuutil and libzfsbootenv) - [ ] Documentation (a change to man pages or other documentation) ### Checklist: - [x] My code follows the OpenZFS [code style requirements](https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md#coding-conventions). - [ ] I have updated the documentation accordingly. - [x] I have read the [**contributing** document](https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md). - [ ] I have added [tests](https://github.com/openzfs/zfs/tree/master/tests) to cover my changes. - [ ] I have run the ZFS Test Suite with this change applied. - [x] All commit messages are properly formatted and contain [`Signed-off-by`](https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md#signed-off-by).

github.com/truenas/zfs

NAS-138720 / 26.04 / Backport Snapshot Automount fixes for SEE-365 (#321)

truenas/zfs-2.4-release ← NAS-138720

opened 12:26PM - 09 Dec 25 UTC

ixhamza

+197 -93

### Motivation and Context Backport snapshot automount fixes for https://ixsyst…ems.atlassian.net/browse/SEE-365. ### Description Backport following upstream PRs: - https://github.com/openzfs/zfs/pull/17941 - https://github.com/openzfs/zfs/pull/17942 - https://github.com/openzfs/zfs/pull/17943 ### How Has This Been Tested? - CI Testing - [Scale Build](http://jenkins.eng.ixsystems.net:8080/job/master/job/custom/1752/) ### Types of changes - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Performance enhancement (non-breaking change which improves efficiency) - [ ] Code cleanup (non-breaking change which makes code smaller or more readable) - [ ] Quality assurance (non-breaking change which makes the code more robust against bugs) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Library ABI change (libzfs, libzfs\_core, libnvpair, libuutil and libzfsbootenv) - [ ] Documentation (a change to man pages or other documentation) ### Checklist: - [x] My code follows the OpenZFS [code style requirements](https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md#coding-conventions). - [ ] I have updated the documentation accordingly. - [x] I have read the [**contributing** document](https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md). - [ ] I have added [tests](https://github.com/openzfs/zfs/tree/master/tests) to cover my changes. - [ ] I have run the ZFS Test Suite with this change applied. - [x] All commit messages are properly formatted and contain [`Signed-off-by`](https://github.com/openzfs/zfs/blob/master/.github/CONTRIBUTING.md#signed-off-by).

nite_route · September 28, 2026, 6:16pm

yeah those SEE-365 / openzfs#18689 snapshot automount deadlock fixes got backported on the truenas/zfs side

-25.10.2.2: truenas/zfs#373 (NAS-138720)
-26.04 line: truenas/zfs#321

so 25.10.0 / 25.10.1 wont have it yet. need 25.10.2.2+ (or a 26 build that pulled the same commits). check System → Update or midclt call system.version

Topic		Replies	Views
"Failed to load datasets" on Datasets page TrueNAS General SCALE , ZFS	11	173	October 16, 2025
Performance Gains for 24.04.0 and 13.3 TrueNAS General Performance , SMB , NFS , iSCSI , ZFS	10	812	April 25, 2024
Time Machine no longer taking snaphots after Fangtooth upgrade TrueNAS General SCALE , macOS-Client , Fangtooth	25	492	November 13, 2025
Zfs invalid name TrueNAS General	19	467	October 25, 2025
25.10.0 snapshot regex error TrueNAS General SCALE	5	112	August 28, 2026

Snapshot Mount Fixes

Related topics