The Datasets page in the WebUI takes minutes to load, then eventually loads or times out. Any API call that touches pool.dataset.* or pool.pool_normalize_info seems to just hang indefinitely.
This affects both the WebUI and direct midclt calls over SSH. I don’t seem to see any issues with SMB/NFS data access, just the webui and middlewared seems to be affected
- TrueNAS SCALE
25.04.2.6 - Kernel:
6.12.15-production+truenas - ZFS:
zfs-2.3.0-1 - Pool: 3-disk raidz1, ONLINE, no errors, last scrub clean
- Boot pool: single NVMe, ONLINE
- ~12 Apps running (sonarr, radarr, lidarr, komga, nextcloud, gitea, tailscale, etc.)
- ~480 snapshots total
- No replication tasks configured
- No SMART errors, no dmesg errors
!! Caveat !!
I’ve only really done some surface level investigation into the root core issue, since middlewared restart fixes the slowness. I’ve used an LLM to help me collect data that might assist with replicating & debugging this issue.
I’m happy to provide any help myself with debugging this issue, specifically getting any logs!
I promise you’re speaking with a real person here!!
Investigation notes:
midctl call core.get_jobs '[["state","=","RUNNING"]]'
shows alert.process_alerts stuck in RUNNING state for hours (longest observed: 5h 33m), but core.threads_stacks shows no active alert-source thread:
[
{
"id": 12611,
"method": "alert.process_alerts",
"arguments": [],
"transient": true,
"description": null,
"abortable": false,
"logs_path": null,
"logs_excerpt": null,
"progress": {
"percent": 0,
"description": "",
"extra": null
},
"result": null,
"result_encoding_error": null,
"error": null,
"exception": null,
"exc_info": null,
"state": "RUNNING",
"time_started": {
"$date": 1778524700000
},
"time_finished": null,
"credentials": null
}
]
[~] midclt call core.threads_stacks > /tmp/stacks.json 2>&1
[~] jq -r '
to_entries[]
| select((.value | join("")) | test("alert/source|plugins/alert\\.py"))
| "=== TID \(.key) ===", (.value | join(""))
' /tmp/stacks.json
# No Output?
The middlewared ProcessPoolExecutor worker child (a multiprocessing.spawn python3 process) is parked in cv_wait_common:
[~] ps --ppid <middlewared_pid> -o pid,stat,etime,cmd
1487 S 13-10:14:37 /usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(8)
1527 Sl 13-10:14:26 middlewared (zettarepl)
1550 S 13-10:14:23 /usr/bin/python3 -c from multiprocessing.spawn import spawn_main; ...
[~] cat /proc/1550/wchan
cv_wait_common
[~] cat /proc/1550/status | grep -E '^(State|Threads)'
State: S (sleeping)
Threads: 1
core.threads_stacks shows middlewared threads piling up behind the wedged worker:
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 80, in details
datasets = self.middleware.call_sync('pool.dataset.query', filters, options)
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset.py", line 230, in query
self.__transform(self.middleware.call_sync(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1034, in call_sync
return self.run_coroutine(self._call_worker(name, *prepared_call.args))
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1067, in run_coroutine
while not event.wait(1):
File "/usr/lib/python3.11/threading.py", line 629, in wait
signaled = self._cond.wait(timeout)
kill -KILL on the wedged worker leaves it as <defunct> (zombie) and middlewared’s ProcessPoolExecutor manager thread does not respawn it. That manager thread is parked in concurrent.futures.process:wait_result_broken_or_wakeup and apparently doesn’t notice the worker died
How I’m dealing with this
Running systemctl restart middlewared over ssh whenever I notice the webui being slow seems to resolve the issue for me almost every single time. I’m not sure if there’s a better way but I’d love to know!