25.10 Web UI unresponsive after a day or so

TrueNAS General
1

Hey everyone. I recently upgraded to 25.10 (not realizing it was still “early adopter”), and I’ve been having a dog of a time with the web UI since. Everything will work fine after a reboot for a while, but then I eventually can’t hit the web UI.

It doesn’t seem like a networking issue (nothing has changed), and I can bring the web UI back with either a reboot or “service middlewared restart”, but surely this shouldn’t be happening with regularity, right? Any ideas?

I’m about to roll back the boot environment, but I had set up a couple containers after 25.10 (mainly PBS) and I gather those don’t carry back over?

Thanks.

2

I ran into a similar problem during 25.10 Beta, but it disappeared sometime later.

In my case the problem seemed to be nginx and a ‘systemctl restart nginx’ brought things back to life.

3

You might want to check the middlewared.log (/var/log/middlewared.log) for abnormalities. But from my personal experience it rather seems to be some regression in middlewared (aka asyncio_loop) itself, that leads to that process tree eating up more and more cpu resources. Using the GUI seem to contribute to this. Just search the forum for “asyncio” and you will probably find quite some people having the same problem. So for now restarting the middlewared service regularly is the only mitigation…

4

Hey everybody. 1st post here. AI helped me debug this issue and write the post, so keep that in mind.

================

Same issue on TrueNAS 25.10.2.1 (Goldeye) on a Beelink mini PC. I’ve hit this twice now and managed to catch it live the second time with some useful diagnostics, in case it helps track down the root cause.

Symptoms: Web UI becomes completely unreachable. SMB shares and running apps continue working normally. systemctl restart middlewared restores the UI immediately.

Diagnostics captured during the hang:

ps aux showed middlewared (PID 4060224) had been running since the previous day and accumulated 22+ minutes of CPU time — high for a daemon that should be mostly idle.

cat /proc/<pid>/wchan returned do_epoll_wait — the event loop itself was running, not stuck in a blocking syscall. This rules out a simple deadlock.

ls -la /proc/<pid>/fd | wc -l returned 639 open file descriptors.

ls -la /proc/<pid>/fd | grep socket | wc -l returned 559 sockets — against a healthy baseline of ~11 sockets on a fresh middlewared start. That’s a ~50x leak ratio.

ss -tp | grep <pid> showed all sockets were ESTAB connections to 127.0.0.1:11 (the internal middlewared IPC socket), all owned by asyncio_loop. No external connections.

Likely trigger:

The middlewared.log shows hundreds of occurrences of this error spread across weeks, repeating every time the UI is used:

[ERROR] middlewared.send():56 - Failed to JSON serialize server message: 
Object of type ValidationErrors is not JSON serializable
TypeError: Object of type ValidationErrors is not JSON serializable

The pattern suggests that each time this serialization error fires, the websocket handler fails to send the error response but the connection remains ESTAB from the OS perspective — leaking a socket. After enough UI interactions over ~24 hours, the accumulated sockets exhaust the event loop’s capacity and the UI becomes unresponsive.

Mitigation: A daily systemctl restart middlewared at 4am prevents the accumulation from reaching the breaking point. Not a fix, but it keeps things running.

Request: Could someone from iXsystems confirm whether the ValidationErrors is not JSON serializable bug in ws_handler/rpc.py is tracked and if there’s a fix planned? The error appears to originate at:

File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", 
line 54, in send
    data_ = json.dumps(data)
TypeError: Object of type ValidationErrors is not JSON serializable

The fix would be ensuring ValidationErrors is properly serialized (or caught and converted to a plain dict) before json.dumps is called in the send path.

Version: TrueNAS-25.10.2.1
Hardware: Beelink EQ12 (Intel N100, 16GB RAM, 4× NVMe in pool)

5

Use Report a Bug in the GUI or the top, upper right of the forum, Report a Bug link. Be sure to attached a debug dump. Just add in the info from your post. Devs see the JIRA tickets.