25.10 Web UI unresponsive after a day or so

TrueNAS General
1

Hey everyone. I recently upgraded to 25.10 (not realizing it was still “early adopter”), and I’ve been having a dog of a time with the web UI since. Everything will work fine after a reboot for a while, but then I eventually can’t hit the web UI.

It doesn’t seem like a networking issue (nothing has changed), and I can bring the web UI back with either a reboot or “service middlewared restart”, but surely this shouldn’t be happening with regularity, right? Any ideas?

I’m about to roll back the boot environment, but I had set up a couple containers after 25.10 (mainly PBS) and I gather those don’t carry back over?

Thanks.

2

I ran into a similar problem during 25.10 Beta, but it disappeared sometime later.

In my case the problem seemed to be nginx and a ‘systemctl restart nginx’ brought things back to life.

3

You might want to check the middlewared.log (/var/log/middlewared.log) for abnormalities. But from my personal experience it rather seems to be some regression in middlewared (aka asyncio_loop) itself, that leads to that process tree eating up more and more cpu resources. Using the GUI seem to contribute to this. Just search the forum for “asyncio” and you will probably find quite some people having the same problem. So for now restarting the middlewared service regularly is the only mitigation…

4

Hey everybody. 1st post here. AI helped me debug this issue and write the post, so keep that in mind.

================

Same issue on TrueNAS 25.10.2.1 (Goldeye) on a Beelink mini PC. I’ve hit this twice now and managed to catch it live the second time with some useful diagnostics, in case it helps track down the root cause.

Symptoms: Web UI becomes completely unreachable. SMB shares and running apps continue working normally. systemctl restart middlewared restores the UI immediately.

Diagnostics captured during the hang:

ps aux showed middlewared (PID 4060224) had been running since the previous day and accumulated 22+ minutes of CPU time — high for a daemon that should be mostly idle.

cat /proc/<pid>/wchan returned do_epoll_wait — the event loop itself was running, not stuck in a blocking syscall. This rules out a simple deadlock.

ls -la /proc/<pid>/fd | wc -l returned 639 open file descriptors.

ls -la /proc/<pid>/fd | grep socket | wc -l returned 559 sockets — against a healthy baseline of ~11 sockets on a fresh middlewared start. That’s a ~50x leak ratio.

ss -tp | grep <pid> showed all sockets were ESTAB connections to 127.0.0.1:11 (the internal middlewared IPC socket), all owned by asyncio_loop. No external connections.

Likely trigger:

The middlewared.log shows hundreds of occurrences of this error spread across weeks, repeating every time the UI is used:

[ERROR] middlewared.send():56 - Failed to JSON serialize server message: 
Object of type ValidationErrors is not JSON serializable
TypeError: Object of type ValidationErrors is not JSON serializable

The pattern suggests that each time this serialization error fires, the websocket handler fails to send the error response but the connection remains ESTAB from the OS perspective — leaking a socket. After enough UI interactions over ~24 hours, the accumulated sockets exhaust the event loop’s capacity and the UI becomes unresponsive.

Mitigation: A daily systemctl restart middlewared at 4am prevents the accumulation from reaching the breaking point. Not a fix, but it keeps things running.

Request: Could someone from iXsystems confirm whether the ValidationErrors is not JSON serializable bug in ws_handler/rpc.py is tracked and if there’s a fix planned? The error appears to originate at:

File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", 
line 54, in send
    data_ = json.dumps(data)
TypeError: Object of type ValidationErrors is not JSON serializable

The fix would be ensuring ValidationErrors is properly serialized (or caught and converted to a plain dict) before json.dumps is called in the send path.

Version: TrueNAS-25.10.2.1
Hardware: Beelink EQ12 (Intel N100, 16GB RAM, 4× NVMe in pool)

5

Use Report a Bug in the GUI or the top, upper right of the forum, Report a Bug link. Be sure to attached a debug dump. Just add in the info from your post. Devs see the JIRA tickets.

6

Confirming this is still present on 25.10.7 (Goldeye), five point releases on from the 25.10.2.1 above, with the identical signature:


[ERROR] middlewared.send():56 - Failed to JSON serialize server message:
  Object of type ValidationErrors is not JSON serializable
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 54, in send
      data_ = json.dumps(data)
```

**A reproducible trigger, in case it helps narrow it down.** Here the error fires on a filesystem ACL failure. An app install failed partway and left an orphaned dataset under `tank/apps`, whose parent is `0770 apps:apps`. Every subsequent ACL operation that walked the parent returned:

```
[EPERM] Filesystem permissions on path /mnt/tank/apps prevent access for group "root" to the path /mnt/tank/apps/<appname>
```

and each EPERM produced a `ValidationErrors` object that then failed to serialise. Deleting the orphaned dataset removed the recurring source.

**Socket counts as a second data point.** On a `middlewared` process up 9 days 21 hours:

```
total_fds = 265
sockets   = 60
```

Roughly 5x the ~11 baseline quoted above, but a long way short of the 559 that produced an unresponsive UI. My trigger was narrow - one bad path - rather than firing every time the UI is used, which would fit the leak accumulating in proportion to how often the error occurs.

**A different symptom from what looks like the same fault.** I do not get a UI hang. I get logged out, six times in one working day. The Web UI persists no auth token - `localStorage['ngx-webstorage|token']` holds the literal string `"null"` and no session cookie is set - and the audit log shows `auth.generate_token` firing every five minutes, then stopping, followed by `Logout - Credentials: Token`. So when the websocket breaks there is nothing to restore from and the session ends within five minutes.

**Being straight about what this does not explain:** 60 sockets should not be enough to exhaust anything, so the leak alone probably is not the cause of my logouts. Either it is contributory or there is a second mechanism. Flagging that rather than overstating the evidence.

Version: TrueNAS-25.10.7, Community Edition, MISSION_CRITICAL update profile, reports up to date.
Hardware: UGREEN NASync DXP4800 Pro, 24 GB RAM, 2x4TB mirror.

Noted as the OP did - AI assistance was used in the diagnosis and in drafting this post.
7

AI should have told you “Use Report a Bug in the GUI or the top, upper right of the forum, Report a Bug link. Be sure to attached a debug dump. Just add in the info from your post. Devs see the JIRA tickets.”