Anyone else seeing this?
I have a middlewared process which, whenever I enter a single keystroke into cli_console or its minions, spikes to over 100% CPU.
The echo of the keystroke can take 3-10 seconds.
It is totally unusable.
Reboot does not clear it.
Anyone know a good way to inspect these running processes to see what they are doing?
I recorded a screen cap to illustrate this, but can’t attach it here.
I looked at /var/log/auth.log, and observed that if I run
sudo cli_console and select 7, then wait for the truenas> prompt,
each time I hit ls\n I get a slew of
Jun 9 11:16:14 truenas.zepherin.com ed[2490]: pam_unix(middleware-unix:session): session closed for user root
Jun 9 11:16:14 truenas.zepherin.com ed[2490]: pam_unix(middleware-unix:session): session opened for user root(uid=0) by (ui
d=0)
Jun 9 11:16:14 truenas.zepherin.com ed[2490]: pam_unix(middleware-unix:session): session closed for user root
Jun 9 11:16:14 truenas.zepherin.com ed[2490]: pam_unix(middleware-unix:session): session opened for user root(uid=0) by (ui
d=0)
This corresponds to a spike in the middlewared process, and it takes quite a while for the ls to be echoed.
I found perf top and ran it against the middlewared process. Not really useful. I guess noone else is finding this extremely sluggish?
Today, I was collecting a debug log (System Settings → Advanced → Download Debug) and attempted to log into another GUI instance. Totally locked out. It never came up to present the login dialog until after the debug log file generation process completed.
If you look at the process tree for middlewared on my system (ps -eLF | grep middlew | grep -v grep | wc), there are 119 threads. This is a massively parallel process, which seems to be seriously choked. Time to re-structure and re-write in rust?
If I compare my vestigial proxmox logins, I can have as many as I like, and each WEB session is independent, and doing stuff in one does not lock out or slow down others.