I have a Truenas SCALE server that I use for both data storage and to run a few important VMs, including the OPNsense router for my house and a Nextcloud server. Most of the time, both the SCALE server and the VMs are nearly idle, and CPU utilization on the SCALE dashboard shows at around 1-3%, with the hottest thread somewhere around 32-38 Celsius.
However, a couple of times recently the SCALE server has shown CPU utilization around 15-20% and the hottest thread at around 53-59 Celsius even though nothing much appears to be running. When this happens, shutting down all the VMs doesn’t make much difference, with CPU utilization remaining over 15% and the hottest thread over 50 Celsius.
Obviously, when this happens, SCALE is running some phantom processes. Rebooting the SCALE server always solves this problem and stops the phantom processes from running, with CPU utilization and hottest thread after reboots down where they should be–like 1-4% for CPU utilization and around 35 Celsius for the hottest thread. However, obviously rebooting isn’t a good solution since it takes down internet in my house and is overall disruptive.
So, what I’m wondering is does someone know how I can identify the phantom processes that, at times, appear to be running on SCALE and shouldn’t be, and that are consuming CPU cycles and watts? Or at least is someone able to point me in the right direction?
In case anyone finds it relevant, the CPU is a Ryzen 7 Pro 5750G, with an ASRock Rack x570d4u-2L2T motherboard, and 128gb ECC RAM.
Well, start with htop or whatever your favorite derivative of top is.
htop is installed by default, so you can use that. If you’re old school, you can just use top and hit P after it launches to sort by CPU usage.
In addition to the VM’s, do you also happen to run any Apps? I find the k3s process sometimes would seem to consume more CPU for no real reason, which is why I don’t plan on ever using it, among other reasons like TrueCharts stuff breaking frequently on updates.
1 Like
This was extremely helpful. At least I’ve diagnosed the problem now , but I don’t know what to do about it.
When I ran htop, it showed 100% CPU utilization by “smbd: client” and listed a specific IP address that’s a Linux VM on SCALE. I was transferring some large files on that client earlier via a Samba share on SCALE, so apparently the smbd process got stuck and was still running.
However, even after I shut down that client, the CPU utilization by smbd still showed as 100%. Also, even turning off the SMB service didn’t help. Finally, I rebooted SCALE, and the shutdown process got stuck on a screen for a few minutes waiting to shut down the smbd process, but it did finally shut down, and after I rebooted, everything was solved with CPU utilization back to normal and Samba running as normal.
I’m pretty sure this same thing happened last week after I transferred some large files over Samba on that client, although I didn’t know about htop at the time so can’t be sure that it was the same cause.
Does anyone have suggestions? Otherwise, it seems that all I can do is wait and watch.
@awalkerix, does this ring a bell?
Depends on context, but an smbd process at 100% CPU generally means a client is doing a lot of ops.
