Increasing CPU Usage of middlewared Service on 25.10

TrueNAS General
1

Hi there,

did by chance anyone notice the CPU usage of the middlewared service increasing by quite a margin (x2/x3) after you have used the GUI for a while ? I mean usually it sits around 1% and when you are navigating the GUI it of course goes up. But even after closing the GUI i had the middlewared service suddenly hogging 7-8% CPU constantly. And i am pretty sure none of the periodic tasks were running at the time:

[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task auditbackend._AuditBackendService__lifecycle_cleanup to run every 86400 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task auth.twofactor.remove_expired_secrets to run every 86400 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task docker.state.periodic_check to run every 86400 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task mail.send_mail_queue to run every 600 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task pool.dataset.sync_db_keys to run every 86400 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task rate.limit.cache_clear to run every 600 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task service.check_deprecated_services to run every 3600 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task truecommand.health_check to run every 1800 seconds
[2025/11/02 17:11:18] (DEBUG) middlewared._setup_periodic_tasks():428 - Setting up periodic task vmware.delete_pending_snapshots to run every 10800.0 seconds

After a restart of the service CPU usage returned back to normal.

Best regards,

Andreas

1 Like
2

This is how middlewared behaves after 2 days - no workers doing anything, not logged into GUI:

grafik
grafik718×198 10.4 KB

and this is how it behaves 10 minutes after a restart of the service:

grafik
grafik630×197 7.97 KB

You rarely see any cpu usage at all. Maybe i’m just interpreting things wrong, but it looks to me like some regression in the service…

3

I have the same problems. And I’ve noticed that it has something to do with Virtual Machines. Running HAOS from a VM increases the CPU load by 2-3 times. And if I use a TrueNAS GUI, it grows even further. If I run HAOS from Incus, then are no problems. Unfortunately, at 25.10 the Incus VM autostart was disabled and you have to do it manually. Why developers broke perfectly fine Incus VM’s?

4

Hmm i actually doubt that - cause vm are running under qemu and i never had any incus based VMs. But yes i agree using the GUI makes things worse and somehow increases the CPU usage of the middlewared service over time even if the GUI isn’t actively being used. I really wonder if the midclt calls i have scheduled contribute to this. Although it’s nothing really fancy: just find a vm by name, get id, current status and autostart option and start it if autostart is on and status is stopped. This is on a 15 min schedule - maybe i try turning that scheduled tasks off and see if it makes a difference…