Funny, I took another attempt at fixing this last night (probably like the 5th time now). I am not sure if I fixed it but what I saw in the logs was a job trying to check over and over for images that didn’t exist. I tracked it down to images that didn’t have a tag at all!
When you replace an image with a new digest but same tag (like latest) the old image ends up having a tag like <none>. Each time you upgrade your stack and replace the existing it leaves a dangling image with a tag of <none>
My hunch is that the update checker is still checking these images and gets into a loop. I had about 20 images with tags of <none> and I think it was looping trying to check them over and over again.
What I ended up doing was pruning all dangling images by running:
docker image prune -a -f
then restarting the middleware service
systemctl restart middlewared
I ended up creating a cron job to do this too.
The jury is still out on whether it fixed it or not but it’s been over 24 hours and asyncio_loop isn’t using 110% CPU anymore (20-40% which I still think it quite high).
I think you might be on to something. I also had a tone of old images that were tagged with
Wound up freeing about 100gb of old images. I’ve restarted middlewared afterwards and we’ll see what happens. I’ll also note that middlewared restart MUCH faster after pruning those old images.
Just tried it out for myself, I think you got it!
When I access the webui asyncio_loop starts using the cpu, which is expected, problem until now was after you exited the webui asyncio_loop kept using cpu and gradually increased over time unless you restarted middlewared.
I’m no longer seeing any activity in the asyncio_loop after I exit the webui, which was only possible with a middlewared restart before technotim’s fix.
I’ll keep an I on it and report back if things change.
there is definitely some errant processes or sticky leftovers floating around once docker is enabled.
just about any login / logout leave idle cpu usage higher so I am essentially having to prune and restart middleware after any login… wether I mess with docker or not… just seems to be the only way to quiet it down for a bit
as an aside… netdata seems to be pretty resource hungry compared to other netdata instances I have run. .to be fair ix have enabled quite a bit… but still just having netdata running with no web sessions doubles the idle cpu on a quad core… seem egregious
Just bumping this thread again to say it’s still happening even after 25.04.2.6. super frustrating. I’m going to set up a cron job to run ever 12 hours or something like that.
just wanted to say, I have the same problem basically from the beginning since I use truenas (~3 months, since 24.10.2.4). I’m using some custom docker apps via yaml, and some communtiy apps. It’s not as extreme as in your case (CPU is at ~60%). Restarting middleware fixed it temporarly. I just tried to delete all unused docker images, will see how this continues.
Yup, super frustrating. Almost find it hard to believe the problem isn’t widespread, or people just don’t notice. Also, crickets from the Truenas guy awalkerix who posted earlier in the thread, I sent him what he wanted months ago and no feedback. Oh well.
I am running 25.04.2.4, and asyncio_loop was at the top of “Top” and not moving, and all my cron jobs ( I have nine cron jobs configured) were not getting triggered, also the GUI was frozen so had to ssh to killl the process to make it all come back again.
For me, the system load is not so severe; it only slowly becomes a issue after several days.
A few days ago, I had an idea and shifted my tests to two things:
I haven’t login to the webui of truenas for several days after restarting middlewared via CLI. After 5 days the systemload was still as low as right after restart. So maybe it has something to do with the webui?
I switched of “Check for docker image updates” under Apps → Configuration → Settings.
No outcome yet, as I have just switched it off.
So 7 days after the restart of ”middlewared”, ”asyncio_loop” has now started to be listed at the top of Top as shown below.
GUI, and cron jobs still not affected, but I guess once it starts hitting the 100% cpu mark, the problem will return.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
Outside of us hacking away and guessing, I think the solution has to come from the TrueNAS team. This process uses way more resources than I ever imagined, this just shined a light on it.
First time looking at this thread so do not have much background.
From my machine, asyncio is sitting using about 30-40% CPU.
I have profiled this and so far my best guess is that it is hammering the docker socket to fetch container stats. I had not actually noticed this myself, but I do not run a massive number of containers.
@technotim out of curiosity, how many docker containers are you running? I am wondering if the people most impacted by this are running a large number of apps/containers.
