Asyncio_loop eating up all RAM

Hey!

My setup: 32GB Ram, Ryzen 5 2600, 5 HDD drives, 1 SSD drive 1 NVME drive, TrueNAS 24.04.2. I’m not listing specifics unless I’m required to. This shouldn’t be any hardware error. Everything was working fine for the past few weeks.

So the current issue, asyncio_loop is eating up my RAM.
image
It’s taking up 25GB of my 32GB system. I can’t run VMs since they crash when the RAM fills back up again. This issue appeared randomly. There wasn’t any action taking place, no restarts, nothing. The strange thing is, it never fills up ALL of the RAM. Other services like immich and zerotier can work fine.

What happens: After a restart, the RAM usage stays at cool 2-3GB. After about 1-2 minutes, it starts filling up by 200-400MB/s. This goes on till it fills almost the whole ram. The system never goes unresponsive in this time. I can notice one cpu core being absolutely pinned in this timeframe. I can manually restart middlewared through shell and it will be the same situation. Free ram untill it’s eaten up.

I’ve tested apps, it’s not them. Unsetting the pool doesn’t do anything. Switching to another drive as the pool doesn’t do anything. Switching off SMB doesn’t do anything. I submitted a bug report over on Jira with a lot of screenshots, I’ll drop them down here:



image

I could reinstall the system, though I’d like to avoid that. I installed the Truenas system as a partition on the NVME drive. If I nuke that, it will also nuke the VMs on the drive and apps.

1 Like

Bug report likely a good call. I would agree likely not a hardware issue, but you never know. Looks more like a continuing memory leak, you are not the first one. The fix is mostly likely with IX.