Services eating up a lot of ram, even after I turn them all off??

TrueNAS General
1

Hi everyone,

Have been running truenas scale for some months now, always no issue but now I was getting performance issues when accessing data, so had a look at the memory dashboard and to my surprise, services was eating up 50 Gb of ram.

memory

So I’m running:

  • One VM running nixos, assigned statically 16gb of ram (in the screenshot above, I had just turned that VM off to see if that was the issue)

  • SMB & NFS, usually with no load except in the evenings when I’m streaming video - I turned that off as well to see if it would bring memory usage down, but no effect

  • 1 application: tailscale. Turned that off and also unset the application pool to turn off application service but no change.

Went into htop, I see a lot of middlewared processes

middlewared
middlewared1303×926 183 KB

Tried service middlewared restart, even just stopped it, but no change in memory.

Anything else I can try? After a reboot, the services ram drops to normal usage again, but it creeps up over a few days until it reaches around 58-59 gbs, only leaving 1-2 gb of ram for my ARC cache…

2

That htop doesn’t show real resource usage, can we get a picture of top -em -oRES to see what is really consuming the memory?

3

Ofcourse, thanks already for the quick reply on this one.

Output of top -em -oRES

image
image625×1024 28.6 KB

4

@kris Here is a new screenshot of htop with userland processes hidden. I had to restart truenas since it force quit my VM because it was running out of memory… In this screenshot, truenas has been up for about 2 days, services ram has expanded in that time from 17gb to 25gb. Running 1 vm with 16 gb, 1 app (tailscale) and nfs/smb/ssh services.

image
image1717×1247 289 KB