ZFS cache preventing app from getting allocated memory it needs

Apps and Virtualization
,
1

I’m running Scale v25.04.2.5with 94 GiB RAM.

I executed a query in Open WebUI and received error message: 500: model requires more system memory (4.9 GiB) than is available (4.6 GiB)

I logged into the TrueNAS web UI and saw memory usage:

  • Free: 4.6 GiB
  • ZFS Cache: 70.0 GiB
  • Services: 19.4 GiB

Open WebUI is configured to use up to 40 GiB, but was only using 930 MiB.

Questions

  1. What’s going on here? I’ve allowed Open WebUI to use plenty more RAM than it needed, and the system clearly had it with 70.0 GiB in cache.
  2. My understanding was that TrueNAS would free up RAM from the cache for apps to use if requested. Is this not the case?
  3. What can I do to ensure my apps get the memory they need from cache if there isn’t enough free RAM?
2

What’s going on is that you’re witnessing how Linux + ZFS is superior to FreeBSD’s terrible and outdated memory management. :flexed_biceps:

“Try adding a tunable bro. Restrict how much memory the ARC is allowed.”

This is the way. :tada:

3

Change arc size-----------------------
echo #number in bytes here#>> /sys/module/zfs/parameters/zfs_arc_max

You’ll have to basically put in that command after every boot and each time you stat/stop a VM. (Edit: Might be easier to simply have a cron every hour or something)

@winnielinnie’s memeing is honestly pretty true - there is something that feels like linux doesn’t free up arc memory as quickly as freebsd did. Am I smart enough to prove or fix this? No, so I just work around it. Should it behave this way? Likely not.

4

The same thing happens in Solaris 10 & 11. If the application checks for currently free memory, it gets what is free. If it attempted to simply allocate the memory and check the return code for success, it would succeed. ZFS would release enough ARC entries, in the proper order, (like any good cache), to not only allow the application it’s requested memory. But, also to keep the minimum free memory.

Now I do agree that FreeBSD’s memory management of ZFS ARC memory is superior to Linux, unfortunately people think Linux is the way to go…

1 Like