High arc_prune CPU & ZFS cache memory pressure on SCALE 24.10.2.2

Running TrueNAS SCALE [24.10.2.2] with 32 GB RAM. I’m repeatedly hitting an issue where arc_prune consumes high CPU and available memory drops to ~1 GiB. I have limited the ZFE cache to different amounts, 0gb, 6gb, 8gb, 25gb but each time the ZFE cache will eventually pass whatever limit is set. This usually takes anywhere from 2 to 48 hours.

  • Pool: Storage_1 – 5× HDDs in RAIDZ2 (no special vdevs)
  • ARC cap: Set via startup script:echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max
  • VMs: One VM with 2 GiB RAM assigned (no ballooning)
  • Apps: Several Docker containers running (unchanged recently)

Any advice on how to prevent this from continuing to happen would be helpful. Thanks a lot

Yes - remove the tuneable and let ZFS / TrueNAS manage itself.

The days when you had to adjust ARC Max are long gone - TrueNAS does it for you.

The problem happened again after removing the tunable. ZFS cache has maxed out the memory and ARC_prune is using all the CPU trying to clear it but the memory isn’t being released and it is making the server sluggish.

The system can be fixed with a hard reboot but the will come back in a few hours. Is there any info that I can provided to help diagnose this issue?

Did you reboot after removing the tuneable?

Ya, I rebooted after removing the tuneable

I seem to have resolved the issue by deleting a new Cloudflared Docker container. I have a few others running without any problems, but this one appeared to be causing the memory issue. Since deleting it, I haven’t encountered the problem again.

1 Like