Need help with ZFS Cache, it is crashing my TrueNAS

Hello All,
I recently built my new TrueNAS system and noticed that ZFS Cache grows around certain time, around 2am, then eventually crashes the system.
I did memtest+ and it passed. No hard drives reported any error. But when I am awake the next morning I notice that my TrueNAS is powered on but unresponsive.
Does anybody experiencing similar issue?

I am using
Ryzen 7 9700X
ASRock B650 Steel Legend ver 3.25
NEMIX RAM 32GB (2X16GB) DDR5 5600MHZ PC5-44800 1Rx8 1.1V CL46 288-PIN ECC Unbuffered

More details on your problem are necessary. What version of Scale? Are you running apps or VMs? What processes are going on? Do you have backups or something else running?

1 Like

I am running latest version 25.04.1. i am running 4 apps only atm. Dockge, qbittorrent, plex and wg-easy. no backups are running only scheduled scrubs and enabled trim on both storage.
Reason why I am suspecting zfs cache is that whenever system decide to grow zfs cache, I see zfs cache fluctuate then system freezes.

I am trying to upload screenshot but it is keep giving me error on this post that I cannot upload embeded media. so annoying

The following is a workaround, not a fix:

echo ##setlimitinbytes>> /sys/module/zfs/parameters/zfs_arc_max

This’ll put a cap on arc size (until system reboot; also starting/stopping a VM seems to reset back to defaults).

If I had to guess, one of your apps is taking more memory suddently, and arc on linux isn’t as graceful as the truenascore days - the slight delay for it to free-up causes oom.

I think I saw a very similar post to yours in the past & it was something to do with qbittorent… can’t remember the details though.

Edit: a quick search seems to imply it had something to do with older versions of TrueNAS scale - what version are you running? More details, more betterer.

1 Like

Thank you very much for your respond.
I do not have any VM and I only run those four apps I posted before.

Now that you mentioned, yes it does freeze often when I download something with qbittorrent. I thought it was NIC, so I got intel I226 card and capped download speed but ultimately it did not fix the issue.

I am going to stop qbittorrent while I am not use then see how stable my system will be.

As I stated I am using version 25.04.1.

Please let me know any information you would like to see.

I ran Scale virtualized for a while, and made the mistake of using ZFS on Proxmox above it. I cranked on some kernel tuning and was trying to push speed limits on a 10g network, and maximize caching in the ARC. Every few days it would start thrashing and Proxmox would be digging into the swap space, hard. It was resource starved. That starvation went downstairs to Truenas and would make it unusable over time. Finally checked ram usage (64gb in the server, 32 for prox and 32 for Scale) and saw it was all used up. Undid all my changes, choked zfs to the 50% of ram limit it started with, and everything behaved again.

You’re not running this under proxmox by any chance?

On my old non-truenas nas I had a similar issue with transmission (running as a docker container). It would eat almost all the RAM eventually, and other docker apps crashed with OOM.

What is more interesting is that portainer’s memory usage reported all this eaten memory as a transmission container’s cache. So, docker is/was stupid to the point it would crash the 1.5GB “fair and square” app instead of purging 8+GB of transmission cache. I have solved it by setting a memory limit for transmission. [1]

AIUI, truenas’s apps use docker under the hood. So, it’s possible that you have the same issue.


  1. IMO, this was a half-assed solution because of no disk cache inside a transmission. Perhaps it was mitigated at the OS level. ↩︎

2 Likes

Ah sorry, missed that. I think the version issue I was thinking of was from like Spring last year; I don’t think it is the version you’re using.

I think either playing with the ARC limit command I gave or the settings in torrenting app should help. I sadly don’t have specific steps on the app.

The ARC cache growing is normal and not necessarily related to your crashes, as others have said, the system views free RAM as wasted ram and will use as much as it thinks it can get away with. It could be an OOM event, but as far as I can see you haven’t determined that conclusively.

I would expect there to be a log trail if oomkiller is getting activated, have you see any mention there? There could also be mentions of crash related events even if they aren’t necessarily oomkiller-related, so the logs are highly relevant.

Don’t mind me. Just enjoying some popcorn. :popcorn:


Does the qBittorrent app use libtorrent version 1.2 or 2.0? You can check in its About menu.

1 Like

With extra industrial grade butter. For me it was an experiment and I learned from it on a secondary box nobody relies on. But watching connectivity to something you’re working on, gradually decay to where you can’t even ping the hypervisor is disturbing.

Thank you for all the responses so far. Sorry for late respond as I was traveling.
Long story short, stopping qbittorrent did not help.
Truenas, ZFS again grew rapidly around 2am and eventually crashed the system.

I really do not understand as I don’t have anything schedule task. This Truenas setup is nearly default as it gets.

These are all the apps I am running.

Let’s take a closer look at the system itself then. I’ll pretend like I’m new to the thread. Is this running under a hypervisor or is it bare metal? Have you changed any zfs ARC settings/tunables? Does this machine ever start eating swap space or never touches it? I’d guess that leading up to an OOM state it would start swapping to stay alive. There are limits you can place on ARC if that’s the problem.

  1. “This Truenas setup is nearly default as it gets.”
  2. Posting a screenshot with a handful of apps.
  3. Me with zero apps on truenas: “Seems legit!”

Back on track. Quick googling showed that plex runs (or at least used to) scheduled tasks at 2AM. You can ensure whether it’s true or not in your case. And then disable all the tasks, or just stop the plex itself.

1 Like

I am sorry, i forgot to mention it. This is bare metal. This system only runs truenas and I do not have and VM running on this machine either. I have not yet changed zfs arc settings as one person suggested.
I do not know what swap space is but this is near vanilla setting as possible because I am new to truenas…

good idea, let me see if there is scheduled tasks within plex. brb

1 Like


this has to be it? but I do not know what this maintenance tasks are for.

Swap space is a pretty basic concept that has been around for decades; basically a hard drive file somewhere that ram can lean on when memory is running out. The system can flush some things to the cache/swap and stay alive until memory gets freed up. Windows does this. Your Truenas install does this in a sneaky way without really asking for it.

<<<just saw your new post as I was typing

Yes that’s the 2am task. Reschedule it so you can catch it, disable it, or stop the plex app when you’re not using it. You can also try out Jellyfin or Emby, the kids all rave about it and I know Jelly can use hardware transcoding with a discrete GPU, like Plex with a Plexpass does.

<<edit Also funny how your zfs load chart a few posts up, is EXACTLY from 2am to 5am. You’ve found the culprit.

Me neither (almost). I didn’t use plex (welp, I was young and drunk). IMO, you should consult with the plex for truenas app devs.