Good evening
I notice my ARC is not being fully utilised. My understanding is that ZFS ARC memory allocations are now identical to TrueNAS CORE (NAS-123034) as of build 24.04-BETA.1. Has this changed in subsequent builds or do I have a mis-configuration or misunderstanding? I am on build Dragonfish-24.04.2.
Thank you kindly
The ARC will only fill up as needed.
So if you were to reboot, you would see your ARC (ZFS Cache) at a very small value.
Without context, there’s no way to guess why your ARC hasn’t filled up beyond 11 GiB. Maybe you just haven’t been using the server long enough since the last reboot? Maybe you haven’t been reading much data from the ZFS pools?
I’m noticing the same, even after weeks of uptime on a fairly-active server. ARC isn’t limited to 50% of RAM any more as it was pre-Dragonfish, but it’s definitely smaller than it had been under CORE:
Thank you, ill monitor it over more time.
Just to add for comparison, TrueNAS core (TrueNAS-13.0-U6.1), up for 55 days:
The behavior is a bit strange from what I am use to under Core. I was using Robocopy to copy from Window 10 to SMB on TrueNAS Dragonfish-24.04.2. Over 10 TB.
For some reason, ARC seems to get flushed almost completely out every so often or just even when machine is just idle for long periods.
Graphs over a day. Not sure if this was normal behavior but I did not put in a bug.
I’ve noticed this issue before with my systems…
my best guess so far is RAM fragmentation combined with High I/O seems to cause ARC to collapse.
it’s also worth asking what recordsizes are you using? huge recordsizes put more pressure on the ARC subsystem as a whole.
For those on SCALE that are seeing this, I’m curious what these module parameters reveal:
arc_summary | grep "lotsfree\|sys_free\|arc_max"
Keep in mind that a value of 0 is the default in OpenZFS, which can differ between Linux and FreeBSD.
On my main system (first one I posted):
root@truenas[~]# arc_summary | grep "lotsfree\|sys_free\|arc_max"
zfs_arc_lotsfree_percent 10
zfs_arc_max 0
zfs_arc_sys_free 0
Now I’m more confused. I thought iX intentionally overrode the Linux default of arc_max, otherwise, it will be limited to 50% of available physical RAM.[1]
The rest appear fine.
Another thing I noticed is that “arc_meta_balance” appears to have been removed from the official docs. (But I doubt it has something to do with this behavior.)
For Linux systems. ↩︎
But both your screenshots show that the ARC is well below even 50% of RAM.
40 GiB out of 126, and 12 GiB out of 39.
Yes, at the time of those screenshots. But it has exceeded 50%. Here’s the ARC size over the past week of my main NAS (the one with 128 GB of RAM):
…and the other one (which I have seen exceed 50%, but apparently not in the past week):
conviently one of my servers was running a copy… so here is an arc_summary and arcstat
arc_summary | grep "lotsfree\|sys_free\|arc_max"
returns:
zfs_arc_lotsfree_percent 10
zfs_arc_max 0
zfs_arc_sys_free 0
arcstat returns:
time read ddread ddh% dmread dmh% pread ph% size c avail
13:30:12 3.9K 2.1K 99 1.7K 100 134 100 104G 104G 250G
so, half my 512GB is empty and only 104GB is even being used for ARC.
I’ve also seen it collapse worse, etc… this isn’t exactly predictable aside from high I/O being a contributing factor.
Me too, here’s mine on Dragonfish:
zfs_arc_lotsfree_percent 10
zfs_arc_max 100049993728
zfs_arc_sys_free 0
I do not set arc max. It’s set to about my memory size but not by me.
I definitely see collapses also. They have tuned some other settings, not just arc_max. But not sure which. Mine collapse (at least) when I start a VM. I recall before Dragonfish, starting a VM did something also with arc, I suspect the code is still there.
Do you experience heavy pressure to shrink the ARC on your Dragonfish server?
Added more comments above, I mostly see it when starting a VM from the UI (or command line). I have a few VMs that I only start here and there. An example is Electric Eel nightly. I’ve seen IX post other ZFS settings I believe they changed for Dragonfish latest due to other issues. I’ve seen a couple in tickets as well, but didn’t keep track.
root@truenas[/home/admin]# arc_summary | grep "lotsfree\|sys_free\|arc_max"
zfs_arc_lotsfree_percent 10
zfs_arc_max 0
zfs_arc_sys_free 0
Ditto, min looks the same as others. I assume this means that the ARC can be any size but reserves 10% for system padding? Any update on this, or is this all as expected?
I think it is as expected. I get same results. Sfatula was the only one with different results and had a zfs_arc_max listed
Yeah, and no idea why! Maybe starting a VM after bootup does that, not sure. It definitely does something as the system flushes a lot of the cache immediately.