Thanks! I’ll go ahead and add that after a while if we determine that is the right path to take. Rock solid today so far!
Yeah, just saying if you happen to reboot, you lose it so type it again. Glad to hear!
Update- still running well with swappiness at 0 and other things at defaults.
Boot drive is also quiet and not getting hit aggressively.
Did a quick load test via some SMB and NFS shares, UI stayed performant during these tests. Used fio to do some more load tests directly on the truenas box via ssh, everything continued to run fine.
I set the var vm.swappiness
to 0 in sysctl under system settings/advanced- this solution is working well for me, so I want to be sure it persists whenever I reboot next.
Edit: it is still using a tiny bit of swap, but I’m fine with that as long as it isn’t affecting performance or hammering my boot drive, etc.
total used free shared buff/cache available
Mem: 228Gi 199Gi 29Gi 12Mi 328Mi 28Gi
Swap: 15Gi 1.2Gi 14Gi
This was after a reboot with swappiness=0
, and yet still it’s using 1.2 GiB of swap? Shouldn’t it be absolutely zero swap used?
Yea, that is the part which bothers me, should be zero ideally.
Did you swapoff/swapon as I suggested? Or just left it as it was before, where you already had swap?
I didn’t reboot, and I didn’t swapoff/swapon. I’ll do the swapoff/swapon real quick- that should take it to zero.
Done!
root@storage01[~]# swapon -s
Filename Type Size Used Priority
/dev/dm-0 partition 16777212 1211860 -2
root@storage01[~]# swapoff -a
root@storage01[~]# swapon /dev/dm-0
root@storage01[~]# free -mh
total used free shared buff/cache available
Mem: 228Gi 213Gi 15Gi 99Mi 332Mi 15Gi
Swap: 15Gi 0B 15Gi
Beginning to get déjà vu
Which just shows that Core was not always perfect either
Update- still no swap usage. Woot woot!
root@storage01[~]# free -mh
total used free shared buff/cache available
Mem: 228Gi 216Gi 12Gi 99Mi 372Mi 12Gi
Swap: 15Gi 0B 15Gi
root@storage01[~]# arc_summary | head -13 | tail -3
Target size (adaptive): 89.4 % 203.1 GiB
Min size (hard limit): 3.1 % 7.1 GiB
Max size (high water): 31:1 227.2 GiB
No.
NO.
NOOOOOOOOOOOO!!!
This can’t be happening. I will not stand for this! I cannot allow SCALE to win! CORE WILL NEVER DIE!
@brywhi: Please update your post and fabricate how everything is breaking, and aggressively swapping, and the ARC caused mold to grow in your house.
Solid over the weekend, still no swappage.
Cross-posting for visibility.
We can confirm that the significant contributor to the issue that has been reported in this thread and others is related to a Linux kernel change in 6.6. While swap adjustments proposed can mitigate the issue to some degree, the multi-gen LRU code in question will be disabled by default in a subsequent Dragonfish patch release, and should resolve the problem. This can be done in advance following the directions below from @mav. Thanks to all of the community members who helped by reporting the issue which aided in reaching the root cause. Additional feedback is welcome from those who apply the change below.
We’ve got some bad news.
@brywhi just DM’d me in private. With his permission, I’ll share his message:
Hm… there’s a fine line between humour and trolling…
Hahaha this is awesome thanks for the laugh!
Perfect, thanks for the info
I hope that someone will feedback this clash between Open ZFS and MGLRU to both the Linux Kernel folks and the Open ZFS folks (and maybe the major Linux distros that support ZFS) so that:
- Any other (non-TrueNAS) users of ZFS can be warned about the current issue.
- The Linux Kernel and / or ZFS code can be tweaked so that they play well together.
Hi @Protopia
We are in conversation with the developer, and have been testing some proposed patches and providing feedback.
This is awesome news @patrickkeane , good work tracking it down. It never made sense to me that arc was doing what harm it was doing without other issues being involved. I am glad the true issue has been found!