Very slow WebUI - Login, Apps, etc. SCALE Dragonfish RC

brywhi · May 8, 2024, 2:47pm

Ha, ok.

root@storage01[/sys/module/zfs/parameters]# cli
[storage01]> system advanced update kernel_extra_options="zfs_arc_max=243926810624"
[storage01]> exit
root@storage01[/sys/module/zfs/parameters]# echo 243926810624 >> /sys/module/zfs/parameters/zfs_arc_max
root@storage01[/sys/module/zfs/parameters]# cat /sys/module/zfs/parameters/zfs_arc_max
243926810624
root@storage01[/sys/module/zfs/parameters]# arc_summary | grep 'Max size'
        Max size (high water):                           31:1  227.2 GiB
root@storage01[/sys/module/zfs/parameters]# arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report                            Wed May 08 08:46:11 2024
Linux 6.6.20-production+truenas                                  2.2.3-1
Machine: storage01 (x86_64)                                      2.2.3-1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    39.0 %   88.7 GiB
        Target size (adaptive):                        39.2 %   89.1 GiB
        Min size (hard limit):                          3.1 %    7.1 GiB
        Max size (high water):                           31:1  227.2 GiB
[...]

root@storage01[/sys/module/zfs/parameters]# free -mh
               total        used        free      shared  buff/cache   available
Mem:           228Gi       100Gi       128Gi        99Mi       398Mi       127Gi
Swap:           15Gi          0B        15Gi

Seems to have worked. I’ll keep ya posted!

brywhi · May 8, 2024, 2:49pm

Should I manually set this to something other than the default of 0?

root@storage01[/sys/module/zfs/parameters]# cat zfs_arc_sys_free
0

winnielinnie · May 8, 2024, 2:49pm

Leave it as is. The fewer changed variables, the better.

kris · May 8, 2024, 3:07pm

Hey now, context is everything here

In my worldview you generally only swap for two reasons:

Memory Leak (Swap only prolongs and extends the suffering in this case)
Overloaded System (I’d rather find out right away that I can’t run 64 simultaneous Apps and 2VMs on my 16GB system)

brywhi · May 8, 2024, 4:01pm

Interesting- it has started swapping. Not much, but it is there.

winnielinnie · May 8, 2024, 4:25pm

Honestly, that’s not a good sign. It means that “swappiness = 1” doesn’t work as advertised. It shouldn’t be swapping at all, unless the RAM is truly at its very limit and cannot release anything.

brywhi · May 8, 2024, 4:45pm

Exactly. I was surprised to see it.

I’ll let it keep running as-is for a bit. I’m curious if it will grow or stay about the same.

winnielinnie · May 8, 2024, 4:48pm

That won’t really indicate much, since it’s the constant swap in/out that is the cause for the slowdowns. (In other words: Acting like it’s a low-RAM system, even though it’s not.)

Keep an eye out for sluggishness and slowdowns, even if the “used swap” remains relatively low.

Jorsher · May 8, 2024, 5:37pm

Not quite 24 hours, but it’s close enough. swapoff -a has definitely resolved the slow-down I was experiencing. It’s really hard to tell with these things, but it seems CPU utilization has dropped as well. It was never really ‘high’, but it seems lower.

This graph is kind of difficult to see, but notice how the iowait (purple) was higher until I disabled swap ~2 days ago. The couple spikes were of course when I rebooted to test without the other sysctl changes:

Could be wrong, but I assume this is one of the drives that swap was using:

Swap usage before/after disabling. No surprise that it’s no longer being used, but unsure why it was ever used considering the ‘mean’ of 40gb free RAM:

sfatula · May 8, 2024, 6:10pm

I thought that was = 0, not 1?

winnielinnie · May 8, 2024, 6:11pm

Used to be, but modern versions of the Linux kernel treat “0” as “disabled” and “1” as the way “0” used to behave.

EDIT: Oh, I see what you mean.

I wrote “It shouldn’t be swapping at all”, because there is no reason for their system to resort to swapping, given the amount of RAM and memory pressure. The value of “1” advertises as “Swap used only if absolutely necessary.” But that’s clearly not the case, from what we can observe.

My guess is it has to do with ZFS/ARC and Linux. (Swap is still in the pre-ARC days.)

sfatula · May 8, 2024, 6:12pm

Yes, 0 should be the same as swapoff. Or are you trying to basically keep swap but only for extreme cases? Have not read it all, sorry.

winnielinnie · May 8, 2024, 6:15pm

Yes, that’s the “supposed” behavior, but it’s not working as expected. There is no emergency, yet it still wants to swap to disk, when there’s plenty of free RAM. (See my post above, I edited it.)

sfatula · May 8, 2024, 6:36pm

Well, I don’t see it, was the system rebooted after changing the setting? It depends how it was changed whether it took effect or not. Also, what was the source of the new swappiness behavior, where did you find it was changed? I wonder if that might be on a newer kernel than Scale is using? Everything I am reading says it’s the same as it’s always been? Which means 0 means smallest possible swap (but not none). If that is the case, it might still swap even with 0.

Here’s a 2024 reference from Redhat: Change in swap behavior between RHEL 7 and RHEL 8 kernels - Red Hat Customer Portal

brywhi · May 8, 2024, 7:17pm

No reboot, won’t be able to do that for a while.

No speed issues on user interface, it is working fine. Not under high load at the moment though, I’ll try to simulate that later.

Current stats:

free -mh
               total        used        free      shared  buff/cache   available
Mem:           228Gi       104Gi       124Gi       9.7Mi       255Mi       124Gi
Swap:           15Gi       701Mi        15Gi


arc_summary | head -13 | tail -3

        Target size (adaptive):                        41.1 %   93.4 GiB
        Min size (hard limit):                          3.1 %    7.1 GiB
        Max size (high water):                           31:1  227.2 GiB

One thing that is interesting is that prior to disabling swap, my boot drive was getting hit pretty hard. (See What is writing to my boot drive? - TrueNAS General - TrueNAS Community Forums) As soon as I turned off swap, that quit hammering the boot disk. With swappiness at 1, the boot disk is still pretty darn quiet. So far, it seems that just changing swappiness to 1 from 60 has been a good move.

Last hour:

sfatula · May 8, 2024, 7:20pm

So you are saying swappiness 1 and swap is back on?

brywhi · May 8, 2024, 7:23pm

Bingo. Swap enabled, swappiness 1. zfs_arc_max at total memory minus 1 gb, which as I understand is boot up default.

UI is responsive, boot disk isn’t having more than expected traffic. Everything seems happy right now… knock on wood

sfatula · May 8, 2024, 7:24pm

I think the way forward will be to disable swap though based on reading the other thread with IX comments. They’ll have to change the memory recommendations. Not for you though! Interesting experiment though. See how it is in a day or so. Report back.

Are you SURE swap is back on? The mdstat output didn’t show any swap partitions. Maybe that changed in Dragonfish, wouldn’t think so. What does free -m show?

Still, this is what it should do. Shouldn’t be swapping unless a good reason. I’m interested in seeing the results, thanks for testing!

ClimbingKid · May 8, 2024, 8:50pm

This is exactly the behavior I am seeing. Been running Freenas/Core for years, currently virtualised in VMWare 8.0 with HBA passed through with triple mirror vdev. Spun up a new VM, with latest Dragonfish 24.04 a few days ago, works for maybe 24 hours before GUI becomes sluggish, but unusable in a large file transfer. File transfers are incredibly fast after a fresh reboot, 550MB/s+, but after a few hours the rate is all over the place and I have seen it as low as 30MB.

Unrelated to this, but I also experienced a scary moment where my passphrase enrypted pool would not unlock, following install of an app, and the creation of the ix-application pool, getting the message:

'/mnt/tank' directory is not empty (please provide "force" flag to override this error and file/directory will be renamed once the dataset is unlocked)

Luckily it looks like a GUI issue as it unlocks fine in Core. I have reverted back to Truenas Core for now - will monitor the bug fixes and try again on a future release.

CC

Stux · May 8, 2024, 9:34pm

Somebody posted a meme for that…