Very slow WebUI - Login, Apps, etc. SCALE Dragonfish RC

TrueNAS General
, ,
81

This is exactly the behavior I am seeing. Been running Freenas/Core for years, currently virtualised in VMWare 8.0 with HBA passed through with triple mirror vdev. Spun up a new VM, with latest Dragonfish 24.04 a few days ago, works for maybe 24 hours before GUI becomes sluggish, but unusable in a large file transfer. File transfers are incredibly fast after a fresh reboot, 550MB/s+, but after a few hours the rate is all over the place and I have seen it as low as 30MB.

Unrelated to this, but I also experienced a scary moment where my passphrase enrypted pool would not unlock, following install of an app, and the creation of the ix-application pool, getting the message:

'/mnt/tank' directory is not empty (please provide "force" flag to override this error and file/directory will be renamed once the dataset is unlocked)

Luckily it looks like a GUI issue as it unlocks fine in Core. I have reverted back to Truenas Core for now - will monitor the bug fixes and try again on a future release.

CC

82

Somebody posted a meme for that…

2 Likes
83

Update:
No performance issues, everything seems fine still.

Arc still growing, but not at 100% yet:

        Target size (adaptive):                        61.0 %  138.6 GiB
        Min size (hard limit):                          3.1 %    7.1 GiB
        Max size (high water):                           31:1  227.2 GiB

Swap usage still increasing slowly from where we were yesterday:

               total        used        free      shared  buff/cache   available
Mem:           228Gi       150Gi        78Gi       6.1Mi       231Mi        77Gi
Swap:           15Gi       1.8Gi        14Gi

Boot disk has had some noise, but not continuous like I was experiencing before:

image
image1602×332 36.1 KB

Disk traffic seems to corelate with timing approximately of swap usage changes in this graph:

image
image1610×711 63.6 KB

2 Likes
84

I’d like to see you try with swappiness 0 as well if you can at some point.

85

image
image739×101 47.7 KB

Done.

1 Like
86

It would have been nice to start with empty swap, can swapoff and swapon I suppose to empty it. Just want to see if any is ever used again.

87

Two days without swap = two days without problems.

2 Likes
88

@brywhi I think you don’t have persistent settings, if you reboot you will lose it. I don’t think you can edit sysctl.conf and be persistent, instead, add it to the UI via System Settings → Advanced → Sysctl and then it will be persistent for sure.

1 Like
89

I’ve said at least twice in this very thread that it shouldn’t be considered a true fix.

2 Likes
90

I’m sure you did, didn’t go back and read all previous posts. I will amend my comment, thanks for pointing it out. We definitely agree it’s not a true fix.

91

Not a big deal. Just reiterating that my two days without problems doesn’t mean I think iX should disable swap and call it a day :wink: For me though, I don’t intend to enable it.

1 Like
92

Yeah, for me, I’d prefer to have it on. To me, it still indicates some other issue, not just zfs vs Linux. Some posters are coming forward in another thread now saying they had the same ARC settings on Cobia without issue. One indicated he had no trouble on Dragonfish on the RC, only the release version. Only a couple, but it does make you wonder what changed.

93

Hate to burst that bubble, but this problem started for me with the RC. Can’t say 100% for sure my RC slowdown was for the same reasons as my Release slowdown, but the symptoms were definitely the same.

94

Thanks! I’ll go ahead and add that after a while if we determine that is the right path to take. :slight_smile: Rock solid today so far!

1 Like
95

Yeah, just saying if you happen to reboot, you lose it so type it again. Glad to hear!

1 Like
96

Update- still running well with swappiness at 0 and other things at defaults.

Boot drive is also quiet and not getting hit aggressively.

Did a quick load test via some SMB and NFS shares, UI stayed performant during these tests. Used fio to do some more load tests directly on the truenas box via ssh, everything continued to run fine.

I set the var vm.swappiness to 0 in sysctl under system settings/advanced- this solution is working well for me, so I want to be sure it persists whenever I reboot next.

Edit: it is still using a tiny bit of swap, but I’m fine with that as long as it isn’t affecting performance or hammering my boot drive, etc.

               total        used        free      shared  buff/cache   available
Mem:           228Gi       199Gi        29Gi        12Mi       328Mi        28Gi
Swap:           15Gi       1.2Gi        14Gi
1 Like
97

This was after a reboot with swappiness=0, and yet still it’s using 1.2 GiB of swap? Shouldn’t it be absolutely zero swap used?

98

Yea, that is the part which bothers me, should be zero ideally.

99

Did you swapoff/swapon as I suggested? Or just left it as it was before, where you already had swap?

100

I didn’t reboot, and I didn’t swapoff/swapon. I’ll do the swapoff/swapon real quick- that should take it to zero.