WebUI slow to log in after entering login details. Progressively worse

TrueNAS General
1

For over a year I have consistently had an issue where after a few weeks/months the log in time to the WebUI was getting progressively slower, even at one point it would time out and not log in at all.
After hours and hours of searching, reading logs, testing I was never able to find the cause and a fresh install and config restore would resolve the issue until it inevitably would occur again.

Then after an afternoon with GBT 5.1 as a sidekick, I eventually found the solution - and I wonder if this also the cause of some others issues they are unable to resolve, so I have decided to share.

For openess, the above is my own words but for speed and conciseness the below is GPT generated, in my style:

After a lot of digging, I traced my progressively slow WebUI logins to an unexpected culprit: a massively bloated /var/log/wtmp file.

TrueNAS SCALE records login sessions (including WebUI sessions and certain service logins like FTP) in the legacy Linux WTMP database. Middlewared updates this file on every successful login and, critically, scans the entire file during authentication.

If WTMP grows very large, this scan becomes slow and results in:

  • Long WebUI login delays

  • Brief CPU spikes in asyncio_loop followed by idle

  • No errors in middlewared logs

  • Fast API responses when tested manually

  • Behaviour returning to normal after reinstalling SCALE

In my case, /var/log/wtmp had grown to over 220 MB, mostly due to automated FTP connections from IP cameras. Reinstalling SCALE temporarily fixed the issue because the WTMP file was recreated empty. Over several weeks, it grew again and the slowdown returned.

Workaround

Reset the bloated WTMP file

sudo systemctl stop middlewared

sudo mv /var/log/wtmp /var/log/wtmp.old.$(date +%s)
sudo touch /var/log/wtmp
sudo chown root:utmp /var/log/wtmp
sudo chmod 664 /var/log/wtmp

sudo systemctl start middlewared

This immediately restored fast WebUI logins.

———————– My own words again ——————-

Probably wtmp is regenerated if you just delete it? I didn’t try though. In which case just mv to make a backup would be enough along with restarting the service.

I have not yet implemented a solution to automatically prevent this either, I intend to set up log rotation or a job to carry out the above workaround automatically once every few weeks. Not sure what anyone elses thoughts are on this or if there is a recommended solution, I am surprised it is not rotated by default?

Anyway, I hope if anyone else had my slightly unusual cause for their slow WebUI login - followed by normal behaviour - this helps.

2

You could file this as a bug on the official tracker:

There already is a monthly rotation of wtmp:

$ cat /etc/logrotate.d/wtmp
# no packages own wtmp -- we'll rotate it here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    minsize 1M
    rotate 1
}

You can safely delete the file. If you want to make sure, you can create it manually afterwards (for example do touch /var/log/wtmp; chmod 0644 /var/log/wtmp; chown root:utmp /var/log/wtmp)

You could try to disable FTP connections logging to wtmp. In the FTP Options you can set Auxilary Parameters:

WtmpLog off

See Configuration Directive List. Don’t know if it works, haven’t tried it personally.