Syncthing makes TrueNAS unresponsive

[ajp_anton]

TrueNAS Scale 24.04.0
Official Syncthing app, 1.27.7 / 2.0.6

Whenever Syncthing is downloading lots of data, the whole system becomes unresponsive. Downloading at ~20MB/s, the system locks up before the first 5GB has transferred. Accessing the web-ui login page can take 10 minutes and actually signing in never succeeds. SSH is really slow, and SMB shares become inaccessible. Syncthing still appears to be transferring data all the time at constant maximum speed, as its web-ui is still somewhat accessible. It does lock up occasionally for a minute, but then resumes.

The NAS has 60TB of usable storage and 24GB of RAM. This is less than the “1GB/1TB” that I see recommended. Not sure if this can cause issues like this, as it’s not like I’m accessing all that data frequently. This NAS is mostly a backup of my main NAS located elsewhere.

While locked up, this is what gets spammed on the console:

I also keep getting alerts like this:

The alerts go away when Syncthing is done, and everything goes back to normal.

[neofusion]

Could it be related to:

Going insane, truenas webUI dies, again

After having re-installed a fresh copy of dragonfish 3 times for various buggy issues, I am once again in a situation where the webui just dies. Eventually my uptime monitors start to fail and then downtime happens for a few seconds for the apps. An hour after booting the system, the webui no longer loads. “Connecting to TrueNAS …” SSH still works but slowly, running heavyscript to list apps fails. Killing all apps seems to get the webui back but that is obviously not the solution. Leaving a p…

Quick summary is that in 24.04.0, due to some unfortunate interactions between RAM, ARC and swap, the system can become unresponsive.

[abonadt]

I had the same problem and found this solution https://forums.truenas.com/t/very-slow-webui-login-apps-etc-scale-dragonfish-rc/825/106. This instantly helped to make WebGUI responsive again.

[ajp_anton]

Thanks, reducing the ARC to 50% seems to have done the trick.

[ajp_anton]

Might look into this at some point, as I’m kind of wasting RAM with the 50% solution. But I might also just wait for an official fix and not mess with kernel stuff. Wasn’t clear from those discussions if that fix survives a reboot.

[Stux]

You need to add the echo command as pre-init startup command so it is performed at each boot.

I’m assuming iX will change the kernel default in future so when the next update comes out you can remove the startup command.

[abonadt]

I just applied this:
echo n >/sys/kernel/mm/lru_gen/enabled
and I use full RAM capacity. Guess, you can use either of the two solutions.