I recently upgraded to dragon fish. I installed some new storage is a separate backup server and performing a copy operation of about 10TB worth of data from the Truenas system to my backup system. The copy is getting great performance, 400-600MB/sec. Pretty much what I would expect or better.
However, when this copy is running, the truenas webui is not responsive and won’t load. The CLI won’t load either. I can SSH into it, but it is pretty slow in SSH. As soon as I stop the transfer or pause it, everything starts working noramlly. I have multiple apps running on the truenas system and they all seem to be working just fine during the transfer.
The system is a dual Xeon e5-2680v4 14 core 28 thread CPUs with 128GB of ram. Netdata and htop both show the CPU usage as 10-50%. Nothing maxed out. It’s almost like the webui and internal components are being starved for resources, but I am not sure how as it appears that I have plenty to spare.
Try increasing fs.inotify.max_user_watches I mentioned in the thread above. If that doesn’t work, try the others.
I have plenty of RAM so not sure it’s a memory-related issue, but haven’t done a massive rsync job in a while to really test. I also didn’t try reducing the ARC percentage which seems to help some. Even with ARC ‘full’ I have 30GB RAM free.
@SnowReborn Thanks for the highlight. I have left ARC at the default and everything has been fine the past day or two – after disabling swap and increasing some of the sysctl limits. Sometime tonight I’ll put the sysctl values back to default and disable swap to see if that alone is enough to resolve this.
Hoping that’s enough. I got the RAM to have more cache.
For those having this issue, I’d be curious if anybody has tried leaving the defaults but disabling Swap entirely? swapoff -a can be used for this and put into post-init script as well.
I’ve just put my system into a heavy load state and am observing swap in use with 6.2 GiB memory free.
NAME TYPE SIZE USED PRIO
/dev/dm-0 partition 16G 1.6G -2
Exact same slowdown (practically unusable, netdata/stat collection stopped entirely) and it only started after the system started swapping.
For reference, I have vm.swappiness = 1 set so it should only be swapping if absolutely necessary, but it still seems to be doing it early. My boot-pool was 100% busy during this time
Running swapoff -a fixed this immediately.
I can’t find any logs during this timeframe to indicate issues. Let me know if there’s anything I can do to help troubleshoot further.
That mirrors a situation I’ve seen elsewhere. Even if the system is bogged down, running swapoff -a seems to bring it back to life quickly. Would be nice if some others want to test and see if that also is true in their case.
I tried a swapoff -a and the system seemed to get worse. Apps and storage still functioned, but CLI became completely unresponsive.
I rebooted the system and performed a swapoff -a and it executed without issues. The system is functioning completely normal right now. I performed another data transfer of 250GB and I received the same transfer speeds, but the web gui and cli remained completely responsive and functioning normally.
Could be because of a fresh reboot, could also be because of swapoff -a.
