SFTP connection keeps dropping out

TrueNAS General
1

I am having an issue on a recently set up (about 10 days ago) TrueNAS Scale machine when transferring a large (60GB in this case) file from a Windows machine to the server. I use rclone to do this as I have successfully many times with my TrueNAS Core server. However, when transferring to Scale, randomly during the file transfer, the connection will drop and rclone will retry the transfer. After 3 failed attempts it gives up and stops the operation. I ran a test of transferring the same file to my TrueNAS Core server and it worked flawlessly as expected. This leads me to believe this is a Scale issue but I’m not sure where else to start looking. Other than this, my Scale server is working normally. Below are the parameters of the tests I ran:

Single file, approx 60 GB.
Rclone version 1.70.2 using the Windows command line.
Source is the Windows desktop. Destination is the TrueNAS Scale (And Core as the control) user’s home folder.
I use SSH keys to authenticate to both servers (though I did test Scale with password only and got the same result).
The transfer always seems to fail randomly, not like a corrupted file.
I have rebooted the Windows client, the TrueNAS Scale server, the network equipment that connects all of them together.
All of these machines are on different subnets but all the routing config appears to be in place, and if it weren’t I would think the transfer would never start at all.

I am not sure what log files might be relevant to provide or where they would be located. This does not appear to be an authentication issue with SSH as I looked at auth.log and it appears normal. Thank you for your assistance.

2

I’m posting this reply with an updated finding. I was attempting to transfer some files, a few thousand of varying sizes from severak kilobytes to 30GB or more, from my TrueNAS Core server to the TrueNAS Scale server that was having this problem and I am encountering the same result. To me that confirms this is an issue with the Scale server and not the originating machine whether it be Windows or TrueNAS Core which have now been used. I have no further update at this time.

3

I did some more testing tonight. Here are the abbreviated results.
-Transfer from TrueNAS Core to TrueNAS Scale using rclone while monitoring dmesg output from both ends. No message output when the problem occurs.
-Monitor auth.log and journalctl -uf ssh output on the TrueNAS Scale side while trying rclone, rsync, and WinSCP from clients. The session appears to close normally when the drop occurs and re-establishes, so there is nothing else interesting there.
-When trying this with rsync on a Windows client (via Windows subsystem for Linux), I did get a more useful error message. “Connection reset by peer.” This leads me to believe the problem is indeed on the TrueNAS Scale side and for some reason it is stopping the connection unexpectedly.
-I verified using htop and filtering to sshd during a transfer. The CPU usage was normal for two processes that came up during the transfer, client@notty and client@internal-sftp. When the problem occurred, the CPU usage on both of these processes drops to zero, and then the processes die and new ones start in their place.

That is all of the additional info I have right now. I appreciate any other suggestions.