So there was this bug on ixsystems dot atlassian dot net (can’t post link)
NAS-132365
NAS-133690
NAS-132587
Which got fixed in 24.10.0.2 but seems to me that I still have the bug. Can be easily reproduced by just running an rsync from the NAS to an SMB share (either Windows or Linux) with big files over 3 GB. After around 10-30 minutes (it’s random), TrueNAS is about 95% unresponsive by listing files and typing commands in SSH. The dashboard is 100% unresponsive. After running sudo killall rsync, everything comes back normally.
One temporary fix is to use for rsync the option --bwlimit=4000 which limit the speed.
Alerts are differents but they all say the same thing which means the system is unresponsive, some examples:
NTP health check failed - No Active NTP peers: [{‘SERVER: NOT_SELECTABLE
Failed to check for alert Quota: Failed connection handshake
Failed to check for alert Smartd: b’System.Error.ENOTCONN’[err -107]: b’Transport endpoint is not connected
I can make a bug report but wanted to ask here before.
Unrelated issue? Quite possibly but they all have speed related issue with SMB. Anyway, not important.
I log in SSH in TrueNAS. I mount my Windows ans Linux SMB share inside TrueNAS. Then I rsync the data in TrueNAS to both my computers. The issue will be the same : TrueNAS freeze / hangs like I explained above.
The SMB server and SMB client are different components of the product, so all linked bug reports are irrelevant.
At most it sounds like you’ve found a potential issue with the kernel SMB client, but we generally don’t support the operations you’re describing (see warning message in shell).
The warning shell message is about “making configuration changes” but in my case it’s just about running rsync to an SMB share while inside SSH, which are all tools provided by TrueNAS.
Note that it’s a pain right now because I’m doing a first synchronization from the NAS to both my laptops but once it’s completed, there’s not much changes to the files so the issue is practically gone after. Like one of my laptop has finished and when doing the rsync, doesn’t even take 2 minutes, it’s blazing fast.
I used to have a Netgear ReadyNAS 214, was doing the same thing, an rsync inside SSH, never had issue of speed.
I just found out something interesting, while starting rsync in SSH from TrueNAS to a mounted SMB share also in TrueNAS, if I go look at the Dashboard, the RAM section will see a drop of 0.1 GB every 10 seconds or so. When it reach around 0.7 GB left, the system knows something is going on and free up some RAM and goes back up to around 1.8 and the cycle continues. At one point however after an hour, it just doesn’t go back up and the whole thing freeze.
Here’s a real use case, the whole system froze at around 20:30. Like I said, a simple sudo killall rsync makes the system usable again.
After it freeze, starting another rsync is a bad idea, the system doesn’t know how to free up memory again and will then freeze yet again in a much shorter time. The only thing to do is reboot TrueNAS. I also noticed that if I don’t reboot and start another rsync, the transfer is so slow (almost frozen) that it’s just not a good idea to continue using it. Gotta reboot.
If you encounter this problem, then know that at least you have transferred about an hour of files
Like I said, I can recreate the issue anytime, takes about an hour, would even be faster to reproduce with less RAM!
I was thinking of doing a bug report, having a dump is a good idea, do you have documentation on how to do it? If you think they can ask anything else, let me know, I’m new here.
Yes, ROBOCOPY and many other solutions exist, there is more than one way to skin cat but I’m not trying to find another solution, I will stick with what I am doing, most files doesn’t change and having an hour is plenty in my case.
Why would you do this? TrueNAS is designed to be a server, not a client. Just mount TrueNAS share in both computers and run rsync from there instead. What you’re doing kinda’ defeats the whole point of a NAS.
I fail to understand the concept that because it is a server, you must not connect to it and you must not install a service. For me, it defeats the whole purpose of being a server
Why would I do this? Because it’s easier in the long term, I just need one script on the NAS that rsync to multiple computers, Windows or Linux. If you do it from Windows, you would need rsync, either cygwin or that new Linux emulator in Windows that I haven’t had the time to try yet or I think they make rsync as an EXE now, not sure. Right now, it works, right on the NAS and if I adjust the script on it (it’s not just an rsync command, it’s a full script with about 200 lines of code in it), it’s only one place and not multiple computers. On the other computers, all I need is to share a folder, easy! I also format my computers from time to time and it’s one less thing to reinstall and think about.
I’m skipping a lot of details but what I do really works and has been working for about 8 years with my last READYNAS without an itch.
Whatever it is needed or not, at the very least, I think we can all agree that doing a simple rsync from TrueNAS in SSH to a mounted SMB shared also right in TrueNAS should work (works everywhere else in every server) but alas, it fails!
Now that I think about it, I haven’t tried an rsync in SSH from TrueNAS to TrueNAS by skipping the SMB share, that would be interesting to try by leave out the SMB protocol. Will report later on!
Except it’s not just a server. It’s a server intended to run as a “firmware/appliance” with a middleware software running on top that is supposed to mediate everything. Hence, the warning on the CLI, because this mode of operation is unsupported.
Well, clearly each OS runs differently. I hope you get the issue worked out. Frankly, I don’t have a lot of confidence your bug reports will be addressed because it’s an unsupported mode of operation. Good luck either way though.
