Tracking abnormal reboot issues in TrueNAS SCALE

TrueNAS General
1

Hey guys,

I don’t know if anyone else has been plagued by abnormal restarts for as long as I have. I’ve encountered varying degrees of abnormal system restarts in 20.10 Beta1, RC1, and the official release.

Currently, I suspect the following factors may cause the abnormal restart: containers using libtorrent 2.0 (such as qBittorrent), OpenWrt virtual machines, and starting a new process that consumes a lot of resources when the system’s CPU and memory usage are high. Here’s the ticket I submitted on Jira

Since uploading via the private link consistently resulted in errors, I uploaded two debug files using Bitwarden’s Seed feature. The abnormal restart in the first file corresponds to the currently reported issue, while the abnormal restart in the second debug file is likely due to an OOM Error. However, I believe this debug file might be helpful in troubleshooting the issue we are currently discussing, so I also uploaded it to Jira.

2

I’m running over a dozen containers on TrueNAS, and simultaneously running OpenWrt as a router in virtual machines. OpenWrt and most of the containers are configured to be on the same subnet via a virtual bridge and MACVLAN.

The issues encountered in versions 25.10 Beta1 and RC1 are the same; the following descriptions are primarily based on verification in the RC1 version. When I run qBittorrent 5.0.4 (based on libtorrent 2.0.11) through containers and use Sing-Box’s TUN mode for traffic proxying in OpenWrt, I experience at least one abnormal reboot per day. After completely disabling Sing-Box, the frequency of abnormal reboots decreases to once every 3-5 days. After downgrading qBittorrent’s libtorrent library to 1.2.20, I haven’t encountered any abnormal reboots since.

When I upgraded TrueNAS to the official 25.10 release, I immediately experienced two abnormal reboots within 10 hours. A significant change during this period was that I deployed Tailscale through TrueNAS. After a manual restart, Tailscale remained in a deployment state (i.e., not running), and the abnormal restart mysteriously disappeared. Therefore, I suspect Tailscale caused the abnormal restart in the 25.10 official release.

Subsequently, I encountered another abnormal restart. I was pulling OneDrive data using TrueNAS’s cloud backup function. After the process lasted for several hours, I launched the Syncthing app on my phone, hoping to synchronize data with the Syncthing container on TrueNAS. TrueNAS immediately restarted abnormally.

Since each abnormal restart carries the risk of data loss, and OpenWrt network interruptions can only be recovered manually, I cannot test frequently.

My hypothesis is as follows: libtorrent 2.0 made significant changes to the Disk I/O model, defaulting to using memory-mapped files and delegating memory scheduling to the kernel. Sing-Box runtime might experience increased memory usage and network latency due to sudden traffic spikes. When memory usage exceeds limits, a severe OutOfMemoryError (OOM) or a driver crash in the kernel occurs, leading to an abnormal system restart. The Syncthing incident is a similar possibility.

Tailscale containers run with root privileges and SYS_MODULE permissions by default, so any errors they encounter will have a far greater impact on the system kernel than those in other containers.

The debug files submitted in Jira correspond to abnormal restarts that may have been caused by Tailscale and Syncthing OOM.

3

My sing-box isn’t handling this traffic, and QB is only using about 1.5 GB of memory, but it still keeps rebooting.

I never encountered this issue on version 25.4.x. I’m currently using 25.10.0, and it has rebooted once in the past two days.

On version 25.4.x, I could run 80 Docker containers without any of these issues.

1 Like
4

My Sing-Box is always configured not to handle any BitTorrent traffic, so it should only classify traffic and not forward it. This shouldn’t affect the probability of abnormal restarts.

I don’t know if your qBittorrent has experienced issues like lost seeds due to frequent abnormal restarts, which is common when using an SQLite database. If a large number of seeds are corrupted, switch SQLite back to quick recovery files and restart qBittorrent. After all quick recovery files have been created, switch back to SQLite and then move the seed files to determine which seeds are no longer managed by qBittorrent. Although lost seeds can only be manually added back, there shouldn’t be any more seed corruptions afterward.

5

What I mean is that mine doesn’t do any processing either. I suspect it’s a problem with the kernel or the system itself, because it used to work perfectly fine before.

I just uploaded a debug log.

6

Do both of you run sing-box - isn’t this an unusual coincidence??

The other case is here: @CacheMeIfYouCan

Do you run sing-box?

7

No I am not using Sing-Box.

I have provided my hardware and software configuration here Spurious reboot on 25.10.0 - #4 by CacheMeIfYouCan

8

Sing-Box significantly increases the probability of abnormal restarts, but it’s not the decisive factor.

  1. All current feedback comes from version 25.10, indicating a possible link to kernel or driver updates in 25.10.

  2. Based on my testing, downgrading libtorrent from version 2.0 to 1.x resolved abnormal restarts on 25.10 RC1 until the official 25.10 release.

  3. Another variable is Tailscale; abnormal restarts stopped after Tailscale was disabled in the official 25.10 release.

These applications share the characteristic of complex network stack calls and may experience increased memory usage during sudden traffic spikes.

1 Like
9

Thanks for the detailed analysis… NAS-138243

I don’t think anyone else is using Tailscale… please confirm/deny if I’m wrong.

10

I’m having similar issue.

After update to 25.10.0 - Goldeye my Truenas was rebooting abnormally, but just during the night. Last 2 nights it reboots 00:18am and 03:47am. During the day, all is working fine - adguard-home, nginx-proxy-manager, tailscale, torrents (ARRs), 2 containers (FreePBX, HAOS).

Today I will stop tailscale and see what happens.

Thanks a lot,

1 Like
11

I’d like to know if there are any other cases of abnormal restarts encountered while using qBittorrent 5.x (libtorrent 2.x).

12

Running Tailscale on virtualized Pfsense on 25.10 without any problems yet. It’s VM vs Container tho.