TrueNAS Random Reboots and NetData Logs Missing

jpwg33691 · April 26, 2025, 4:39am

I spun up a new TrueNas server running 25.04 on a dell R550 it’s been running a couple of weeks but the past 4 days it has randomly crashed 3 times. All three completely different times two within 6 hours and the third 30+ hrs later. iDrac shows host rebooted and NetData I’ve installed through the app store doesn’t show anything abnormal, all resources not being stressed at all before the crash. I discovered though that NetData is not getting the log files for some reason. So I’ve manually combed through a lot of the var/logs and not seeing anything either except evidence that there was a reboot. BUT I am still pretty knew to Linux so easily could be missing something.

Two questions:

What is my best course of action to discover the root source of the crash?

Is there a way to get the log files over to NetData for analysis?

jpwg33691 · April 26, 2025, 2:01pm

Ok, so a little more info. From what I can tell in NetData, the system started shutting down services around1:10, completely down at 1:20, and NetData started getting info again at 1:31. Below is from my syslog server. Any thoughts?

Apr 26 00:50:00 r syslog-ng[97997]: Log statistics; processed=‘destination(_DEFAULT)=10839’, dropped=‘global(internal_source)=0’, processed=‘global(internal_source)=151’, queued=‘global(internal_source)=0’, processed=‘global(msg_clones)=0’, processed=‘source(_DEFAULT)=10839’, processed=‘src.internal(_DEFAULT#0)=151’, processed=‘global(sdata_updates)=17’, stamp=‘src.internal(_DEFAULT#0)=1745646000’, queued=‘global(scratch_buffers_count)=0’, processed=‘global(payload_reallocs)=148’, processed=‘center(queued)=10839’, processed=‘center(received)=10839’, queued=‘global(scratch_buffers_bytes)=0’
Apr 26 00:50:01 TRUENAS systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
Apr 26 00:50:01 TRUENAS systemd[1]: Starting sysstat-collect.service - system activity accounting tool…
Apr 26 00:50:01 TRUENAS systemd[1]: sysstat-collect.service: Deactivated successfully.
Apr 26 00:55:39 TRUENAS systemd-journald[1010]: /var/log/journal/d713ce0d97a945fcb7a46def24e0846b/user-100004411.journal: Journal header limits reached or header out-of-date, rotating.
Apr 26 00:55:39 TRUENAS systemd-journald[1010]: Data hash table of /var/log/journal/d713ce0d97a945fcb7a46def24e0846b/user-100004411.journal has a fill level at 75.0 (8534 of 11377 items, 6553600 file size, 767 bytes per hash table item), suggesting rotation.
Apr 26 01:00:00 r syslog-ng[97997]: Log statistics; processed=‘destination(_DEFAULT)=10903’, dropped=‘global(internal_source)=0’, processed=‘global(internal_source)=152’, queued=‘global(internal_source)=0’, processed=‘global(msg_clones)=0’, processed=‘source(_DEFAULT)=10903’, processed=‘src.internal(_DEFAULT#0)=152’, processed=‘global(sdata_updates)=17’, stamp=‘src.internal(_DEFAULT#0)=1745646600’, queued=‘global(scratch_buffers_count)=0’, processed=‘global(payload_reallocs)=149’, processed=‘center(queued)=10903’, processed=‘center(received)=10903’, queued=‘global(scratch_buffers_bytes)=0’
Apr 26 01:00:01 TRUENAS audit: PROCTITLE proctitle=2F7573722F7362696E2F43524F4E002D66
Apr 26 01:00:01 TRUENAS audit[1370147]: SYSCALL arch=c000003e syscall=1 success=yes exit=1 a0=7 a1=7ffe43229ae0 a2=1 a3=7ffe432297f7 items=0 ppid=5490 pid=1370147 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=93 comm=“cron” exe=“/usr/sbin/cron” subj=unconfined key=(null)
Apr 26 01:00:01 TRUENAS CRON[1370147]: pam_unix(cron:session): session closed for user root
Apr 26 01:00:01 TRUENAS CRON[1370147]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Apr 26 01:00:01 TRUENAS CRON[1370148]: (root) CMD (PATH=“/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin” midclt call cloudsync.sync 1 > /dev/null 2> /dev/null)
Apr 26 01:00:01 TRUENAS systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
Apr 26 01:00:01 TRUENAS systemd[1]: Starting sysstat-collect.service - system activity accounting tool…
Apr 26 01:00:01 TRUENAS systemd[1]: sysstat-collect.service: Deactivated successfully.
Apr 26 01:03:01 TRUENAS systemd-journald[1010]: /var/log/journal/d713ce0d97a945fcb7a46def24e0846b/user-100004411.journal: Journal header limits reached or header out-of-date, rotating.
Apr 26 01:03:01 TRUENAS systemd-journald[1010]: Data hash table of /var/log/journal/d713ce0d97a945fcb7a46def24e0846b/user-100004411.journal has a fill level at 75.0 (8534 of 11377 items, 6553600 file size, 767 bytes per hash table item), suggesting rotation.
Apr 26 01:10:00 r syslog-ng[97997]: Log statistics; processed=‘destination(_DEFAULT)=10973’, dropped=‘global(internal_source)=0’, processed=‘global(internal_source)=153’, queued=‘global(internal_source)=0’, processed=‘global(msg_clones)=0’, processed=‘source(_DEFAULT)=10973’, processed=‘src.internal(_DEFAULT#0)=153’, processed=‘global(sdata_updates)=17’, stamp=‘src.internal(_DEFAULT#0)=1745647200’, queued=‘global(scratch_buffers_count)=0’, processed=‘global(payload_reallocs)=150’, processed=‘center(queued)=10973’, processed=‘center(received)=10973’, queued=‘global(scratch_buffers_bytes)=0’
Apr 26 01:10:05 TRUENAS systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
Apr 26 01:10:05 TRUENAS systemd[1]: Starting sysstat-collect.service - system activity accounting tool…
Apr 26 01:10:05 TRUENAS systemd[1]: sysstat-collect.service: Deactivated successfully.
Apr 26 01:10:17 TRUENAS systemd-journald[1010]: /var/log/journal/d713ce0d97a945fcb7a46def24e0846b/user-100004411.journal: Journal header limits reached or header out-of-date, rotating.
Apr 26 01:10:17 TRUENAS systemd-journald[1010]: Data hash table of /var/log/journal/d713ce0d97a945fcb7a46def24e0846b/user-100004411.journal has a fill level at 75.0 (8534 of 11377 items, 6553600 file size, 767 bytes per hash table item), suggesting rotation.
Apr 26 01:15:05 TRUENAS smartd[67449]: Device: /dev/sdi [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 70 to 69
Apr 26 01:15:05 TRUENAS smartd[67449]: Device: /dev/sdi [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 70 to 69
Apr 26 01:17:01 TRUENAS audit: PROCTITLE proctitle=2F7573722F7362696E2F43524F4E002D66
Apr 26 01:17:01 TRUENAS audit: PROCTITLE proctitle=43524F4E002D66002D4C003135
Apr 26 01:17:01 TRUENAS audit[1384796]: SYSCALL arch=c000003e syscall=1 success=yes exit=1 a0=7 a1=7ffe43229ae0 a2=1 a3=7ffe432297f7 items=0 ppid=5490 pid=1384796 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=94 comm=“cron” exe=“/usr/sbin/cron” subj=unconfined key=(null)
Apr 26 01:17:01 TRUENAS audit[1384799]: SYSCALL arch=c000003e syscall=1 success=yes exit=1 a0=6 a1=7fffe024a470 a2=1 a3=7fffe024a187 items=0 ppid=9849 pid=1384799 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=95 comm=“cron” exe=“/usr/sbin/cron” subj=unconfined key=(null)
Apr 26 01:17:01 TRUENAS CRON[1384796]: pam_unix(cron:session): session closed for user root
Apr 26 01:17:01 TRUENAS CRON[1384796]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Apr 26 01:17:01 TRUENAS CRON[1384797]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Apr 26 01:18:01 TRUENAS systemd-journald[1010]: /var/log/journal/d713ce0d97a945fcb7a46def24e0846b/user-100004411.journal: Journal header limits reached or header out-of-date, rotating.
Apr 26 01:18:01 TRUENAS systemd-journald[1010]: Data hash table of /var/log/journal/d713ce0d97a945fcb7a46def24e0846b/user-100004411.journal has a fill level at 75.0 (8533 of 11377 items, 6553600 file size, 768 bytes per hash table item), suggesting rotation.
Apr 26 01:20:00 r syslog-ng[97997]: Log statistics; processed=‘destination(_DEFAULT)=11048’, dropped=‘global(internal_source)=0’, processed=‘global(internal_source)=154’, queued=‘global(internal_source)=0’, processed=‘global(msg_clones)=0’, processed=‘source(_DEFAULT)=11048’, processed=‘src.internal(_DEFAULT#0)=154’, processed=‘global(sdata_updates)=17’, stamp=‘src.internal(_DEFAULT#0)=1745647800’, queued=‘global(scratch_buffers_count)=0’, processed=‘global(payload_reallocs)=151’, processed=‘center(queued)=11048’, processed=‘center(received)=11048’, queued=‘global(scratch_buffers_bytes)=0’
Apr 26 01:20:01 TRUENAS systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
Apr 26 01:20:01 TRUENAS systemd[1]: Starting sysstat-collect.service - system activity accounting tool…
Apr 26 01:20:01 TRUENAS systemd[1]: sysstat-collect.service: Deactivated successfully.
Apr 26 01:24:46 r syslog-ng[8886]: syslog-ng starting up; version=‘4.8.1’
Apr 26 01:34:03 r syslog-ng[8886]: syslog-ng shutting down; version=‘4.8.1’
Apr 26 01:34:08 r syslog-ng[94388]: syslog-ng starting up; version='4.8.1