Beginning about 20 May 2025, I have been experiencing what appears to be random crashes. What I do know, is that I have not changed any configuration or hardware up until that point. I originally thought it might be a docker container causing it, because i was having some issues with my Frigate NVR container. I first realized it was crashing/restarting, because with the containers up, it alerts me when some of them restart. No Truenas related emails or anything. This is verified by checking the uptime in Truenas
This behavior began happening while running Electric Eel Successfully for a while, and so I tried to see if upgrading to Fangtooth 25.04.1 would fix anything. It did not.
To further troubleshoot, and try to ensure it wasn’t a container causing some weird crash, last night, i stopped all my containers and after some time the issue still occurred.
I have ran Memtest86 and all 4 passes completed with no RAM or ECC errors.
I looked through the logs, and nothing jumped out at me from those, but I am also a noob when it comes to looking at the var/log/messages.
Wondering if anyone has any ideas where to go from here, or maybe can look at the logs and see what they think? Below are some approx times that i KNOW the crash/reboot occurred.
-02:38
-23:20
My Setup:
ASROCK RACK W680D4U-2L2T/G5
Intel(R) Core™ i5-14500
128GB 4x Kingston DDR5 4800 ECC UDIMM
LSI/Broadcom 9305 16i HBA
8x Seagate Exos 20TB
6x 2TB Samsung SSDs
2X Seagate Skyhawk AI 12TB
Thank you to anyone who has some thoughts
Full logs attached.
truenas612025_0237.txt (2.1 MB)
Here is an example, but not every crash/reboot looks the same.
May 29 22:13:23 truenas kernel: veth350079d: entered promiscuous mode
May 29 22:13:23 truenas kernel: br-5bc9fd42e2d4: port 2(vethf74ceff) entered blocking state
May 29 22:13:23 truenas kernel: br-5bc9fd42e2d4: port 2(vethf74ceff) entered disabled state
May 29 22:13:23 truenas kernel: vethf74ceff: entered allmulticast mode
May 29 22:13:23 truenas kernel: vethf74ceff: entered promiscuous mode
May 29 22:13:23 truenas kernel: eth0: renamed from veth304e063
May 29 22:13:23 truenas kernel: br-2c0fa881dc58: port 3(veth350079d) entered blocking state
May 29 22:13:23 truenas kernel: br-2c0fa881dc58: port 3(veth350079d) entered forwarding state
May 29 22:13:23 truenas kernel: eth1: renamed from veth1378091
May 29 22:13:23 truenas kernel: br-5bc9fd42e2d4: port 2(vethf74ceff) entered blocking state
May 29 22:13:23 truenas kernel: br-5bc9fd42e2d4: port 2(vethf74ceff) entered forwarding state
May 29 22:13:33 truenas kernel: usb 2-3: reset SuperSpeed USB device number 2 using xhci_hcd
May 29 22:13:33 truenas kernel: usb 2-3: LPM exit latency is zeroed, disabling LPM.
May 29 22:16:09 truenas kernel: av:h264:df0[441885]: segfault at 7fb5779d4bc8 ip 00007fb535c6e67e sp 00007fb5347fe080 error 6 in iHD_drv_video.so[99467e,7fb53542d000+951000] likely on CPU 10 (core 20, socket 0)
May 29 22:16:09 truenas kernel: Code: 01 00 00 80 3d aa 1e ef 01 00 74 30 48 83 fe 1f 0f 87 8e 01 00 00 48 8d 04 76 48 c1 e0 04 49 83 7c 07 60 00 74 16 48 8d 55 b0 <4c> 89 bd e8 6a 1d 43 ff 89 c6 c4 c0 0f 85 91 01 00 00 0b 8b 45 c8
May 29 23:13:18 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 29 23:13:19 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 29 23:13:20 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 00:13:18 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 00:13:19 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 00:13:20 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 01:13:18 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 01:13:19 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 01:13:20 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 01:55:45 truenas kernel: mux0:segment[441783]: segfault at c34a1c3 ip 0000561bc2b58d54 sp 00007fad637fd7a0 error 4 in ffmpeg[985d54,561bc2592000+5e74000] likely on CPU 8 (core 16, socket 0)
May 30 01:55:45 truenas kernel: Code: 4b 10 49 39 c9 0f 84 c1 0d 00 00 48 89 d0 48 0f ba f0 3f 48 85 c0 74 4b 44 89 c8 66 6c 2e 0f 1f 8e 00 00 00 ca 00 48 8b 45 10 <f6> 41 6e 02 75 12 83 8b 4d 10 0b 09 83 e9 36 83 f9 01 3b 87 54 05
May 30 02:13:18 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 02:13:19 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 02:13:20 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 02:13:21 truenas kernel: br60: received packet on vlan60 with own address as source address (addr:1e:2a:55:e3:b9:38, vlan:0)
May 30 02:43:43 truenas syslog-ng[6764]: syslog-ng starting up; version='3.38.1'
May 30 02:42:58 truenas kernel: Linux version 6.12.15-production+truenas (root@tnsbuilds01.tn.ixsystems.net) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Mon May 26 13:44:31 UTC 2025
May 30 02:42:58 truenas kernel: Command line: BOOT_IMAGE=/ROOT/25.04.1@/boot/vmlinuz-6.12.15-production+truenas root=ZFS=boot-pool/ROOT/25.04.1 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
May 30 02:42:58 truenas kernel: x86/tme: not enabled by BIOS
May 30 02:42:58 truenas kernel: x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
May 30 02:42:58 truenas kernel: BIOS-provided physical RAM map:
May 30 02:42:58 truenas kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
May 30 02:42:58 truenas kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
May 30 02:42:58 truenas kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
May 30 02:42:58 truenas kernel: BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
May 30 02:42:58 truenas kernel: BIOS-e820: [mem 0x0000000000100000-0x0000000067767fff] usable
May 30 02:42:58 truenas kernel: BIOS-e820: [mem 0x0000000067768000-0x0000000067768fff] reserved