Hi all,
I’m seeing repeated, clean shutdowns on a TrueNAS SCALE system and I’d like to understand whether this is a watchdog / kernel / Hardware / SCALE issue.
The new system shuts down on its own after running for some time, roughly every 9-12 hours. According to the system reports, the system is down for anywhere from 10-15 minutes, where there is no activity recorded. Then it reboots/restarts.
There have been no external power loss and on the next boot there’s no indication of an unclean shutdown or crash.
Hardware
Board: MSI PRO B850-P WIFI (AM5, AMD chipset)
CPU: Ryzen 7 7700 RAM:
Corsair 64gb Boot device:
Kingston 500gb NVMe
Data drives: none
Graphics card: Nvidia GT710
PSU: bequiet Powerzone 2 750W
Software
TrueNAS version: 25.10.0.1 Fresh Install
I have done a google search and looked at a couple of logs. These look normal to me ( though I have not much experience in finding log errors.) The the logs show what looks like a normal, orderly systemd shutdown sequence, as if I had requested a shutdown from the UI.
What I see in logs
From journalctl -b -1 / journalctl -k -b -1 around the time of shutdown:
I see the usual shutdown steps:
systemd-shutdown[1]: Syncing filesystems and block devices.
systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Services stopping in a clean order.
I do not see:
Kernel panics
Obvious I/O errors (no I/O error, blk_update_request, nvme reset, etc. in the immediate lead‑up)
LOGS:
[ 21.097474] kvm_amd: AVIC enabled
[ 21.097475] kvm_amd: x2AVIC enabled
[ 21.097475] kvm_amd: Virtual GIF supported
[ 21.097476] kvm_amd: Virtual NMI enabled
[ 21.868501] MCE: In-kernel MCE decoding enabled.
[ 21.869145] [drm] amdgpu kernel modesetting enabled.
[ 21.869158] amdgpu: vga_switcheroo: detected switching method \_SB_.PCI0.GP17.VGA_.ATPX handle
[ 21.869246] amdgpu: ATPX version 1, functions 0x00000000
[ 21.876040] amdgpu: Virtual CRAT table created for CPU
[ 21.876049] amdgpu: Topology: Add CPU node
[ 21.876135] amdgpu 0000:0e:00.0: enabling device (0006 -> 0007)
[ 21.876159] [drm] initializing kernel modesetting (IP DISCOVERY 0x1002:0x164E 0x1462:0x7E56 0xC5).
[ 21.876166] [drm] register mmio base: 0xF6500000
[ 21.876167] [drm] register mmio size: 524288
[ 21.877670] [drm] add ip block number 0 <nv_common>
[ 21.877671] [drm] add ip block number 1 <gmc_v10_0>
[ 21.877672] [drm] add ip block number 2 <navi10_ih>
[ 21.877673] [drm] add ip block number 3 <psp>
[ 21.877674] [drm] add ip block number 4 <smu>
[ 21.877675] [drm] add ip block number 5 <dm>
[ 21.877676] [drm] add ip block number 6 <gfx_v10_0>
[ 21.877677] [drm] add ip block number 7 <sdma_v5_2>
[ 21.877678] [drm] add ip block number 8 <vcn_v3_0>
[ 21.877679] [drm] add ip block number 9 <jpeg_v3_0>
[ 21.877688] amdgpu 0000:0e:00.0: amdgpu: Fetched VBIOS from VFCT
[ 21.877690] amdgpu: ATOM BIOS: 102-RAPHAEL-008
[ 21.885178] amdgpu 0000:0e:00.0: amdgpu: Trusted Memory Zone (TMZ) feature disabled as experimental (default)
[ 21.885206] [drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
[ 21.885210] amdgpu 0000:0e:00.0: amdgpu: VRAM: 512M 0x000000F400000000 - 0x000000F41FFFFFFF (512M used)
[ 21.885212] amdgpu 0000:0e:00.0: amdgpu: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
[ 21.885218] [drm] Detected VRAM RAM=512M, BAR=512M
[ 21.885219] [drm] RAM width 128bits DDR5
[ 21.885270] [drm] amdgpu: 512M of VRAM memory ready
[ 21.885272] [drm] amdgpu: 31677M of GTT memory ready.
[ 21.885281] [drm] GART: num cpu pages 262144, num gpu pages 262144
[ 21.885372] [drm] PCIE GART of 1024M enabled (table at 0x000000F41FC00000).
[ 21.885811] [drm] Loading DMUB firmware via PSP: version=0x05002000
[ 21.886100] [drm] use_doorbell being set to: [true]
[ 21.886113] [drm] Found VCN firmware Version ENC: 1.33 DEC: 4 VEP: 0 Revision: 6
[ 21.907904] amdgpu 0000:0e:00.0: amdgpu: reserve 0xa00000 from 0xf41e000000 for PSP TMR
[ 21.964506] intel_rapl_common: Found RAPL domain package
[ 21.964508] intel_rapl_common: Found RAPL domain core
[ 21.964661] amd_atl: AMD Address Translation Library initialized
[ 21.975087] amdgpu 0000:0e:00.0: amdgpu: RAS: optional ras ta ucode is not available
[ 21.980935] amdgpu 0000:0e:00.0: amdgpu: RAP: optional rap ta ucode is not available
[ 21.980936] amdgpu 0000:0e:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
[ 21.982290] amdgpu 0000:0e:00.0: amdgpu: SMU is initialized successfully!
[ 21.982292] [drm] Seamless boot condition check passed
[ 21.982589] [drm] Display Core v3.2.301 initialized on DCN 3.1.5
[ 21.982590] [drm] DP-HDMI FRL PCON supported
[ 21.983347] [drm] DMUB hardware initialized: version=0x05002000
[ 21.983802] snd_hda_intel 0000:0e:00.1: bound 0000:0e:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
[ 21.984429] [drm] kiq ring mec 2 pipe 1 q 0
[ 21.987344] kfd kfd: amdgpu: Allocated 3969056 bytes on gart
[ 21.987352] kfd kfd: amdgpu: Total number of KFD nodes to be created: 1
[ 21.987443] amdgpu: Virtual CRAT table created for GPU
[ 21.987526] amdgpu: Topology: Add dGPU node [0x164e:0x1002]
[ 21.987527] kfd kfd: amdgpu: added device 1002:164e
[ 21.987535] amdgpu 0000:0e:00.0: amdgpu: SE 1, SH per SE 1, CU per SH 2, active_cu_number 2
[ 21.987538] amdgpu 0000:0e:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
[ 21.987539] amdgpu 0000:0e:00.0: amdgpu: ring gfx_0.1.0 uses VM inv eng 1 on hub 0
[ 21.987540] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 4 on hub 0
[ 21.987541] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 5 on hub 0
[ 21.987542] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
[ 21.987543] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
[ 21.987545] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
[ 21.987546] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
[ 21.987547] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
[ 21.987548] amdgpu 0000:0e:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
[ 21.987549] amdgpu 0000:0e:00.0: amdgpu: ring kiq_0.2.1.0 uses VM inv eng 12 on hub 0
[ 21.987550] amdgpu 0000:0e:00.0: amdgpu: ring sdma0 uses VM inv eng 13 on hub 0
[ 21.987551] amdgpu 0000:0e:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 8
[ 21.987552] amdgpu 0000:0e:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 8
[ 21.987553] amdgpu 0000:0e:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 8
[ 21.987554] amdgpu 0000:0e:00.0: amdgpu: ring jpeg_dec uses VM inv eng 5 on hub 8
[ 21.988370] amdgpu 0000:0e:00.0: amdgpu: Runtime PM not available
[ 21.988505] [drm] Initialized amdgpu 3.61.0 for 0000:0e:00.0 on minor 0
[ 21.990637] amdgpu 0000:0e:00.0: [drm] Cannot find any crtc or sizes
[ 21.990648] [drm] pre_validate_dsc:1622 MST_DSC dsc precompute is not needed
[ 45.661991] dca service started, version 1.12.1
[ 45.663848] ioatdma: Intel(R) QuickData Technology Driver 5.00
[ 45.666263] NTB Resource Split driver, version 1
[ 45.667719] Software Queue-Pair Transport over NTB, version 4
[ 45.680480] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[ 46.084341] RTL8126A-internal 5Gbps PHY r8169-0-b00:00: attached PHY driver (mii_bus:phy_addr=r8169-0-b00:00, irq=MAC)
[ 47.076433] r8169 0000:0b:00.0 enp11s0: Link is Down
[ 49.941667] r8169 0000:0b:00.0 enp11s0: Link is Up - 1Gbps/Full - flow control off
[ 53.968967] audit: type=1400 audit(1764280256.865:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=2379 comm="apparmor_parser"
[ 53.969701] audit: type=1400 audit(1764280256.865:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=2377 comm="apparmor_parser"
[ 53.969704] audit: type=1400 audit(1764280256.865:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=2377 comm="apparmor_parser"
[ 53.969833] audit: type=1400 audit(1764280256.865:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="passt" pid=2381 comm="apparmor_parser"
[ 53.969940] audit: type=1400 audit(1764280256.865:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=2375 comm="apparmor_parser"
[ 53.978637] audit: type=1400 audit(1764280256.865:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="virt-aa-helper" pid=2383 comm="apparmor_parser"
[ 53.978639] audit: type=1400 audit(1764280256.865:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=2380 comm="apparmor_parser"
[ 53.978640] audit: type=1400 audit(1764280256.865:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=2380 comm="apparmor_parser"
[ 53.983884] audit: type=1400 audit(1764280256.865:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=2380 comm="apparmor_parser"
[ 53.983885] audit: type=1400 audit(1764280256.865:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd" pid=2386 comm="apparmor_parser"
[ 54.203582] nvme nvme0: using unchecked data buffer
[ 54.451697] NFSD: Using nfsdcld client tracking operations.
[ 54.451700] NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000)
Nov 28 08:50:24 truenas kernel: kvm_amd: x2AVIC enabled
Nov 28 08:50:24 truenas kernel: kvm_amd: Virtual GIF supported
Nov 28 08:50:24 truenas kernel: kvm_amd: Virtual NMI enabled
Nov 28 08:50:24 truenas kernel: MCE: In-kernel MCE decoding enabled.
Nov 28 08:50:24 truenas kernel: [drm] amdgpu kernel modesetting enabled.
Nov 28 08:50:24 truenas kernel: amdgpu: vga_switcheroo: detected switching method \_SB_.PCI0.GP17.VGA_.ATPX handle
Nov 28 08:50:24 truenas kernel: amdgpu: ATPX version 1, functions 0x00000000
Nov 28 08:50:24 truenas kernel: amdgpu: Virtual CRAT table created for CPU
Nov 28 08:50:24 truenas kernel: amdgpu: Topology: Add CPU node
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: enabling device (0006 -> 0007)
Nov 28 08:50:24 truenas kernel: [drm] initializing kernel modesetting (IP DISCOVERY 0x1002:0x164E 0x1462:0x7E56 0xC5).
Nov 28 08:50:24 truenas kernel: [drm] register mmio base: 0xF6500000
Nov 28 08:50:24 truenas kernel: [drm] register mmio size: 524288
Nov 28 08:50:24 truenas kernel: [drm] add ip block number 0 <nv_common>
Nov 28 08:50:24 truenas kernel: [drm] add ip block number 1 <gmc_v10_0>
Nov 28 08:50:24 truenas kernel: [drm] add ip block number 2 <navi10_ih>
Nov 28 08:50:24 truenas kernel: [drm] add ip block number 3 <psp>
Nov 28 08:50:24 truenas kernel: [drm] add ip block number 4 <smu>
Nov 28 08:50:24 truenas kernel: [drm] add ip block number 5 <dm>
Nov 28 08:50:24 truenas kernel: [drm] add ip block number 6 <gfx_v10_0>
Nov 28 08:50:24 truenas kernel: [drm] add ip block number 7 <sdma_v5_2>
Nov 28 08:50:24 truenas kernel: [drm] add ip block number 8 <vcn_v3_0>
Nov 28 08:50:24 truenas kernel: [drm] add ip block number 9 <jpeg_v3_0>
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: Fetched VBIOS from VFCT
Nov 28 08:50:24 truenas kernel: amdgpu: ATOM BIOS: 102-RAPHAEL-008
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: Trusted Memory Zone (TMZ) feature disabled as experimental (default)
Nov 28 08:50:24 truenas kernel: [drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: VRAM: 512M 0x000000F400000000 - 0x000000F41FFFFFFF (512M used)
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
Nov 28 08:50:24 truenas kernel: [drm] Detected VRAM RAM=512M, BAR=512M
Nov 28 08:50:24 truenas kernel: [drm] RAM width 128bits DDR5
Nov 28 08:50:24 truenas kernel: [drm] amdgpu: 512M of VRAM memory ready
Nov 28 08:50:24 truenas kernel: [drm] amdgpu: 31677M of GTT memory ready.
Nov 28 08:50:24 truenas kernel: [drm] GART: num cpu pages 262144, num gpu pages 262144
Nov 28 08:50:24 truenas kernel: [drm] PCIE GART of 1024M enabled (table at 0x000000F41FC00000).
Nov 28 08:50:24 truenas kernel: [drm] Loading DMUB firmware via PSP: version=0x05002000
Nov 28 08:50:24 truenas kernel: [drm] use_doorbell being set to: [true]
Nov 28 08:50:24 truenas kernel: [drm] Found VCN firmware Version ENC: 1.33 DEC: 4 VEP: 0 Revision: 6
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: reserve 0xa00000 from 0xf41e000000 for PSP TMR
Nov 28 08:50:24 truenas kernel: intel_rapl_common: Found RAPL domain package
Nov 28 08:50:24 truenas kernel: intel_rapl_common: Found RAPL domain core
Nov 28 08:50:24 truenas kernel: amd_atl: AMD Address Translation Library initialized
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: RAS: optional ras ta ucode is not available
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: RAP: optional rap ta ucode is not available
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: SMU is initialized successfully!
Nov 28 08:50:24 truenas kernel: [drm] Seamless boot condition check passed
Nov 28 08:50:24 truenas kernel: [drm] Display Core v3.2.301 initialized on DCN 3.1.5
Nov 28 08:50:24 truenas kernel: [drm] DP-HDMI FRL PCON supported
Nov 28 08:50:24 truenas kernel: [drm] DMUB hardware initialized: version=0x05002000
Nov 28 08:50:24 truenas kernel: snd_hda_intel 0000:0e:00.1: bound 0000:0e:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
Nov 28 08:50:24 truenas kernel: [drm] kiq ring mec 2 pipe 1 q 0
Nov 28 08:50:24 truenas kernel: kfd kfd: amdgpu: Allocated 3969056 bytes on gart
Nov 28 08:50:24 truenas kernel: kfd kfd: amdgpu: Total number of KFD nodes to be created: 1
Nov 28 08:50:24 truenas kernel: amdgpu: Virtual CRAT table created for GPU
Nov 28 08:50:24 truenas kernel: amdgpu: Topology: Add dGPU node [0x164e:0x1002]
Nov 28 08:50:24 truenas kernel: kfd kfd: amdgpu: added device 1002:164e
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: SE 1, SH per SE 1, CU per SH 2, active_cu_number 2
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring gfx_0.1.0 uses VM inv eng 1 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 4 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 5 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring kiq_0.2.1.0 uses VM inv eng 12 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring sdma0 uses VM inv eng 13 on hub 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 8
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 8
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 8
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring jpeg_dec uses VM inv eng 5 on hub 8
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: Runtime PM not available
Nov 28 08:50:24 truenas kernel: [drm] Initialized amdgpu 3.61.0 for 0000:0e:00.0 on minor 0
Nov 28 08:50:24 truenas kernel: amdgpu 0000:0e:00.0: [drm] Cannot find any crtc or sizes
Nov 28 08:50:24 truenas kernel: [drm] pre_validate_dsc:1622 MST_DSC dsc precompute is not needed
Nov 28 08:50:48 truenas kernel: dca service started, version 1.12.1
Nov 28 08:50:48 truenas kernel: ioatdma: Intel(R) QuickData Technology Driver 5.00
Nov 28 08:50:48 truenas kernel: NTB Resource Split driver, version 1
Nov 28 08:50:48 truenas kernel: Software Queue-Pair Transport over NTB, version 4
Nov 28 08:50:48 truenas kernel: NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
Nov 28 08:50:48 truenas kernel: RTL8126A-internal 5Gbps PHY r8169-0-b00:00: attached PHY driver (mii_bus:phy_addr=r8169-0-b00:00, irq=MAC)
Nov 28 08:50:49 truenas kernel: r8169 0000:0b:00.0 enp11s0: Link is Down
Nov 28 08:50:52 truenas kernel: r8169 0000:0b:00.0 enp11s0: Link is Up - 1Gbps/Full - flow control off
Nov 28 08:50:56 truenas kernel: audit: type=1400 audit(1764280256.865:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=2379 comm="apparmor_parser"
Nov 28 08:50:56 truenas kernel: audit: type=1400 audit(1764280256.865:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=2377 comm="apparmor_parser"
Nov 28 08:50:56 truenas kernel: audit: type=1400 audit(1764280256.865:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=2377 comm="apparmor_parser"
Nov 28 08:50:56 truenas kernel: audit: type=1400 audit(1764280256.865:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="passt" pid=2381 comm="apparmor_parser"
Nov 28 08:50:56 truenas kernel: audit: type=1400 audit(1764280256.865:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=2375 comm="apparmor_parser"
Nov 28 08:50:56 truenas kernel: audit: type=1400 audit(1764280256.865:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="virt-aa-helper" pid=2383 comm="apparmor_parser"
Nov 28 08:50:56 truenas kernel: audit: type=1400 audit(1764280256.865:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=2380 comm="apparmor_parser"
Nov 28 08:50:56 truenas kernel: audit: type=1400 audit(1764280256.865:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=2380 comm="apparmor_parser"
Nov 28 08:50:56 truenas kernel: audit: type=1400 audit(1764280256.865:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=2380 comm="apparmor_parser"
Nov 28 08:50:56 truenas kernel: audit: type=1400 audit(1764280256.865:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd" pid=2386 comm="apparmor_parser"
Nov 28 08:50:57 truenas kernel: nvme nvme0: using unchecked data buffer
Nov 28 08:50:57 truenas kernel: NFSD: Using nfsdcld client tracking operations.
Nov 28 08:50:57 truenas kernel: NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000)
Nov 28 08:50:58 truenas netdata[3543]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
truenas_ pts/0 Fri Nov 28 09:24 still logged in
runlevel (to lvl 3) 6.12.33-producti Fri Nov 28 08:50 still running
reboot system boot 6.12.33-producti Fri Nov 28 08:50 still running
shutdown system down 6.12.33-producti Fri Nov 28 08:49 - 08:50 (00:00)
Thorin pts/1 Fri Nov 28 08:46 - down (00:03)
Thorin pts/0 Fri Nov 28 08:44 - down (00:04)
runlevel (to lvl 3) 6.12.33-producti Fri Nov 28 07:22 - 08:49 (01:27)
reboot system boot 6.12.33-producti Fri Nov 28 07:21 - 08:49 (01:28)
root wsu/10 tn-mw.c2813656-8 Fri Nov 28 03:45 - crash (03:36)
root wsu/9 tn-mw.377f0d27-b Fri Nov 28 03:45 - crash (03:36)
Thorin wss/6 tn-mw.0cb467ee-1 Thu Nov 27 23:00 - crash (08:20)
Thorin wss/5 tn-mw.0cb467ee-1 Thu Nov 27 21:23 - crash (09:58)
runlevel (to lvl 3) 6.12.33-producti Thu Nov 27 10:05 - 07:22 (21:16)
reboot system boot 6.12.33-producti Thu Nov 27 10:05 - 08:49 (22:44)
runlevel (to lvl 3) 6.12.33-producti Thu Nov 27 09:45 - 10:05 (00:19)
reboot system boot 6.12.33-producti Thu Nov 27 09:45 - 08:49 (23:04)
runlevel (to lvl 3) 6.12.33-producti Thu Nov 27 09:39 - 09:45 (00:06)
reboot system boot 6.12.33-producti Thu Nov 27 09:39 - 08:49 (23:10)
runlevel (to lvl 3) 6.12.33-producti Thu Nov 27 09:34 - 09:39 (00:05)
reboot system boot 6.12.33-producti Thu Nov 27 09:34 - 08:49 (23:15)
runlevel (to lvl 3) 6.12.33-producti Thu Nov 27 09:27 - 09:34 (00:06)
reboot system boot 6.12.33-producti Thu Nov 27 09:27 - 08:49 (23:22)
shutdown system down 6.12.33-producti Thu Nov 27 09:05 - 09:27 (00:21)
Thorin wss/12 tn-mw.a0d164f8-3 Thu Nov 27 09:05 - down (00:00)
Thorin wss/11 tn-mw.a0d164f8-3 Thu Nov 27 09:03 - down (00:02)
Thorin wss/10 tn-mw.bd688302-a Thu Nov 27 08:34 - down (00:31)
Thorin wss/9 tn-mw.a0d164f8-3 Thu Nov 27 08:06 - down (00:59)
root wsu/8 tn-mw.7be61a7e-c Fri Nov 28 03:30 - down (-18:24)
runlevel (to lvl 3) 6.12.33-producti Wed Nov 26 12:04 - 09:05 (21:01)
reboot system boot 6.12.33-producti Wed Nov 26 12:03 - 09:05 (21:02)
root wsu/3 tn-mw.c9f0db56-7 Tue Nov 25 22:30 - crash (13:33)
runlevel (to lvl 3) 6.12.33-producti Tue Nov 25 14:53 - 12:04 (21:10)
reboot system boot 6.12.33-producti Tue Nov 25 14:53 - 09:05 (1+18:12)
shutdown system down 6.12.33-producti Tue Nov 25 13:23 - 14:53 (01:29)
Thorin wss/4 tn-mw.0cb467ee-1 Thu Nov 27 17:24 - down (-2+04:00)
runlevel (to lvl 3) 6.12.33-producti Tue Nov 25 08:46 - 13:23 (04:37)
reboot system boot 6.12.33-producti Tue Nov 25 08:45 - 13:23 (04:37)
root wsu/7 tn-mw.15f0017e-4 Fri Nov 28 02:38 - crash (-2+17:52)
root wsu/6 tn-mw.87513e16-2 Thu Nov 27 03:30 - crash (-1+18:44)
root wsu/5 tn-mw.ef82012d-5 Thu Nov 27 01:53 - crash (-1+17:07)
Nov 28 07:21:25 truenas systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
Nov 28 07:21:25 truenas systemd[1]: Starting modprobe@fuse.service - Load Kernel Module fuse...
Nov 28 07:21:25 truenas kernel: pstore: Using crash dump compression: deflate
Nov 28 07:21:25 truenas systemd[1]: Starting modprobe@loop.service - Load Kernel Module loop...
Nov 28 07:21:25 truenas systemd[1]: Starting systemd-journald.service - Journal Service...
Nov 28 07:21:25 truenas systemd[1]: systemd-pcrmachine.service - TPM2 PCR Machine ID Measurement was skipped because of an unmet condition check (ConditionPathExists=/sys/firmware/efi/efivars/StubPcrKernelImage-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f).
Nov 28 07:21:25 truenas kernel: RPC: Registered named UNIX socket transport module.
Nov 28 07:21:25 truenas kernel: RPC: Registered udp transport module.
Nov 28 07:21:25 truenas kernel: RPC: Registered tcp transport module.
Nov 28 07:21:25 truenas kernel: RPC: Registered tcp-with-tls transport module.
Nov 28 07:21:25 truenas kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Nov 28 07:21:25 truenas systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
Nov 28 07:21:25 truenas kernel: pstore: Registered efi_pstore as persistent store backend
Nov 28 07:21:25 truenas kernel: loop: module loaded
Nov 28 07:21:25 truenas systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
Nov 28 07:21:25 truenas systemd[1]: Starting zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev...
Nov 28 07:21:25 truenas systemd-journald[777]: Collecting audit messages is disabled.
Nov 28 07:21:25 truenas systemd[1]: Started dbus.service - D-Bus System Message Bus.
Nov 28 07:21:25 truenas kernel: ACPI: bus type drm_connector registered
Nov 28 07:21:25 truenas systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
Nov 28 07:21:25 truenas systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
Nov 28 07:21:25 truenas systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
Nov 28 07:21:25 truenas systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
Nov 28 07:21:25 truenas systemd[1]: Mounted tmp.mount - Temporary Directory (/tmp).
Nov 28 07:21:25 truenas systemd[1]: Finished keyboard-setup.service - Set the console keyboard layout.
Nov 28 07:21:25 truenas systemd[1]: Finished kmod-static-nodes.service - Create List of Static Device Nodes.
Nov 28 07:21:25 truenas systemd[1]: modprobe@configfs.service: Deactivated successfully.
Nov 28 07:21:25 truenas systemd[1]: Finished modprobe@configfs.service - Load Kernel Module configfs.
Nov 28 07:21:25 truenas systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
Nov 28 07:21:25 truenas systemd[1]: Finished modprobe@dm_mod.service - Load Kernel Module dm_mod.
Nov 28 07:21:25 truenas systemd[1]: modprobe@drm.service: Deactivated successfully.
Nov 28 07:21:25 truenas systemd[1]: Finished modprobe@drm.service - Load Kernel Module drm.
Nov 28 07:21:25 truenas systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
Nov 28 07:21:25 truenas systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
Nov 28 07:21:25 truenas systemd[1]: modprobe@fuse.service: Deactivated successfully.
Nov 28 07:21:25 truenas systemd[1]: Finished modprobe@fuse.service - Load Kernel Module fuse.
Nov 28 07:21:25 truenas systemd[1]: modprobe@loop.service: Deactivated successfully.
Nov 28 07:21:25 truenas systemd[1]: Finished modprobe@loop.service - Load Kernel Module loop.
Nov 28 07:21:25 truenas systemd[1]: Finished systemd-remount-fs.service - Remount Root and Kernel File Systems.
Nov 28 07:21:25 truenas systemd[1]: Mounting sys-fs-fuse-connections.mount - FUSE Control File System...
Nov 28 07:21:25 truenas systemd[1]: Mounting sys-kernel-config.mount - Kernel Configuration File System...
Nov 28 07:21:25 truenas systemd[1]: systemd-hwdb-update.service - Rebuild Hardware Database was skipped because of an unmet condition check (ConditionNeedsUpdate=/etc).
Nov 28 07:21:25 truenas systemd[1]: Starting systemd-pstore.service - Platform Persistent Storage Archival...
Nov 28 07:21:25 truenas systemd[1]: Starting systemd-random-seed.service - Load/Save OS Random Seed...
Nov 28 07:21:25 truenas systemd[1]: systemd-repart.service - Repartition Root Disk was skipped because no trigger condition checks were met.
Nov 28 07:21:25 truenas systemd[1]: Starting systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully...
Nov 28 07:21:25 truenas systemd[1]: Mounted proc-fs-nfsd.mount - NFSD configuration filesystem.
Nov 28 07:21:25 truenas systemd[1]: Started systemd-journald.service - Journal Service.
Nov 28 07:21:25 truenas systemd-journald[777]: Received client request to flush runtime journal.
Nov 28 07:21:25 truenas systemd-journald[777]: /var/log/journal/744081bbc1f94d52a03531b323b8cbd4/system.journal: Journal file uses a different sequence number ID, rotating.
Nov 28 07:21:25 truenas systemd-journald[777]: Rotating system journal.
Nov 28 07:21:25 truenas kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input6
Nov 28 07:21:25 truenas kernel: ACPI: button: Power Button [PWRF]
Nov 28 07:21:25 truenas kernel: ccp 0000:0e:00.2: enabling device (0000 -> 0002)
Nov 28 07:21:25 truenas kernel: input: PC Speaker as /devices/platform/pcspkr/input/input7
Nov 28 07:21:25 truenas kernel: ccp 0000:0e:00.2: tee enabled
Nov 28 07:21:25 truenas kernel: ccp 0000:0e:00.2: psp enabled
Nov 28 07:21:25 truenas kernel: sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver
Nov 28 07:21:25 truenas kernel: sp5100-tco sp5100-tco: Using 0xfeb00000 for watchdog MMIO address
Nov 28 07:21:25 truenas kernel: sp5100-tco sp5100-tco: initialized. heartbeat=60 sec (nowayout=0)
Nov 28 07:21:25 truenas kernel: Error: Driver 'pcspkr' is already registered, aborting...
Nov 28 07:21:26 truenas kernel: RAPL PMU: API unit is 2^-32 Joules, 1 fixed counters, 163840 ms ovfl timer
Nov 28 07:21:26 truenas kernel: RAPL PMU: hw unit of domain package 2^-16 Joules
Nov 28 07:21:26 truenas kernel: cryptd: max_cpu_qlen set to 1000
Nov 28 07:21:26 truenas kernel: snd_hda_intel 0000:01:00.1: enabling device (0000 -> 0002)
Nov 28 07:21:26 truenas kernel: snd_hda_intel 0000:01:00.1: Disabling MSI
Nov 28 07:21:26 truenas kernel: snd_hda_intel 0000:01:00.1: Handle vga_switcheroo audio client
Nov 28 07:21:26 truenas kernel: snd_hda_intel 0000:0e:00.1: enabling device (0000 -> 0002)
Nov 28 07:21:26 truenas kernel: snd_hda_intel 0000:0e:00.1: Handle vga_switcheroo audio client
Nov 28 07:21:26 truenas kernel: AES CTR mode by8 optimization enabled
Nov 28 07:21:26 truenas kernel: input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.1/0000:01:00.1/sound/card0/input12
Nov 28 07:21:26 truenas kernel: input: HD-Audio Generic HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:08.1/0000:0e:00.1/sound/card1/input8
Nov 28 07:21:26 truenas kernel: input: HD-Audio Generic HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:08.1/0000:0e:00.1/sound/card1/input9
Nov 28 07:21:26 truenas kernel: input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.1/0000:01:00.1/sound/card0/input13
Nov 28 07:21:26 truenas kernel: input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.1/0000:01:00.1/sound/card0/input14
Nov 28 07:21:26 truenas kernel: input: HD-Audio Generic HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:08.1/0000:0e:00.1/sound/card1/input10
Nov 28 07:21:26 truenas kernel: input: HD-Audio Generic HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:08.1/0000:0e:00.1/sound/card1/input11
Nov 28 07:21:26 truenas kernel: input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.1/0000:01:00.1/sound/card0/input15
Nov 28 07:21:26 truenas kernel: kvm_amd: TSC scaling supported
Nov 28 07:21:26 truenas kernel: kvm_amd: Nested Virtualization enabled
Nov 28 07:21:26 truenas kernel: kvm_amd: Nested Paging enabled
Nov 28 07:21:26 truenas kernel: kvm_amd: LBR virtualization supported
Nov 28 07:21:26 truenas kernel: kvm_amd: AVIC enabled
Nov 28 07:21:26 truenas kernel: kvm_amd: x2AVIC enabled
Nov 28 07:21:26 truenas kernel: kvm_amd: Virtual GIF supported
Nov 28 07:21:26 truenas kernel: kvm_amd: Virtual NMI enabled
Nov 28 07:21:26 truenas kernel: MCE: In-kernel MCE decoding enabled.
Nov 28 07:21:27 truenas kernel: [drm] amdgpu kernel modesetting enabled.
Nov 28 07:21:27 truenas kernel: amdgpu: vga_switcheroo: detected switching method \_SB_.PCI0.GP17.VGA_.ATPX handle
Nov 28 07:21:27 truenas kernel: amdgpu: ATPX version 1, functions 0x00000000
Nov 28 07:21:27 truenas kernel: amdgpu: Virtual CRAT table created for CPU
Nov 28 07:21:27 truenas kernel: amdgpu: Topology: Add CPU node
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: enabling device (0006 -> 0007)
Nov 28 07:21:27 truenas kernel: [drm] initializing kernel modesetting (IP DISCOVERY 0x1002:0x164E 0x1462:0x7E56 0xC5).
Nov 28 07:21:27 truenas kernel: [drm] register mmio base: 0xF6500000
Nov 28 07:21:27 truenas kernel: [drm] register mmio size: 524288
Nov 28 07:21:27 truenas kernel: [drm] add ip block number 0 <nv_common>
Nov 28 07:21:27 truenas kernel: [drm] add ip block number 1 <gmc_v10_0>
Nov 28 07:21:27 truenas kernel: [drm] add ip block number 2 <navi10_ih>
Nov 28 07:21:27 truenas kernel: [drm] add ip block number 3 <psp>
Nov 28 07:21:27 truenas kernel: [drm] add ip block number 4 <smu>
Nov 28 07:21:27 truenas kernel: [drm] add ip block number 5 <dm>
Nov 28 07:21:27 truenas kernel: [drm] add ip block number 6 <gfx_v10_0>
Nov 28 07:21:27 truenas kernel: [drm] add ip block number 7 <sdma_v5_2>
Nov 28 07:21:27 truenas kernel: [drm] add ip block number 8 <vcn_v3_0>
Nov 28 07:21:27 truenas kernel: [drm] add ip block number 9 <jpeg_v3_0>
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: Fetched VBIOS from VFCT
Nov 28 07:21:27 truenas kernel: amdgpu: ATOM BIOS: 102-RAPHAEL-008
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: Trusted Memory Zone (TMZ) feature disabled as experimental (default)
Nov 28 07:21:27 truenas kernel: [drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: VRAM: 512M 0x000000F400000000 - 0x000000F41FFFFFFF (512M used)
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
Nov 28 07:21:27 truenas kernel: [drm] Detected VRAM RAM=512M, BAR=512M
Nov 28 07:21:27 truenas kernel: [drm] RAM width 128bits DDR5
Nov 28 07:21:27 truenas kernel: [drm] amdgpu: 512M of VRAM memory ready
Nov 28 07:21:27 truenas kernel: [drm] amdgpu: 31677M of GTT memory ready.
Nov 28 07:21:27 truenas kernel: [drm] GART: num cpu pages 262144, num gpu pages 262144
Nov 28 07:21:27 truenas kernel: [drm] PCIE GART of 1024M enabled (table at 0x000000F41FC00000).
Nov 28 07:21:27 truenas kernel: [drm] Loading DMUB firmware via PSP: version=0x05002000
Nov 28 07:21:27 truenas kernel: [drm] use_doorbell being set to: [true]
Nov 28 07:21:27 truenas kernel: [drm] Found VCN firmware Version ENC: 1.33 DEC: 4 VEP: 0 Revision: 6
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: reserve 0xa00000 from 0xf41e000000 for PSP TMR
Nov 28 07:21:27 truenas kernel: intel_rapl_common: Found RAPL domain package
Nov 28 07:21:27 truenas kernel: intel_rapl_common: Found RAPL domain core
Nov 28 07:21:27 truenas kernel: amd_atl: AMD Address Translation Library initialized
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: RAS: optional ras ta ucode is not available
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: RAP: optional rap ta ucode is not available
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: SMU is initialized successfully!
Nov 28 07:21:27 truenas kernel: [drm] Seamless boot condition check passed
Nov 28 07:21:27 truenas kernel: [drm] Display Core v3.2.301 initialized on DCN 3.1.5
Nov 28 07:21:27 truenas kernel: [drm] DP-HDMI FRL PCON supported
Nov 28 07:21:27 truenas kernel: [drm] DMUB hardware initialized: version=0x05002000
Nov 28 07:21:27 truenas kernel: snd_hda_intel 0000:0e:00.1: bound 0000:0e:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
Nov 28 07:21:27 truenas kernel: [drm] kiq ring mec 2 pipe 1 q 0
Nov 28 07:21:27 truenas kernel: kfd kfd: amdgpu: Allocated 3969056 bytes on gart
Nov 28 07:21:27 truenas kernel: kfd kfd: amdgpu: Total number of KFD nodes to be created: 1
Nov 28 07:21:27 truenas kernel: amdgpu: Virtual CRAT table created for GPU
Nov 28 07:21:27 truenas kernel: amdgpu: Topology: Add dGPU node [0x164e:0x1002]
Nov 28 07:21:27 truenas kernel: kfd kfd: amdgpu: added device 1002:164e
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: SE 1, SH per SE 1, CU per SH 2, active_cu_number 2
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring gfx_0.1.0 uses VM inv eng 1 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 4 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 5 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring kiq_0.2.1.0 uses VM inv eng 12 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring sdma0 uses VM inv eng 13 on hub 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 8
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 8
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 8
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: ring jpeg_dec uses VM inv eng 5 on hub 8
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: amdgpu: Runtime PM not available
Nov 28 07:21:27 truenas kernel: [drm] Initialized amdgpu 3.61.0 for 0000:0e:00.0 on minor 0
Nov 28 07:21:27 truenas kernel: amdgpu 0000:0e:00.0: [drm] Cannot find any crtc or sizes
Nov 28 07:21:27 truenas kernel: [drm] pre_validate_dsc:1622 MST_DSC dsc precompute is not needed
Nov 28 07:21:50 truenas kernel: dca service started, version 1.12.1
Nov 28 07:21:50 truenas kernel: ioatdma: Intel(R) QuickData Technology Driver 5.00
Nov 28 07:21:50 truenas kernel: NTB Resource Split driver, version 1
Nov 28 07:21:50 truenas kernel: Software Queue-Pair Transport over NTB, version 4
Nov 28 07:21:50 truenas kernel: NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
Nov 28 07:21:51 truenas kernel: RTL8126A-internal 5Gbps PHY r8169-0-b00:00: attached PHY driver (mii_bus:phy_addr=r8169-0-b00:00, irq=MAC)
Nov 28 07:21:52 truenas kernel: r8169 0000:0b:00.0 enp11s0: Link is Down
Nov 28 07:21:54 truenas kernel: r8169 0000:0b:00.0 enp11s0: Link is Up - 1Gbps/Full - flow control off
Nov 28 07:22:05 truenas kernel: audit: type=1400 audit(1764274925.209:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=2382 comm="apparmor_parser"
Nov 28 07:22:05 truenas kernel: audit: type=1400 audit(1764274925.209:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="passt" pid=2384 comm="apparmor_parser"
Nov 28 07:22:05 truenas kernel: audit: type=1400 audit(1764274925.209:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=2378 comm="apparmor_parser"
Nov 28 07:22:05 truenas kernel: audit: type=1400 audit(1764274925.209:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="virt-aa-helper" pid=2387 comm="apparmor_parser"
Nov 28 07:22:05 truenas kernel: audit: type=1400 audit(1764274925.209:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=2380 comm="apparmor_parser"
Nov 28 07:22:05 truenas kernel: audit: type=1400 audit(1764274925.209:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=2380 comm="apparmor_parser"
Nov 28 07:22:05 truenas kernel: audit: type=1400 audit(1764274925.209:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd" pid=2389 comm="apparmor_parser"
Nov 28 07:22:05 truenas kernel: audit: type=1400 audit(1764274925.209:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd//qemu_bridge_helper" pid=2389 comm="apparmor_parser"
Nov 28 07:22:05 truenas kernel: audit: type=1400 audit(1764274925.209:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=2383 comm="apparmor_parser"
Nov 28 07:22:05 truenas kernel: audit: type=1400 audit(1764274925.209:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=2383 comm="apparmor_parser"
Nov 28 07:22:05 truenas kernel: nvme nvme0: using unchecked data buffer
Nov 28 07:22:05 truenas kernel: NFSD: Using nfsdcld client tracking operations.
Nov 28 07:22:05 truenas kernel: NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000)
Nov 28 08:49:38 truenas kernel: show_signal_msg: 11 callbacks suppressed
Nov 28 08:49:38 truenas kernel: zsh[6412]: segfault at 5565aa4061fa ip 00005560ec5a0fa9 sp 00007ffcf6425240 error 4 in zsh[62fa9,5560ec555000+9a000] likely on CPU 4 (core 4, socket 0)
Nov 28 08:49:38 truenas kernel: Code: 74 0f 48 8b 57 08 48 89 50 08 c3 66 0f 1f 44 00 00 48 8b 47 08 48 89 42 08 c3 0f 1f 80 00 00 00 00 53 48 8b 07 48 85 c0 74 37 <48> 8b 10 48 8b 58 10 48 89 17 48 85 d2 74 18 48 89 7a 08 be 18 00
Nov 28 08:49:38 truenas kernel: audit: type=1305 audit(1764280178.784:101): op=set audit_pid=0 old=2401 auid=4294967295 ses=4294967295 subj=unconfined res=1
Nov 28 08:49:38 truenas kernel: audit: type=1305 audit(1764280178.788:102): op=remove_rule dir="/data/" key="escalation" list=4 res=1
Nov 28 08:49:38 truenas kernel: audit: type=1334 audit(1764280178.816:103): prog-id=24 op=UNLOAD
Nov 28 08:49:38 truenas kernel: audit: type=1334 audit(1764280178.816:104): prog-id=23 op=UNLOAD
Nov 28 08:49:38 truenas kernel: audit: type=1334 audit(1764280178.820:105): prog-id=31 op=UNLOAD
Nov 28 08:49:38 truenas kernel: audit: type=1334 audit(1764280178.820:106): prog-id=30 op=UNLOAD
Nov 28 08:49:38 truenas kernel: audit: type=1334 audit(1764280178.820:107): prog-id=27 op=UNLOAD
Nov 28 08:49:38 truenas kernel: audit: type=1334 audit(1764280178.820:108): prog-id=26 op=UNLOAD
Nov 28 08:49:38 truenas kernel: audit: type=1334 audit(1764280178.820:109): prog-id=25 op=UNLOAD
Nov 28 08:49:38 truenas kernel: watchdog: watchdog0: watchdog did not stop!
Nov 28 08:49:38 truenas systemd-shutdown[1]: Using hardware watchdog 'SP5100 TCO timer', version 0, device /dev/watchdog0
Nov 28 08:49:38 truenas systemd-shutdown[1]: Watchdog running with a hardware timeout of 10min.
Nov 28 08:49:38 truenas systemd-shutdown[1]: Syncing filesystems and block devices.
Nov 28 08:49:38 truenas systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Nov 28 08:49:38 truenas systemd-journald[777]: Received SIGTERM from PID 1 (systemd-shutdow).
I do see AM5 / SP5100 related lines earlier in boot, and the kernel watchdog thread is present:
Nov 28 08:49:38 truenas kernel: zsh[6412]: segfault ...
...
Nov 28 08:49:38 truenas kernel: watchdog: watchdog0: watchdog did not stop!
Nov 28 08:49:38 truenas systemd-shutdown[1]: Using hardware watchdog 'SP5100 TCO timer'
...
Nov 28 08:49:38 truenas systemd-shutdown[1]: Sending SIGTERM to remaining processes...
bash
ps aux | grep -i watchdog | head
grep -i watchdog /etc/systemd/system.conf /etc/systemd/*.d/* 2>/dev/null || echo "no system.conf watchdog settings"
root 951 0.0 0.0 0 0 ? S 08:50 0:00 [watchdogd]
truenas+ 5458 0.0 0.0 3748 1456 pts/0 S+ 09:36 0:00 grep -i watchdog
/etc/systemd/system.conf:#RuntimeWatchdogSec=off
/etc/systemd/system.conf:#RuntimeWatchdogPreSec=off
/etc/systemd/system.conf:#RuntimeWatchdogPreGovernor=
/etc/systemd/system.conf:#RebootWatchdogSec=10min
/etc/systemd/system.conf:#KExecWatchdogSec=off
/etc/systemd/system.conf:#WatchdogDevice=
Claude says ( please don’t hate me)
This is the crucial bit:
text
Copy
root 951 0.0 0.0 0 0 ? S 08:50 0:00 [watchdogd]
That [watchdogd] is the kernel watchdog thread for your SP5100 TCO hardware watchdog. Combined with:
text
Copy
sp5100-tco ... WatchDog Timer Driver
watchdog: watchdog0: watchdog did not stop!
systemd-shutdown[1]: Using hardware watchdog 'SP5100 TCO timer'
the picture is:
The hardware watchdog is armed.
Something (kernel / watchdogd / systemd) decides the system is “not healthy” or reaches a watchdog condition.
That results in a clean shutdown (systemd‑shutdown runs, filesystems sync, processes SIGTERM’d).
This is not a simple cron job or user typing shutdown.
So now the focus is:
Minimize or disable the hardware watchdog.
See if the reboots stop.
If they do, we know the watchdog was being tripped and we can decide whether to leave it off, increase its timeout, or investigate underlying hangs.
What I’ve checked so far
BIOS: I went through the MSI PRO B650-P WIFI BIOS (using the English manual) and found no explicit “watchdog” toggle for SP5100 or other hardware watchdogs.
Usual power/ACPI/thermal options look normal.
Obvious cron/UPS/middleware triggers:
No cron jobs that call shutdown/poweroff.
No UPS service configured to shut the system down.
I’m shutting down via the UI normally when I do it manually.
What I’m not seeing
No obvious SSD/NVMe failing signs (no repeated I/O errors in logs before shutdown). My questions
Could the kernel watchdog ([watchdogd] – possibly tied to the SP5100 on this board) be initiating these clean shutdowns on SCALE even though systemd’s watchdog is disabled? Is there a recommended way on SCALE to temporarily disable or “de‑arm” hardware watchdog behavior, specifically for debugging? e.g. kernel parameters / sysctls / known tunables that are safe on SCALE. Has anyone seen similar behavior on AMD B650 / SP5100 platforms (especially MSI boards), where TrueNAS performs a clean shutdown seemingly on its own? Is there anything specific in SCALE (middleware, UPS service defaults, ACPI events) that could be triggering midclt call system.shutdown or a systemd target under certain conditions, without obvious log entries?
Before I fully “turn off” any watchdogs, I’m planning to:
Capture full journalctl -b -1 including a few minutes before shutdown. Capture zpool status and smartctl -a on all pools & boot SSD. If safe/approved, try a temporary kernel watchdog tunable (e.g. disable or set to log‑only) and see whether the shutdowns stop.
Any help trying to get to the bottom of this would be great. I have never had any issues with Truenas in the past, I was hoping this new server would be a easy to get started as my last ones.
Thanks!