Random reboots with new hardware

TrueNAS General
,
1

Hi, a few month ago I have upgraded my System with new hardware combined with my older Intel X710-DA2 10G NIC.
Since then I have recognized several reboots with no obvious reason.
The NIC was alrready installed in the old system and was running without any abnormalities. In the nwe system the card is connected to an x4 slot although it has a x8 pcb.

XMP-Profile for the RAM is disabled.
Power Supply Idle Control is set to Typical Current Idle
Power saving features should be disabled
undervolting the CPU is enabled with -20 on all cores.

In the log files I can’t identify a substantiation for the behavior.

debug log-file:

Apr 13 09:04:53 TrueNAS kernel: Monitor-Mwait will be used to enter C-1 state
Apr 13 09:04:53 TrueNAS kernel:   with arguments:
Apr 13 09:04:53 TrueNAS kernel:     /init
Apr 13 09:04:53 TrueNAS kernel:   with environment:
Apr 13 09:04:53 TrueNAS kernel:     HOME=/
Apr 13 09:04:53 TrueNAS kernel:     TERM=linux
Apr 13 09:04:53 TrueNAS kernel:     BOOT_IMAGE=/ROOT/24.10.2.1@/boot/vmlinuz-6.6.44-production+truenas
Apr 13 09:04:53 TrueNAS kernel:     zfsforce=1
Apr 13 09:04:53 TrueNAS kernel: libata version 3.00 loaded.
Apr 13 09:04:53 TrueNAS kernel: ahci 0000:0b:00.0: version 3.0
Apr 13 09:04:53 TrueNAS kernel: sd 2:0:0:0: [sdd] Mode Sense: 00 3a 00 00
Apr 13 09:04:53 TrueNAS kernel: sd 1:0:0:0: [sdc] Mode Sense: 00 3a 00 00
Apr 13 09:04:53 TrueNAS kernel: sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
Apr 13 09:04:53 TrueNAS kernel: sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Apr 15 06:48:01 TrueNAS kernel: e820: remove [mem 0xe0000000-0xefffffff] reserved
Apr 15 06:48:01 TrueNAS kernel: e820: remove [mem 0xf7000000-0xfedfffff] reserved
Apr 15 06:48:01 TrueNAS kernel: e820: remove [mem 0xfee01000-0xffffffff] reserved
Apr 15 06:48:01 TrueNAS kernel: e820: remove [mem 0x1080000000-0x10a01fffff] reserved
Apr 15 06:48:01 TrueNAS kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Apr 15 06:48:01 TrueNAS kernel: e820: remove [mem 0x000a0000-0x000fffff] usable
Apr 15 06:48:01 TrueNAS kernel: e820: update [mem 0x80000000-0xffffffff] usable ==> reserved
Apr 15 06:48:01 TrueNAS kernel: e820: update [mem 0x6765b000-0x6765bfff] usable ==> reserved
Apr 15 06:48:01 TrueNAS kernel: pcpu-alloc: s221184 r8192 d28672 u262144 alloc=1*2097152
Apr 15 06:48:01 TrueNAS kernel: pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 12 13 14 15 
Apr 15 06:48:01 TrueNAS kernel: pcpu-alloc: [0] 16 17 18 19 20 21 22 23 
Apr 15 06:48:01 TrueNAS kernel: PCI: pci_cache_line_size set to 64 bytes
Apr 15 06:48:01 TrueNAS kernel: e820: reserve RAM buffer [mem 0x09aff000-0x0bffffff]
Apr 15 06:48:01 TrueNAS kernel: e820: reserve RAM buffer [mem 0x0a200000-0x0bffffff]
Apr 15 06:48:01 TrueNAS kernel: e820: reserve RAM buffer [mem 0x0b000000-0x0bffffff]
Apr 15 06:48:01 TrueNAS kernel: e820: reserve RAM buffer [mem 0x6765b000-0x67ffffff]
Apr 15 06:48:01 TrueNAS kernel: e820: reserve RAM buffer [mem 0x6b0b2000-0x6bffffff]
Apr 15 06:48:01 TrueNAS kernel: e820: reserve RAM buffer [mem 0x79ff9000-0x7bffffff]
Apr 15 06:48:01 TrueNAS kernel: e820: reserve RAM buffer [mem 0x7a000000-0x7bffffff]
Apr 15 06:48:01 TrueNAS kernel: e820: reserve RAM buffer [mem 0x105de80000-0x105fffffff]
Apr 15 06:48:01 TrueNAS kernel: pnp 00:04: [dma 0 disabled]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xfcf0800000-0xfcf0ffffff 64bit pref]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xfcf1000000-0xfcf17fffff 64bit pref]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xfcf1800000-0xfcf1807fff 64bit pref]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xfcf1808000-0xfcf180ffff 64bit pref]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xf6b00000-0xf6b7ffff pref]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xf6b80000-0xf6bfffff pref]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xf6a00000-0xf6a07fff 64bit]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xf6900000-0xf697ffff pref]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xf6980000-0xf69803ff]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xf6900000-0xf697ffff pref]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xf6980000-0xf69fffff pref]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xf6a00000-0xf6a07fff 64bit]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xf6b00000-0xf6b7ffff pref]
Apr 15 06:48:01 TrueNAS kernel: release child resource [mem 0xf6b80000-0xf6b803ff]
Apr 15 06:48:01 TrueNAS kernel: Monitor-Mwait will be used to enter C-1 state
Apr 15 06:48:01 TrueNAS kernel:   with arguments:
Apr 15 06:48:01 TrueNAS kernel:     /init
Apr 15 06:48:01 TrueNAS kernel:   with environment:
Apr 15 06:48:01 TrueNAS kernel:     HOME=/
Apr 15 06:48:01 TrueNAS kernel:     TERM=linux
Apr 15 06:48:01 TrueNAS kernel:     BOOT_IMAGE=/ROOT/24.10.2.1@/boot/vmlinuz-6.6.44-production+truenas
Apr 15 06:48:01 TrueNAS kernel:     zfsforce=1
Apr 15 06:48:01 TrueNAS kernel: libata version 3.00 loaded.
Apr 15 06:48:01 TrueNAS kernel: ahci 0000:0b:00.0: version 3.0
Apr 15 06:48:01 TrueNAS kernel: sd 1:0:0:0: [sdc] Mode Sense: 00 3a 00 00
Apr 15 06:48:01 TrueNAS kernel: sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
Apr 15 06:48:01 TrueNAS kernel: sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Apr 15 06:48:01 TrueNAS kernel: sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00

error log-file:

Apr 13 09:00:56 TrueNAS kernel: watchdog: watchdog0: watchdog did not stop!
Apr 13 09:04:53 TrueNAS kernel: hub 8-0:1.0: config failed, hub doesn't have any ports! (err -19)
Apr 13 09:04:53 TrueNAS kernel: Error: Driver 'pcspkr' is already registered, aborting...
Apr 13 09:04:53 TrueNAS (udev-worker)[940]: event8: Failed to call EVIOCSKEYCODE with scan code 0x7c, and key code 190: Invalid argument
Apr 13 09:05:21 TrueNAS blkmapd[3873]: open pipe file /run/rpc_pipefs/nfs/blocklayout failed: No such file or directory
Apr 13 09:05:22 TrueNAS libvirtd[4296]: invalid argument: cannot find architecture arm
Apr 15 06:48:01 TrueNAS kernel: hub 8-0:1.0: config failed, hub doesn't have any ports! (err -19)
Apr 15 06:48:01 TrueNAS kernel: Error: Driver 'pcspkr' is already registered, aborting...
Apr 15 06:48:01 TrueNAS (udev-worker)[922]: event4: Failed to call EVIOCSKEYCODE with scan code 0x7c, and key code 190: Invalid argument
Apr 15 06:48:31 TrueNAS blkmapd[3856]: open pipe file /run/rpc_pipefs/nfs/blocklayout failed: No such file or directory
Apr 15 06:48:32 TrueNAS libvirtd[4286]: invalid argument: cannot find architecture arm

Can anybody of you see a hint what’s going wrong?

UPDATE:
I have load the standard bios settings and switched off all power saving options I could find.
XMP-Profile for the RAM is disabled.
Power Supply Idle Control is set to Typical Current Idle
Power saving features should be disabled
PBO is disabled

2

So after 9 days of continues running I had today again a spontaneous reboot of my server during the night.
I’m slowly starting to think that it’s not the motherboard’s fault, even it’s a gaming board.
In the messages, syslog and kernel log file I have a lot of those entries:

Apr 29 04:15:20 TrueNAS kernel: i40e 0000:06:00.0: ARQ: Unknown event 0x0000 ignored

what do these entries mean?
Is there e problem with my X710-DA2 network card?
Has anybody else the same experience with this card and TrueNAS Scale?

3

are these error messages maybe related to the fact that i am using the network card only in a x4 PCIe slot?

4

At this point I’d say that the NIC would not be my primary suspect since, as you say, it worked with the previous hardware. Your new and untested server however…

Overclocking or undervolting a server is unusual and typically not advised. I suggest you remove that and similar optimisation you have done. Get the system stable first at least.

Run memtest86 overnight and also verify that your CPU can handle load as well as idle without locking up or rebooting unexpectedly.

“Power Supply Idle Control” set to Typical Current Idle is good, leave it at that setting.

5

Thanks for your advice. How to check the CPU with load? idle is more or less easy, thats the normal operation point for my system at the moment.

6

So, I think it looks like I have maybe found the problem. In the bios settings under “Advanced - AMD CBS” is the ECC option located, this was set to Auto. I have now disabled the option and the system is running since 10 days without any problem.

I have also to mention that I replaced the Intel X710 network card with a Mellanox ConnectX-3 CX311A. Later on I will switch back to my Intel card to observe the behavior.