Truenas Crashing when I generate significant loads

TrueNAS General
, ,
1

This is my first time posting in this forums so I will attempt to keep it short and simple. I have used Truenas Core for a number of years with no issues, I recently rebuilt my system with the exception of bringing over some drives and my sata expansion card. (All parts are listed below)

In this rebuild I have gone with Scale as I want to ensure that I get access to new features when they are released particulary when the raidz expansion feature is finally released in ZFS and eventually released into Truenas Scale.

I have noticed that my system when under load on the drives will tend to crash, as an example, I can install and use plex (Truenas app) and a number of truechart apps no problem but the second I request plex to scan my library my system will crash within a few minutes. I feel like it might be to do with my sata expansion card but I am not sure where to start in troubleshooting this.

I am happy to get logs/ect whatever is needed I just might need some guidance as to what to provide. Here is a sample of the syslog (first 50 lines around the time of the first reboot) between two restarts that happened today. (I was trying to sync a rather large git repo for a custom app to my drives)

Jun 23 14:30:09 truenas kernel: kube-bridge: port 17(vethda4d9f96) entered disabled state
Jun 23 14:30:09 truenas systemd[1]: run-netns-cni\x2dea4d784a\x2d8e26\x2d9a42\x2dc6d4\x2d91b464fe9dfa.mount: Deactivated successfully.
Jun 23 14:30:09 truenas systemd[1]: Removed slice kubepods-burstable-pod46f89fd4_2c01_4304_8155_76a4194b3b4f.slice - libcontainer container kubepods-burstable-pod46f89fd4_2c01_4304_8155_76a4194b3b4f.slice.
Jun 23 14:40:00 truenas systemd[1]: Starting sysstat-collect.service - system activity accounting tool...
Jun 23 14:40:00 truenas systemd[1]: sysstat-collect.service: Deactivated successfully.
Jun 23 14:40:00 truenas systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
Jun 23 14:43:44 truenas syslog-ng[5088]: syslog-ng starting up; version='3.38.1'
Jun 23 14:42:59 truenas kernel: Linux version 6.6.29-production+truenas (root@tnsbuilds01.tn.ixsystems.net) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Wed May 29 15:05:57 UTC 2024
Jun 23 14:42:59 truenas kernel: Command line: BOOT_IMAGE=/ROOT/24.04.1.1@/boot/vmlinuz-6.6.29-production+truenas root=ZFS=boot-pool/ROOT/24.04.1.1 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
Jun 23 14:42:59 truenas kernel: x86/tme: not enabled by BIOS
Jun 23 14:42:59 truenas kernel: x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
Jun 23 14:42:59 truenas kernel: BIOS-provided physical RAM map:
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x0000000000100000-0x0000000030d92fff] usable
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x0000000030d93000-0x0000000030d93fff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x0000000030d94000-0x00000000316d6fff] usable
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000316d7000-0x00000000347d6fff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000347d7000-0x0000000034a53fff] ACPI data
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x0000000034a54000-0x0000000034bd7fff] ACPI NVS
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x0000000034bd8000-0x0000000035f65fff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x0000000035f66000-0x0000000035ffefff] type 20
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x0000000035fff000-0x0000000035ffffff] usable
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x0000000036000000-0x0000000039ffffff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x000000003a800000-0x000000003abfffff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x000000003b000000-0x00000000403fffff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000c0000000-0x00000000cfffffff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000fc000000-0x00000000fc00ffff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000fed20000-0x00000000fed7ffff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
Jun 23 14:42:59 truenas kernel: BIOS-e820: [mem 0x0000000100000000-0x00000010bfbfffff] usable
Jun 23 14:42:59 truenas kernel: NX (Execute Disable) protection: active
Jun 23 14:42:59 truenas kernel: APIC: Static calls initialized
Jun 23 14:42:59 truenas kernel: efi: EFI v2.8 by American Megatrends
Jun 23 14:42:59 truenas kernel: efi: ACPI=0x34af5000 ACPI 2.0=0x34af5014 TPMFinalLog=0x34ac2000 SMBIOS=0x35a54000 SMBIOS 3.0=0x35a53000 MEMATTR=0x293ba018 ESRT=0x2f62eb98 MOKvar=0x35aac000 
Jun 23 14:42:59 truenas kernel: efi: Remove mem89: MMIO range=[0xc0000000-0xcfffffff] (256MB) from e820 map
Jun 23 14:42:59 truenas kernel: efi: Not removing mem90: MMIO range=[0xfc000000-0xfc00ffff] (64KB) from e820 map
Jun 23 14:42:59 truenas kernel: efi: Not removing mem91: MMIO range=[0xfe000000-0xfe010fff] (68KB) from e820 map
Jun 23 14:42:59 truenas kernel: efi: Not removing mem92: MMIO range=[0xfec00000-0xfec00fff] (4KB) from e820 map
Jun 23 14:42:59 truenas kernel: efi: Not removing mem93: MMIO range=[0xfed00000-0xfed00fff] (4KB) from e820 map
Jun 23 14:42:59 truenas kernel: efi: Not removing mem95: MMIO range=[0xfee00000-0xfee00fff] (4KB) from e820 map
Jun 23 14:42:59 truenas kernel: efi: Remove mem96: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 map
Jun 23 14:42:59 truenas kernel: SMBIOS 3.6.0 present.
Jun 23 14:42:59 truenas kernel: DMI: Gigabyte Technology Co., Ltd. Z790 UD/Z790 UD, BIOS F10 12/14/2023
Jun 23 14:42:59 truenas kernel: tsc: Detected 2100.000 MHz processor
Jun 23 14:42:59 truenas kernel: tsc: Detected 2112.000 MHz TSC
Jun 23 14:42:59 truenas kernel: last_pfn = 0x10bfc00 max_arch_pfn = 0x400000000
Jun 23 14:42:59 truenas kernel: MTRR map: 6 entries (3 fixed + 3 variable; max 23), built from 10 variable MTRRs
Jun 23 14:42:59 truenas kernel: x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
Jun 23 14:42:59 truenas kernel: last_pfn = 0x36000 max_arch_pfn = 0x400000000
Jun 23 14:42:59 truenas kernel: esrt: Reserving ESRT space from 0x000000002f62eb98 to 0x000000002f62ec48.
Jun 23 14:42:59 truenas kernel: Using GB pages for direct mapping
Jun 23 14:42:59 truenas kernel: Incomplete global flushes, disabling PCID
Jun 23 14:42:59 truenas kernel: Secure boot disabled
Jun 23 14:42:59 truenas kernel: RAMDISK: [mem 0x19c04000-0x1e260fff]
Jun 23 14:42:59 truenas kernel: ACPI: Early table checksum verification disabled
Jun 23 14:42:59 truenas kernel: ACPI: RSDP 0x0000000034AF5014 000024 (v02 ALASKA)
Jun 23 14:42:59 truenas kernel: ACPI: XSDT 0x0000000034AF4728 00010C (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: FACP 0x0000000034A51000 000114 (v06 ALASKA A M I    01072009 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: DSDT 0x00000000349C8000 088540 (v02 ALASKA A M I    01072009 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: FACS 0x0000000034BD7000 000040
Jun 23 14:42:59 truenas kernel: ACPI: FIDT 0x00000000349C7000 00009C (v01 ALASKA A M I    01072009 AMI  00010013)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x00000000349BB000 00BD29 (v02 GBT    GSWApp   00000001 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: HWIN 0x0000000034A53000 0000D4 (v00 GBT    INTEL    00070000 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x00000000349B7000 0031C9 (v02 INTEL  DTbtSsdt 00001000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x0000000034A52000 00038C (v02 PmaxDv Pmax_Dev 00000001 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x00000000349B1000 005D34 (v02 CpuRef CpuSsdt  00003000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x00000000349AE000 002A81 (v02 SaSsdt SaSsdt   00003000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x00000000349AA000 00334F (v02 INTEL  IgfxSsdt 00003000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: HPET 0x00000000349A9000 000038 (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: APIC 0x00000000349A8000 0001DC (v05 ALASKA A M I    01072009 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: MCFG 0x00000000349A7000 00003C (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x000000003499E000 008595 (v02 ALASKA A M I    00001000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x000000003499C000 001F1A (v02 ALASKA A M I    00001000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: NHLT 0x000000003499B000 00002D (v00 ALASKA A M I    01072009 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: LPIT 0x000000003499A000 0000CC (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x0000000034996000 002A83 (v02 ALASKA A M I    00001000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x000000003498C000 0092F4 (v02 ALASKA A M I    00000000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: DBGP 0x000000003498B000 000034 (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: DBG2 0x000000003498A000 000054 (v00 ALASKA A M I    01072009 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x0000000034988000 00190A (v02 ALASKA A M I    00001000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: FPDT 0x0000000034987000 000044 (v01 ALASKA A M I    01072009 AMI  01000013)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x0000000034985000 0019FA (v02 INTEL  xh_rps14 00000000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x0000000034981000 0039DA (v02 SocGpe SocGpe   00003000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: SSDT 0x000000003497D000 0039DA (v02 SocCmn SocCmn   00003000 INTL 20200717)
Jun 23 14:42:59 truenas kernel: ACPI: WPBT 0x0000000034859000 000034 (v01 ALASKA A M I    00000001 GBT  20221021)
Jun 23 14:42:59 truenas kernel: ACPI: PHAT 0x0000000034858000 00083C (v01 ALASKA A M I    00000005 MSFT 0100000D)
Jun 23 14:42:59 truenas kernel: ACPI: TPM2 0x0000000034857000 00004C (v04 ALASKA A M I    00000001 AMI  00000000)
Jun 23 14:42:59 truenas kernel: ACPI: WSMT 0x0000000034999000 000028 (v01 ALASKA A M I    01072009 AMI  00010013)
Jun 23 14:42:59 truenas kernel: ACPI: Reserving FACP table memory at [mem 0x34a51000-0x34a51113]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving DSDT table memory at [mem 0x349c8000-0x34a5053f]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving FACS table memory at [mem 0x34bd7000-0x34bd703f]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving FIDT table memory at [mem 0x349c7000-0x349c709b]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x349bb000-0x349c6d28]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving HWIN table memory at [mem 0x34a53000-0x34a530d3]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x349b7000-0x349ba1c8]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x34a52000-0x34a5238b]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x349b1000-0x349b6d33]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x349ae000-0x349b0a80]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x349aa000-0x349ad34e]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving HPET table memory at [mem 0x349a9000-0x349a9037]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving APIC table memory at [mem 0x349a8000-0x349a81db]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving MCFG table memory at [mem 0x349a7000-0x349a703b]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x3499e000-0x349a6594]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x3499c000-0x3499df19]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving NHLT table memory at [mem 0x3499b000-0x3499b02c]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving LPIT table memory at [mem 0x3499a000-0x3499a0cb]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x34996000-0x34998a82]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x3498c000-0x349952f3]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving DBGP table memory at [mem 0x3498b000-0x3498b033]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving DBG2 table memory at [mem 0x3498a000-0x3498a053]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x34988000-0x34989909]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving FPDT table memory at [mem 0x34987000-0x34987043]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x34985000-0x349869f9]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x34981000-0x349849d9]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving SSDT table memory at [mem 0x3497d000-0x349809d9]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving WPBT table memory at [mem 0x34859000-0x34859033]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving PHAT table memory at [mem 0x34858000-0x3485883b]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving TPM2 table memory at [mem 0x34857000-0x3485704b]
Jun 23 14:42:59 truenas kernel: ACPI: Reserving WSMT table memory at [mem 0x34999000-0x34999027]
Jun 23 14:42:59 truenas kernel: No NUMA configuration found
Jun 23 14:42:59 truenas kernel: Faking a node at [mem 0x0000000000000000-0x00000010bfbfffff]
Jun 23 14:42:59 truenas kernel: NODE_DATA(0) allocated [mem 0x10bfbd4000-0x10bfbfffff]
Jun 23 14:42:59 truenas kernel: cma: Reserved 256 MiB at 0x0000000100000000 on node -1
Jun 23 14:42:59 truenas kernel: Zone ranges:
Jun 23 14:42:59 truenas kernel:   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
Jun 23 14:42:59 truenas kernel:   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
Jun 23 14:42:59 truenas kernel:   Normal   [mem 0x0000000100000000-0x00000010bfbfffff]
Jun 23 14:42:59 truenas kernel:   Device   empty
Jun 23 14:42:59 truenas kernel: Movable zone start for each node
Jun 23 14:42:59 truenas kernel: Early memory node ranges

here is a similar log from the error log:

Jun 23 13:18:02 truenas kernel: IPVS: rr: TCP 172.17.192.3:5432 - no destination available
Jun 23 13:18:07 truenas kernel: IPVS: rr: TCP 172.17.192.3:5432 - no destination available
Jun 23 14:43:00 truenas kernel: Error: Driver 'pcspkr' is already registered, aborting...
Jun 23 14:43:28 truenas dhclient[3860]: Timeout too large reducing to: 2147483646 (TIME_MAX - 1)
Jun 23 14:43:44 truenas blkmapd[5045]: open pipe file /run/rpc_pipefs/nfs/blocklayout failed: No such file or directory
Jun 23 14:44:42 truenas kernel: IPVS: rr: TCP 192.168.0.3:443 - no destination available
Jun 23 14:44:43 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:43 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:44 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:44 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:44 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:44 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:46 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:46 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:46 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:48 truenas kernel: IPVS: rr: TCP 192.168.0.3:10400 - no destination available
Jun 23 14:44:48 truenas kernel: IPVS: rr: TCP 192.168.0.3:10400 - no destination available
Jun 23 14:44:48 truenas kernel: IPVS: rr: TCP 192.168.0.3:10400 - no destination available
Jun 23 14:44:48 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:48 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:49 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:49 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:49 truenas kernel: IPVS: rr: UDP 192.168.0.3:53 - no destination available
Jun 23 14:44:49 truenas kernel: IPVS: rr: UDP 192.168.0.3:53 - no destination available
Jun 23 14:44:49 truenas kernel: IPVS: rr: TCP 192.168.0.3:10400 - no destination available
Jun 23 14:44:53 truenas kernel: IPVS: rr: UDP 192.168.0.3:53 - no destination available
Jun 23 14:44:53 truenas kernel: IPVS: rr: UDP 192.168.0.3:53 - no destination available
Jun 23 14:44:53 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:53 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:54 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:54 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:54 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:54 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:55 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:55 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:58 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:58 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:59 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:44:59 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:45:00 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:45:00 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:45:00 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:45:00 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:45:00 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:45:00 truenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Jun 23 14:45:04 truenas kernel: IPVS: rr: UDP 192.168.0.3:53 - no destination available
Jun 23 14:45:04 truenas kernel: IPVS: rr: TCP 172.17.226.209:80 - no destination available
Jun 23 14:45:04 truenas kernel: IPVS: rr: UDP 192.168.0.3:53 - no destination available
Jun 23 14:45:04 truenas kernel: IPVS: rr: UDP 192.168.0.3:53 - no destination available
Jun 23 14:45:04 truenas kernel: IPVS: rr: TCP 172.17.117.95:5432 - no destination available

My use case is a home server, I typically run plex that my family and a few relatives will use, I use nextcloud to backup iphone photos so that I can keep the device clear and I like to experiment with my own docker apps ect. I do also run pihole for local dns services. I will note that I have a pool of ironwolf 4s that are existing and I want to move the data from them onto my new Ironwolf 8s as the ironwolf4s have a powered on time of 9 years each and I am starting to be paranoid over the fact they have lived as long as they have without a significant issue.

I cant remember the setup of the old system but I can say that my configuration was 2 drives were plugged into the SATA expansion card and the rest (boot drive included as it was a 256gb sata ssd) was plugged into the motherboard.

My new drive mapping is:

4 drives from the new pool are plugged into the motherboard onboard SATA ports 2 drives from the ironwolf8s pool is plugged into the motherboard and the remaining 4 drives are plugged into the sata expansion card.

I would get a secondhand HBA but I have no idea what to look for in that or if that would even solve the issue I have been experiencing.

New System is:

CPU: Intel I7 14-700 20 core 28 threads
Motherboard: Gigabyte S1700 ATX Z790 UD DDR5
RAM: Crucial 64GB (2x32gb) CL40 4800 MHz DDR5 Ram (non ECC in dual channel mode)
Power Supply: Thermaltake Toughpower GF A3 750W 80+ Gold Cen5 Fully Modular Power Supply
CASE: Thermaltake Core W200
Boot SSD: 500GB Samsung Evo PLUS M.2 PCIe SSD MZ-V7S500BW
Sata Expansion card: Silverstone 6 port Sata 3 PCIe Card SST-ECS06)
Pools:
Ironwolf4s - Raid Z1 3x Ironwolf 4tb 5900 RPM ST4000DM000 (these are old and were used as individual drives not in any form of raid for approximately 3 of their 9 year life span)
Ironwolf8s - Raid Z1 3x Ironwolf 8TB 7200 RPM ST8000VN004 (these have about 18 months on time)
newironwolfs - Raid Z2 4x Ironwolf 8TB 7200 RPM ST8000VN004 (this is my new one)

2

Thanks… there are no alerts within TrueNAS?

Are there any messages which indicate that it might be a disk issue?

If its a hard crash with no messages, it might be a CPU/RAM issue. High scan load is also more RAM activity as well as disk activity. Without ECC there are no warning if there is any memory integrity issues.

The IP error messages would seem to be irrelevant, but have you verified the Ip setup and why those addresses are failing?

1 Like
3

Hey thanks for the reply, there are no alerts nothing emails me about alerts and nothing in the UI I had to setup an init script to email me after boot to know it occurred. The ip address issues are all related to my apps I don’t use that subnet for anything else and I don’t consider it relevant. I believe all the ones going to port 53 are the dns calls looking for my pihole.

I would describe it as more of a reboot than a crash. I did go through what I thought was an issue in the logs and found something I believed to be the issue but I suspect it is a red haring https://www.truenas.com/community/threads/truenas-kernel-x86-split-lock-detection-ac-crashing-the-kernel-on-kernel-split_locks-and-warning-on-user-space-split_locks.114282/

Should I be doing a ram test? and CPU test? A Mobo test? Is there any recommendations for all of those things? (Would it even matter with the lack of ECC Memory)

I should probably update the bios on the motherboard too I didn’t update that when I got it and I guess it could be out of date.

I think the next thing I want to look at is a proper HBA instead of the sata expansion card I have but I am not sure what to look for in that department, I am interested in keeping the power consumption/heat down but don’t know where to start when looking for one.

I can also upload more of the log file but between the two reboots today alone is around 4.8k of logs I am not sure where that would be best uploaded to.

4

@Captain_Morgan i did also find this thread https://www.truenas.com/community/threads/asm1166-on-truenas-scale-22-02-rc-2.98134/

Where people were crapping on the sata card I am using. I just wonder if it is the problem (I probably want it to be that instead of any of the new hardware I bought haha)

1 Like
5

Process of elimination from low cost to high. I’d start with a memtest, I went with the paid version that you can boot off a USB stick. Verify all is well there.

You may also have a PSU that is dying for some reason. For example, my Seasonic 750w is only pulling 120W max at the socket during boot but it still died after several years of use. But unless you’re standing next to the cpu when this is happening, it’s hard to diagnose.

An IT flashed genuine LSI HBA is cheap insurance when you consider that the trigger is a heavy workload. Make sure it gets good air flow, those ASICs (or whatever those chips are) get super hot. Heat may also be affecting your current HBA. Try putting a fan over / on it.

In the end, I’ve accumulated a bunch of spares here. Spare drives, a HBA, a PSU, etc. It speeds up troubleshooting significantly.

2 Likes
6

Definitely good advice to start with a memtest but what do you mean by:

You shouldn’t need to pay for a version of memtest that boots of a USB-stick. Use the free version.

Edit: They even offer a video showing you how to create the bootable USB-stick. I imagine Rufus or similar tools ought to work as well.

7

As far as I know the paid version allora you to inject errors in the memory in order to test the ECC functionality. But we are digressing.

1 Like
8

I really appreciate the tips will run a memtest tomorrow when I wake up and leave it running for a bit but I want to look at a LSI hba any recommendations for any or just find one that is flashed to it mode?

9

If you want to try, buy something like this.

But yeah, I would totally run memtest first… Make also sure you are not running OC of any kind, including to the memory… as a side note, you have a rather power-hungry nas there with that 14700 i7; what is the power limit you have set in the BIOS? Intel is having a lot of troubles with these chips lately.

10

eBay has some resellers that specialize in reselling IT equipment. They usually will also advertise that the HBA has been cross flashed to IT Mode. AFAIK, if you’re attaching HDDs, pretty much any LSI HBA will do but check the resources section here and in the old forum for hardware advice re: what models / firmwares you should be using.

11

Correct, the incremental cost was low enough and my NAS memory issues were persnickety enough that I went for the paid + version.

12

My take on this.
Memory test followd by replacing that card with a proper HBA

As for the case - I love it. But when you put an HBA in, make sure the HBA is covered by extra airflow (add a fan to the card is some manner)

2 Likes
13

While that card may not be as reliable as a good HBA, I doubt that it would cause a crash. It might appear as drives missing and pool failure. Power and memory can cause crashes…

2 Likes
14

https://hardforum.com/threads/intel-finds-root-cause-of-raptor-lake-cpu-stability-issues-bios-with-new-microcode-underway.2035393/#post-1045904805

2 Likes
15

IMG_1414
IMG_14141170Ă—952 201 KB

But that implies that there is an issue.

2 Likes
16

Okay I did ran a memory test and left if for a few hours and it passed. I also did a bios update (gigabytes F12c) includes an intel defaults I know that the motherboards going over the intel recommended max volts is mainly a K series issue but I have seen some reports of non K series having similar issues so I figured that might be a go here.

I have noticed that my cpu isn’t going as hard at library scans or metadata refreshes. Previously anything like that would spike my cpu temps into the 60-70-80s and now it’s pretty stable around 40-50 maximum.

I am starting to believe the issue was as many of you mentioned a cpu/ram issue and I suspect that it was turboing too hard for too long and would crash.

I will do some more tests and validate. Maybe spin up some vms (would previously set my cpu to 70-80 degrees to spin up a vm).

Are there any other recommendations for what I can do that will push the system to test stability?

Oh I am going to go with a LSI 9300-8i for a HBA and up my system fan count from 11 to 20 to bring in more airflow the size of the core w200 I feel like I need to really add more fans to get the air moving.

17

Run prime95

Lots of various ways of running it, single threads, 4 threads, all threads, avx enabled etc

You should just be able to run the Linux binary after decompressing it somewhere, say on your pool.

3 Likes
18

I’ve made some really janky solutions in terms of fitting more [edit] fans cases - I’ve even used radiator brackets internally from other cases to get fans pushing air on drives inside of the case… with surprising results (-6*c across the board). A lot of it isn’t just moving air into the case, but actually getting it across the HDDs.

19

Prime95 kills them all. How are you cooling that monster of silicon?

1 Like
20

Not my systems. 48 hours.

Anything more is prematurely aging your equipment :wink:

For bonus fun run solnet array tester across all disks at same time :wink:

1 Like