Nov 9 01:00:28 TNAS1 kernel: bash invoked oom-killer: gfp_mask=0x102dc2(GFP_HIGHUSER|__GFP_NOWARN|__GFP_ZERO), order=0, oom_score_adj=-900
Nov 9 01:00:28 TNAS1 kernel: CPU: 3 UID: 999 PID: 2479756 Comm: bash Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:00:28 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:00:28 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:00:28 TNAS1 kernel: Call Trace:
Nov 9 01:00:28 TNAS1 kernel:
Nov 9 01:00:28 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:00:28 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:00:28 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:00:28 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:00:28 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:00:28 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:00:28 TNAS1 kernel: alloc_pages_bulk_noprof+0x3df/0x720
Nov 9 01:00:28 TNAS1 kernel: __vmalloc_node_range_noprof+0x24f/0x850
Nov 9 01:00:28 TNAS1 kernel: ? kernel_clone+0xc2/0x430
Nov 9 01:00:28 TNAS1 kernel: copy_process+0xde5/0x26b0
Nov 9 01:00:28 TNAS1 kernel: ? kernel_clone+0xc2/0x430
Nov 9 01:00:28 TNAS1 kernel: ? list_head+0x9/0x30 [zfs]
Nov 9 01:00:28 TNAS1 kernel: ? mutex_lock+0x12/0x30
Nov 9 01:00:28 TNAS1 kernel: ? spl_kmem_cache_free+0xe6/0x130 [spl]
Nov 9 01:00:28 TNAS1 kernel: ? cityhash_impl+0x73/0x90 [zfs]
Nov 9 01:00:28 TNAS1 kernel: ? dbuf_cache_multilist_index_func+0x30/0x50 [zfs]
Nov 9 01:00:28 TNAS1 kernel: ? mutex_lock+0x12/0x30
Nov 9 01:00:28 TNAS1 kernel: kernel_clone+0xc2/0x430
Nov 9 01:00:28 TNAS1 kernel: ? dbuf_evict_notify+0xe/0x50 [zfs]
Nov 9 01:00:28 TNAS1 kernel: __do_sys_clone+0x66/0x90
Nov 9 01:00:28 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:00:28 TNAS1 kernel: ? __lruvec_stat_mod_folio+0x83/0xd0
Nov 9 01:00:28 TNAS1 kernel: ? rrw_exit+0x5e/0xc0 [zfs]
Nov 9 01:00:28 TNAS1 kernel: ? folio_add_file_rmap_ptes+0x38/0xb0
Nov 9 01:00:28 TNAS1 kernel: ? next_uptodate_folio+0x85/0x2a0
Nov 9 01:00:28 TNAS1 kernel: ? filemap_map_pages+0x521/0x650
Nov 9 01:00:28 TNAS1 kernel: ? do_fault+0x281/0x4d0
Nov 9 01:00:28 TNAS1 kernel: ? __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:00:28 TNAS1 kernel: ? syscall_exit_to_user_mode_prepare+0x148/0x170
Nov 9 01:00:28 TNAS1 kernel: ? __count_memcg_events+0x53/0xf0
Nov 9 01:00:28 TNAS1 kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 9 01:00:28 TNAS1 kernel: ? handle_mm_fault+0xae/0x2d0
Nov 9 01:00:28 TNAS1 kernel: ? do_user_addr_fault+0x17f/0x670
Nov 9 01:00:28 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:00:28 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:00:28 TNAS1 kernel: RIP: 0033:0x7fe37389c313
Nov 9 01:00:28 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7fe37389c2e9.
Nov 9 01:00:28 TNAS1 kernel: RSP: 002b:00007ffe09353958 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Nov 9 01:00:28 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe37389c313
Nov 9 01:00:28 TNAS1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Nov 9 01:00:28 TNAS1 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000055cbcc0923e0
Nov 9 01:00:28 TNAS1 kernel: R10: 00007fe3737c5a10 R11: 0000000000000246 R12: 0000000000000001
Nov 9 01:00:28 TNAS1 kernel: R13: 00007ffe09353b90 R14: 000055cb8d949b08 R15: 00000000ffffffff
Nov 9 01:00:28 TNAS1 kernel:
Nov 9 01:00:28 TNAS1 kernel: Mem-Info:
Nov 9 01:00:28 TNAS1 kernel: active_anon:2535528 inactive_anon:5214016 isolated_anon:0
active_file:242 inactive_file:230 isolated_file:0
unevictable:1000 dirty:5 writeback:29
slab_reclaimable:59411 slab_unreclaimable:100329
mapped:4644 shmem:92881 pagetables:20631
sec_pagetables:5454 bounce:0
kernel_misc_reclaimable:0
free:60798 free_pcp:239 free_cma:0
Nov 9 01:00:28 TNAS1 kernel: Node 0 active_anon:10142112kB inactive_anon:20856064kB active_file:1000kB inactive_file:596kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:18576kB dirty:20kB writeback:116kB shmem:371524kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:11175936kB writeback_tmp:0kB kernel_stack:14000kB pagetables:82524kB sec_pagetables:21816kB all_unreclaimable? no
Nov 9 01:00:28 TNAS1 kernel: Node 0 DMA free:48kB boost:2048kB min:2076kB low:2088kB high:2100kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11208kB active_file:4kB inactive_file:0kB unevictable:0kB writepending:4kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:00:28 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:00:28 TNAS1 kernel: Node 0 DMA32 free:41628kB boost:21136kB min:28292kB low:31760kB high:35228kB reserved_highatomic:0KB active_anon:26628kB inactive_anon:3398276kB active_file:828kB inactive_file:1348kB unevictable:0kB writepending:4kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:804kB local_pcp:0kB free_cma:0kB
Nov 9 01:00:28 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:00:28 TNAS1 kernel: Node 0 Normal free:201080kB boost:141312kB min:201704kB low:230964kB high:260224kB reserved_highatomic:14336KB active_anon:10115484kB inactive_anon:17446124kB active_file:0kB inactive_file:384kB unevictable:4000kB writepending:128kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:00:28 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:00:28 TNAS1 kernel: Node 0 DMA: 3*4kB (U) 1*8kB (U) 0*16kB 1*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 52kB
Nov 9 01:00:28 TNAS1 kernel: Node 0 DMA32: 125*4kB (UME) 75*8kB (UME) 77*16kB (UME) 61*32kB (UME) 42*64kB (UME) 13*128kB (UM) 9*256kB (UM) 9*512kB (UM) 5*1024kB (UM) 11*2048kB (UME) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 43196kB
Nov 9 01:00:28 TNAS1 kernel: Node 0 Normal: 2909*4kB (UM) 1913*8kB (UM) 1642*16kB (UM) 705*32kB (UM) 299*64kB (UM) 165*128kB (UM) 102*256kB (UM) 68*512kB (UM) 24*1024kB (UM) 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 201532kB
Nov 9 01:00:28 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:00:28 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:00:28 TNAS1 kernel: 93276 total pagecache pages
Nov 9 01:00:28 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:00:28 TNAS1 kernel: Free swap = 0kB
Nov 9 01:00:28 TNAS1 kernel: Total swap = 0kB
Nov 9 01:00:28 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:00:28 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:00:28 TNAS1 kernel: 170227 pages reserved
Nov 9 01:00:28 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:00:28 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:00:28 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:00:28 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:00:28 TNAS1 kernel: [ 728] 104 728 2091 342 256 86 0 65536 0 -900 dbus-daemon
Nov 9 01:00:28 TNAS1 kernel: [ 741] 0 741 11506 316 224 92 0 102400 0 -250 systemd-journal
Nov 9 01:00:28 TNAS1 kernel: [ 763] 0 763 6768 569 512 57 0 77824 0 -1000 systemd-udevd
Nov 9 01:00:28 TNAS1 kernel: [ 1630] 0 1630 5703780 4837890 4837577 249 64 39522304 0 -1000 asyncio_loop
Nov 9 01:00:28 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:00:28 TNAS1 kernel: [ 1681] 0 1681 485024 35321 35252 69 0 651264 0 -1000 middlewared (ze
Nov 9 01:00:28 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:00:28 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:00:28 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:00:28 TNAS1 kernel: [ 3516] 0 3516 21731 203 94 109 0 57344 0 -1000 auditd
Nov 9 01:00:28 TNAS1 kernel: [ 3540] 0 3540 27526 4273 4148 125 0 114688 0 0 truenas_audit_h
Nov 9 01:00:28 TNAS1 kernel: [ 3593] 106 3593 1970 209 96 113 0 57344 0 0 rpcbind
Nov 9 01:00:28 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:00:28 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:00:28 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:00:28 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:00:28 TNAS1 kernel: [ 3599] 0 3599 5605 451 320 131 0 86016 0 0 certmonger
Nov 9 01:00:28 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:00:28 TNAS1 kernel: [ 3629] 0 3629 38187 137 64 73 0 65536 0 -1000 lxcfs
Nov 9 01:00:28 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:00:28 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:00:28 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:00:28 TNAS1 kernel: [ 3681] 0 3681 280105 6284 6236 48 0 581632 0 0 syslog-ng
Nov 9 01:00:28 TNAS1 kernel: [ 3691] 0 3691 12416 358 224 134 0 94208 0 0 systemd-logind
Nov 9 01:00:28 TNAS1 kernel: [ 3695] 0 3695 60178 420 256 164 0 86016 0 0 zed
Nov 9 01:00:28 TNAS1 kernel: [ 3708] 0 3708 523983 4140 4140 0 0 327680 0 -999 containerd
Nov 9 01:00:28 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:00:28 TNAS1 kernel: [ 3809] 122 3809 3089 147 121 26 0 49152 0 0 usbhid-ups
Nov 9 01:00:28 TNAS1 kernel: [ 3891] 0 3891 1005 150 32 118 0 45056 0 0 cron
Nov 9 01:00:28 TNAS1 kernel: [ 3962] 122 3962 1695 129 96 33 0 57344 0 0 upsd
Nov 9 01:00:28 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:00:28 TNAS1 kernel: [ 3995] 0 3995 74338 14663 14472 191 0 217088 0 0 cli
Nov 9 01:00:28 TNAS1 kernel: [ 4009] 122 4009 2127 226 96 130 0 53248 0 0 upsmon
Nov 9 01:00:28 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:00:28 TNAS1 kernel: [ 4132] 0 4132 4136 347 224 123 0 69632 0 0 systemd-machine
Nov 9 01:00:28 TNAS1 kernel: [ 4133] 0 4133 427247 2880 2880 0 0 339968 0 0 libvirtd
Nov 9 01:00:28 TNAS1 kernel: [ 5386] 105 5386 1781 175 96 79 0 49152 0 0 avahi-daemon
Nov 9 01:00:28 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:00:28 TNAS1 kernel: [ 5692] 1 5692 8070 3945 3840 105 0 98304 0 0 wsdd.py
Nov 9 01:00:28 TNAS1 kernel: [ 5694] 0 5694 17601 614 544 70 0 118784 0 0 nmbd
Nov 9 01:00:28 TNAS1 kernel: [ 6054] 0 6054 16985 608 448 160 0 114688 0 -900 virtlogd
Nov 9 01:00:28 TNAS1 kernel: [ 6171] 986 6171 800227 294971 294906 65 0 2891776 0 0 qemu-system-x86
Nov 9 01:00:28 TNAS1 kernel: [ 6672] 0 6672 13921 5608 5472 136 0 139264 0 -1000 websockify
Nov 9 01:00:28 TNAS1 kernel: [ 6757] 999 6757 116708 29374 29218 92 64 475136 0 -900 netdata
Nov 9 01:00:28 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:00:28 TNAS1 kernel: [ 7021] 999 7021 22248 4935 4846 89 0 147456 0 -900 python.d.plugin
Nov 9 01:00:28 TNAS1 kernel: [ 7189] 986 7189 2887945 924115 924024 91 0 14200832 0 0 qemu-system-x86
Nov 9 01:00:28 TNAS1 kernel: [ 7210] 0 7210 18963 787 608 83 96 131072 0 0 winbindd
Nov 9 01:00:28 TNAS1 kernel: [ 7212] 0 7212 18982 780 640 102 38 131072 0 0 wb[TNAS1]
Nov 9 01:00:28 TNAS1 kernel: [ 7230] 0 7230 13921 5432 5413 19 0 131072 0 -1000 websockify
Nov 9 01:00:28 TNAS1 kernel: [ 7324] 986 7324 2788322 1381735 1381576 159 0 17756160 0 0 qemu-system-x86
Nov 9 01:00:28 TNAS1 kernel: [ 7368] 0 7368 13921 5635 5539 96 0 139264 0 -1000 websockify
Nov 9 01:00:28 TNAS1 kernel: [ 7371] 33 7371 7296 672 567 105 0 69632 0 0 nginx
Nov 9 01:00:28 TNAS1 kernel: [ 10337] 0 10337 4476 549 416 133 0 73728 0 0 sshd
Nov 9 01:00:28 TNAS1 kernel: [ 10339] 0 10339 4477 534 448 86 0 81920 0 0 sshd
Nov 9 01:00:28 TNAS1 kernel: [ 10342] 0 10342 4941 727 608 119 0 86016 0 100 systemd
Nov 9 01:00:28 TNAS1 kernel: [ 10343] 0 10343 42782 1639 1482 157 0 94208 0 100 (sd-pam)
Nov 9 01:00:28 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:00:28 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:00:28 TNAS1 kernel: [ 11235] 0 11235 19579 818 640 103 75 135168 0 0 wb-idmap
Nov 9 01:00:28 TNAS1 kernel: [1629815] 0 1629815 20194 1429 608 96 725 147456 0 0 smbd
Nov 9 01:00:28 TNAS1 kernel: [1629817] 0 1629817 19634 712 630 82 0 131072 0 0 smbd-notifyd
Nov 9 01:00:28 TNAS1 kernel: [1629818] 0 1629818 19638 676 630 46 0 135168 0 0 smbd-cleanupd
Nov 9 01:00:28 TNAS1 kernel: [1632448] 0 1632448 34145 6436 2476 188 3772 212992 0 0 smbd[192.168.1.
Nov 9 01:00:28 TNAS1 kernel: [1636144] 0 1636144 33944 12476 8678 116 3682 262144 0 0 smbd[fe80::3b34
Nov 9 01:00:28 TNAS1 kernel: [1653950] 3000 1653950 34260 7349 3613 126 3610 225280 0 0 smbd[192.168.1.
Nov 9 01:00:28 TNAS1 kernel: [2479756] 999 2479756 1113 243 160 83 0 49152 0 -900 bash
Nov 9 01:00:28 TNAS1 kernel: [2709452] 0 2709452 45062 44 32 12 0 73728 0 0 nscd
Nov 9 01:00:28 TNAS1 kernel: [2773557] 0 2773557 206757 12450 12274 176 0 303104 0 -1000 middlewared (wo
Nov 9 01:00:28 TNAS1 kernel: [2773561] 0 2773561 207385 13099 12971 128 0 315392 0 -1000 middlewared (wo
Nov 9 01:00:28 TNAS1 kernel: [2776151] 0 2776151 207108 13000 12886 114 0 303104 0 -1000 middlewared (wo
Nov 9 01:00:28 TNAS1 kernel: [2777444] 0 2777444 206501 12387 12278 109 0 299008 0 -1000 middlewared (wo
Nov 9 01:00:28 TNAS1 kernel: [2777446] 0 2777446 207155 12674 12561 113 0 303104 0 -1000 middlewared (wo
Nov 9 01:00:28 TNAS1 kernel: [2779991] 3005 2779991 25663 2217 893 21 1303 180224 0 0 smbd[192.168.1.
Nov 9 01:00:28 TNAS1 kernel: [2780551] 0 2780551 22242 1020 637 123 260 155648 0 0 smbd[192.168.1.
Nov 9 01:00:28 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/machine.slice/machine-qemu\x2d3\x2d10HTPC.scope/libvirt,task=qemu-system-x86,pid=7324,uid=986
Nov 9 01:00:28 TNAS1 kernel: Out of memory: Killed process 7324 (qemu-system-x86) total-vm:11153288kB, anon-rss:5526304kB, file-rss:636kB, shmem-rss:0kB, UID:986 pgtables:17340kB oom_score_adj:0
Nov 9 01:00:28 TNAS1 systemd[1]: machine-qemu\x2d3\x2d10HTPC.scope: A process of this unit has been killed by the OOM killer.
Nov 9 01:00:28 TNAS1 kernel: br0: port 4(vnet2) entered disabled state
Nov 9 01:00:28 TNAS1 kernel: vnet2 (unregistering): left allmulticast mode
Nov 9 01:00:28 TNAS1 kernel: vnet2 (unregistering): left promiscuous mode
Nov 9 01:00:28 TNAS1 kernel: br0: port 4(vnet2) entered disabled state
Nov 9 01:00:28 TNAS1 systemd[1]: machine-qemu\x2d3\x2d10HTPC.scope: Deactivated successfully.
Nov 9 01:00:28 TNAS1 systemd[1]: machine-qemu\x2d3\x2d10HTPC.scope: Consumed 2d 5h 53min 47.219s CPU time.
Nov 9 01:00:28 TNAS1 nscd[2709452]: 2709452 monitoring file `/etc/nsswitch.conf` (4)
Nov 9 01:00:28 TNAS1 nscd[2709452]: 2709452 monitoring directory `/etc` (2)
Nov 9 01:00:28 TNAS1 nscd[2709452]: 2709452 monitoring file `/etc/resolv.conf` (3)
Nov 9 01:00:28 TNAS1 nscd[2709452]: 2709452 monitoring directory `/etc` (2)
Nov 9 01:01:13 TNAS1 kernel: qemu-system-x86 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Nov 9 01:01:13 TNAS1 kernel: CPU: 0 UID: 986 PID: 7189 Comm: qemu-system-x86 Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:13 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:13 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:13 TNAS1 kernel: Call Trace:
Nov 9 01:01:13 TNAS1 kernel:
Nov 9 01:01:13 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:13 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:13 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:13 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:13 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:13 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:13 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:13 TNAS1 kernel: get_free_pages_noprof+0x11/0x40
Nov 9 01:01:13 TNAS1 kernel: __pollwait+0xa5/0x120
Nov 9 01:01:13 TNAS1 kernel: eventfd_poll+0x2c/0x70
Nov 9 01:01:13 TNAS1 kernel: do_sys_poll+0x2dd/0x600
Nov 9 01:01:13 TNAS1 kernel: ? __pfx___pollwait+0x10/0x10
Nov 9 01:01:13 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:13 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:13 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:13 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:13 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:13 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:13 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:13 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:13 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:13 TNAS1 kernel: __x64_sys_ppoll+0xe2/0x170
Nov 9 01:01:13 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:13 TNAS1 kernel: ? syscall_exit_to_user_mode+0x10/0x1f0
Nov 9 01:01:13 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:13 TNAS1 kernel: ? syscall_exit_to_user_mode+0x10/0x1f0
Nov 9 01:01:13 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:13 TNAS1 kernel: ? __irq_exit_rcu+0x38/0xb0
Nov 9 01:01:13 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:13 TNAS1 kernel: RIP: 0033:0x7f687e5f72d6
Nov 9 01:01:13 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7f687e5f72ac.
Nov 9 01:01:13 TNAS1 kernel: RSP: 002b:00007ffe7f909c60 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
Nov 9 01:01:13 TNAS1 kernel: RAX: ffffffffffffffda RBX: 00005630866a0130 RCX: 00007f687e5f72d6
Nov 9 01:01:13 TNAS1 kernel: RDX: 00007ffe7f909c80 RSI: 000000000000004b RDI: 00005630875e2050
Nov 9 01:01:13 TNAS1 kernel: RBP: 00007ffe7f909cec R08: 0000000000000008 R09: 0000000000000000
Nov 9 01:01:13 TNAS1 kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 00007ffe7f909c80
Nov 9 01:01:13 TNAS1 kernel: R13: 00005630866a0130 R14: 000056304ffc4920 R15: 00007ffe7f909cf0
Nov 9 01:01:13 TNAS1 kernel:
Nov 9 01:01:13 TNAS1 kernel: Mem-Info:
Nov 9 01:01:13 TNAS1 kernel: active_anon:1090196 inactive_anon:6611642 isolated_anon:0
active_file:143 inactive_file:97 isolated_file:0
unevictable:1000 dirty:0 writeback:0
slab_reclaimable:59105 slab_unreclaimable:96615
mapped:4479 shmem:92870 pagetables:18940
sec_pagetables:3348 bounce:0
kernel_misc_reclaimable:0
free:45182 free_pcp:2235 free_cma:7
Nov 9 01:01:13 TNAS1 kernel: Node 0 active_anon:4360784kB inactive_anon:26446860kB active_file:572kB inactive_file:440kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:18000kB dirty:224kB writeback:0kB shmem:371480kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:4552704kB writeback_tmp:0kB kernel_stack:13840kB pagetables:75760kB sec_pagetables:13392kB all_unreclaimable? no
Nov 9 01:01:13 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11232kB active_file:0kB inactive_file:4kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:13 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:13 TNAS1 kernel: Node 0 DMA32 free:6164kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:20300kB inactive_anon:3452596kB active_file:184kB inactive_file:220kB unevictable:0kB writepending:72kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:1380kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:13 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:13 TNAS1 kernel: Node 0 Normal free:171972kB boost:0kB min:60392kB low:89652kB high:118912kB reserved_highatomic:0KB active_anon:4340484kB inactive_anon:22983216kB active_file:340kB inactive_file:596kB unevictable:4000kB writepending:152kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:8312kB local_pcp:0kB free_cma:28kB
Nov 9 01:01:13 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:13 TNAS1 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 32kB
Nov 9 01:01:13 TNAS1 kernel: Node 0 DMA32: 91*4kB (UME) 61*8kB (UME) 55*16kB (UME) 36*32kB (UE) 35*64kB (UE) 5*128kB (UE) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6020kB
Nov 9 01:01:13 TNAS1 kernel: Node 0 Normal: 1232*4kB (UE) 6208*8kB (UE) 3672*16kB (UE) 1780*32kB (UE) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 170368kB
Nov 9 01:01:13 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:13 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:13 TNAS1 kernel: 93278 total pagecache pages
Nov 9 01:01:13 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:13 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:13 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:13 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:13 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:13 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:13 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:13 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:13 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:13 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:13 TNAS1 kernel: [ 728] 104 728 2091 353 256 97 0 65536 0 -900 dbus-daemon
Nov 9 01:01:13 TNAS1 kernel: [ 741] 0 741 11506 482 224 258 0 102400 0 -250 systemd-journal
Nov 9 01:01:13 TNAS1 kernel: [ 763] 0 763 6768 551 512 39 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:13 TNAS1 kernel: [ 1630] 0 1630 7126624 6262897 6262557 276 64 50954240 0 -1000 asyncio_loop
Nov 9 01:01:14 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:14 TNAS1 kernel: [ 1681] 0 1681 485024 35397 35252 145 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:14 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:14 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:14 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:14 TNAS1 kernel: [ 3516] 0 3516 21731 210 94 116 0 57344 0 -1000 auditd
Nov 9 01:01:14 TNAS1 kernel: [ 3540] 0 3540 27526 4307 4148 159 0 114688 0 0 truenas_audit_h
Nov 9 01:01:14 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:14 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:14 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:14 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:14 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:14 TNAS1 kernel: [ 3599] 0 3599 5605 390 320 70 0 86016 0 0 certmonger
Nov 9 01:01:14 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:14 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:14 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:14 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:14 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:14 TNAS1 kernel: [ 3681] 0 3681 280105 6284 6236 48 0 577536 0 0 syslog-ng
Nov 9 01:01:14 TNAS1 kernel: [ 3691] 0 3691 12416 345 224 121 0 94208 0 0 systemd-logind
Nov 9 01:01:14 TNAS1 kernel: [ 3695] 0 3695 60178 346 256 90 0 86016 0 0 zed
Nov 9 01:01:14 TNAS1 kernel: [ 3708] 0 3708 523983 4180 4140 40 0 327680 0 -999 containerd
Nov 9 01:01:14 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:14 TNAS1 kernel: [ 3809] 122 3809 3089 240 121 119 0 49152 0 0 usbhid-ups
Nov 9 01:01:14 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:14 TNAS1 kernel: [ 3962] 122 3962 1695 134 96 38 0 57344 0 0 upsd
Nov 9 01:01:14 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:14 TNAS1 kernel: [ 3995] 0 3995 74338 14724 14472 252 0 217088 0 0 cli
Nov 9 01:01:14 TNAS1 kernel: [ 4009] 122 4009 2127 202 96 106 0 53248 0 0 upsmon
Nov 9 01:01:14 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:14 TNAS1 kernel: [ 4132] 0 4132 4136 337 224 113 0 69632 0 0 systemd-machine
Nov 9 01:01:14 TNAS1 kernel: [ 4133] 0 4133 427247 2973 2880 93 0 339968 0 0 libvirtd
Nov 9 01:01:14 TNAS1 kernel: [ 5386] 105 5386 1781 195 96 99 0 49152 0 0 avahi-daemon
Nov 9 01:01:14 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:14 TNAS1 kernel: [ 5692] 1 5692 8070 3987 3840 147 0 98304 0 0 wsdd.py
Nov 9 01:01:14 TNAS1 kernel: [ 5694] 0 5694 17601 599 544 55 0 118784 0 0 nmbd
Nov 9 01:01:14 TNAS1 kernel: [ 6054] 0 6054 16985 529 448 81 0 114688 0 -900 virtlogd
Nov 9 01:01:14 TNAS1 kernel: [ 6171] 986 6171 804325 294997 294906 91 0 2904064 0 0 qemu-system-x86
Nov 9 01:01:14 TNAS1 kernel: [ 6672] 0 6672 13921 5557 5472 85 0 139264 0 -1000 websockify
Nov 9 01:01:14 TNAS1 kernel: [ 6757] 999 6757 116708 29419 29218 137 64 475136 0 -900 netdata
Nov 9 01:01:14 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:14 TNAS1 kernel: [ 7021] 999 7021 22248 5006 4846 160 0 147456 0 -900 python.d.plugin
Nov 9 01:01:14 TNAS1 kernel: [ 7189] 986 7189 2887945 839390 839320 70 0 14200832 0 0 qemu-system-x86
Nov 9 01:01:14 TNAS1 kernel: [ 7210] 0 7210 18963 783 608 79 96 131072 0 0 winbindd
Nov 9 01:01:14 TNAS1 kernel: [ 7212] 0 7212 18982 765 640 87 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:14 TNAS1 kernel: [ 7230] 0 7230 13921 5517 5413 104 0 131072 0 -1000 websockify
Nov 9 01:01:14 TNAS1 kernel: [ 7371] 33 7371 7296 618 567 51 0 69632 0 0 nginx
Nov 9 01:01:14 TNAS1 kernel: [ 10337] 0 10337 4476 567 416 151 0 73728 0 0 sshd
Nov 9 01:01:14 TNAS1 kernel: [ 10339] 0 10339 4477 528 448 80 0 81920 0 0 sshd
Nov 9 01:01:14 TNAS1 kernel: [ 10342] 0 10342 4941 708 608 100 0 86016 0 100 systemd
Nov 9 01:01:14 TNAS1 kernel: [ 10343] 0 10343 42782 1639 1482 157 0 94208 0 100 (sd-pam)
Nov 9 01:01:14 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:14 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:14 TNAS1 kernel: [ 11235] 0 11235 19579 796 640 81 75 135168 0 0 wb-idmap
Nov 9 01:01:14 TNAS1 kernel: [1629815] 0 1629815 20194 1495 608 162 725 147456 0 0 smbd
Nov 9 01:01:14 TNAS1 kernel: [1629817] 0 1629817 19634 716 630 86 0 131072 0 0 smbd-notifyd
Nov 9 01:01:14 TNAS1 kernel: [1629818] 0 1629818 19638 705 630 75 0 135168 0 0 smbd-cleanupd
Nov 9 01:01:14 TNAS1 kernel: [1632448] 0 1632448 34145 6351 2476 103 3772 212992 0 0 smbd[192.168.1.
Nov 9 01:01:14 TNAS1 kernel: [1636144] 0 1636144 33944 12502 8678 142 3682 262144 0 0 smbd[fe80::3b34
Nov 9 01:01:14 TNAS1 kernel: [1653950] 0 1653950 34260 7712 3997 105 3610 225280 0 0 smbd[192.168.1.
Nov 9 01:01:14 TNAS1 kernel: [2479756] 999 2479756 1113 225 160 65 0 49152 0 -900 bash
Nov 9 01:01:14 TNAS1 kernel: [2709452] 0 2709452 77830 121 32 89 0 81920 0 0 nscd
Nov 9 01:01:14 TNAS1 kernel: [2773557] 0 2773557 206757 12393 12274 119 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:14 TNAS1 kernel: [2773561] 0 2773561 207385 13141 12971 170 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:14 TNAS1 kernel: [2776151] 0 2776151 207108 13018 12886 132 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:14 TNAS1 kernel: [2777444] 0 2777444 206501 12406 12278 128 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:14 TNAS1 kernel: [2777446] 0 2777446 207155 12693 12561 132 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:14 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=emulator,mems_allowed=0,global_oom,task_memcg=/machine.slice/machine-qemu\x2d2\x2d6DockerVM.scope/libvirt,task=qemu-system-x86,pid=7189,uid=986
Nov 9 01:01:14 TNAS1 kernel: Out of memory: Killed process 7189 (qemu-system-x86) total-vm:11551780kB, anon-rss:3357280kB, file-rss:280kB, shmem-rss:0kB, UID:986 pgtables:13868kB oom_score_adj:0
Nov 9 01:01:14 TNAS1 libvirtd[4133]: internal error: End of file from agent socket
Nov 9 01:01:14 TNAS1 systemd[1]: machine-qemu\x2d2\x2d6DockerVM.scope: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:16 TNAS1 kernel: br0: port 3(vnet1) entered disabled state
Nov 9 01:01:16 TNAS1 kernel: vnet1 (unregistering): left allmulticast mode
Nov 9 01:01:16 TNAS1 kernel: vnet1 (unregistering): left promiscuous mode
Nov 9 01:01:16 TNAS1 kernel: br0: port 3(vnet1) entered disabled state
Nov 9 01:01:16 TNAS1 systemd[1]: machine-qemu\x2d2\x2d6DockerVM.scope: Deactivated successfully.
Nov 9 01:01:16 TNAS1 systemd[1]: machine-qemu\x2d2\x2d6DockerVM.scope: Consumed 8h 10min 17.577s CPU time.
Nov 9 01:01:16 TNAS1 nscd[2709452]: 2709452 monitoring file `/etc/nsswitch.conf` (4)
Nov 9 01:01:16 TNAS1 nscd[2709452]: 2709452 monitoring directory `/etc` (2)
Nov 9 01:01:16 TNAS1 nscd[2709452]: 2709452 monitoring file `/etc/resolv.conf` (3)
Nov 9 01:01:16 TNAS1 nscd[2709452]: 2709452 monitoring directory `/etc` (2)
Nov 9 01:01:36 TNAS1 kernel: upsc invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=-900
Nov 9 01:01:36 TNAS1 kernel: CPU: 4 UID: 999 PID: 2782071 Comm: upsc Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:36 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:36 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:36 TNAS1 kernel: Call Trace:
Nov 9 01:01:36 TNAS1 kernel:
Nov 9 01:01:36 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:36 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:36 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:36 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:36 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:36 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:36 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:36 TNAS1 kernel: __pmd_alloc+0x34/0x210
Nov 9 01:01:36 TNAS1 kernel: move_page_tables+0xc70/0x1180
Nov 9 01:01:36 TNAS1 kernel: relocate_vma_down+0x11c/0x210
Nov 9 01:01:36 TNAS1 kernel: setup_arg_pages+0x1fb/0x3a0
Nov 9 01:01:36 TNAS1 kernel: load_elf_binary+0x37f/0x1720
Nov 9 01:01:36 TNAS1 kernel: ? load_misc_binary+0x20d/0x350 [binfmt_misc]
Nov 9 01:01:36 TNAS1 kernel: bprm_execve+0x23c/0x610
Nov 9 01:01:36 TNAS1 kernel: do_execveat_common.isra.0+0x17f/0x1e0
Nov 9 01:01:36 TNAS1 kernel: __x64_sys_execve+0x36/0x40
Nov 9 01:01:36 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:36 TNAS1 kernel: ? audit_reset_context+0x232/0x300
Nov 9 01:01:36 TNAS1 kernel: ? syscall_exit_to_user_mode_prepare+0x148/0x170
Nov 9 01:01:36 TNAS1 kernel: ? syscall_exit_to_user_mode+0x10/0x1f0
Nov 9 01:01:36 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:36 TNAS1 kernel: ? path_openat+0x114/0x1330
Nov 9 01:01:36 TNAS1 kernel: ? do_filp_open+0xc4/0x170
Nov 9 01:01:36 TNAS1 kernel: ? audit_filter_rules.constprop.0+0x140/0x1120
Nov 9 01:01:36 TNAS1 kernel: ? mntput_no_expire+0x4a/0x270
Nov 9 01:01:36 TNAS1 kernel: ? audit_reset_context+0x232/0x300
Nov 9 01:01:36 TNAS1 kernel: ? syscall_exit_to_user_mode_prepare+0x148/0x170
Nov 9 01:01:36 TNAS1 kernel: ? syscall_exit_to_user_mode+0x10/0x1f0
Nov 9 01:01:36 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:36 TNAS1 kernel: ? syscall_exit_to_user_mode+0x10/0x1f0
Nov 9 01:01:36 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:36 TNAS1 kernel: ? do_user_addr_fault+0x379/0x670
Nov 9 01:01:36 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:01:36 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:36 TNAS1 kernel: RIP: 0033:0x7feb5e885a97
Nov 9 01:01:36 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7feb5e885a6d.
Nov 9 01:01:36 TNAS1 kernel: RSP: 002b:00007fff9e176738 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
Nov 9 01:01:36 TNAS1 kernel: RAX: ffffffffffffffda RBX: 00007fff9e17850c RCX: 00007feb5e885a97
Nov 9 01:01:36 TNAS1 kernel: RDX: 00007fff9e176b20 RSI: 00007fff9e176b08 RDI: 00007fff9e176740
Nov 9 01:01:36 TNAS1 kernel: RBP: 00007fff9e176860 R08: 00007fff9e178eef R09: 00007feb5e7ae740
Nov 9 01:01:36 TNAS1 kernel: R10: 00007feb5e7c2e70 R11: 0000000000000202 R12: 00007fff9e176b08
Nov 9 01:01:36 TNAS1 kernel: R13: 00007fff9e176b20 R14: 00007fff9e178ee7 R15: 0000000000000001
Nov 9 01:01:36 TNAS1 kernel:
Nov 9 01:01:36 TNAS1 kernel: Mem-Info:
Nov 9 01:01:36 TNAS1 kernel: active_anon:328144 inactive_anon:7376402 isolated_anon:0
active_file:3480 inactive_file:1015 isolated_file:0
unevictable:1000 dirty:82 writeback:3
slab_reclaimable:58826 slab_unreclaimable:82918
mapped:8382 shmem:92861 pagetables:17135
sec_pagetables:637 bounce:0
kernel_misc_reclaimable:0
free:52289 free_pcp:5512 free_cma:0
Nov 9 01:01:36 TNAS1 kernel: Node 0 active_anon:1312576kB inactive_anon:29505900kB active_file:13920kB inactive_file:4060kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:33528kB dirty:328kB writeback:12kB shmem:371444kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:2072576kB writeback_tmp:0kB kernel_stack:12608kB pagetables:68540kB sec_pagetables:2548kB all_unreclaimable? no
Nov 9 01:01:36 TNAS1 kernel: Node 0 DMA free:32kB boost:2048kB min:2076kB low:2088kB high:2100kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11232kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:36 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:36 TNAS1 kernel: Node 0 DMA32 free:8108kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3469164kB active_file:12kB inactive_file:104kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:7704kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:36 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:36 TNAS1 kernel: Node 0 Normal free:201016kB boost:88256kB min:148648kB low:177908kB high:207168kB reserved_highatomic:0KB active_anon:1312572kB inactive_anon:26025960kB active_file:13560kB inactive_file:5020kB unevictable:4000kB writepending:356kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:13336kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:36 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:36 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:36 TNAS1 kernel: Node 0 DMA32: 84*4kB (UME) 64*8kB (UE) 63*16kB (UE) 36*32kB (UME) 37*64kB (UME) 14*128kB (UME) 0*256kB 2*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 8192kB
Nov 9 01:01:36 TNAS1 kernel: Node 0 Normal: 869*4kB (UE) 5210*8kB (UME) 4084*16kB (UE) 2555*32kB (UME) 55*64kB (UME) 1*128kB (M) 0*256kB 2*512kB (M) 2*1024kB (M) 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 201028kB
Nov 9 01:01:36 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:36 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:36 TNAS1 kernel: 97550 total pagecache pages
Nov 9 01:01:36 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:36 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:36 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:36 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:36 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:36 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:36 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:36 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:36 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:36 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:36 TNAS1 kernel: [ 728] 104 728 2091 449 256 193 0 65536 0 -900 dbus-daemon
Nov 9 01:01:36 TNAS1 kernel: [ 741] 0 741 11506 709 224 485 0 102400 0 -250 systemd-journal
Nov 9 01:01:36 TNAS1 kernel: [ 763] 0 763 6768 755 512 243 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:36 TNAS1 kernel: [ 1630] 0 1630 7973216 7106317 7104426 1827 64 57704448 0 -1000 asyncio_loop
Nov 9 01:01:36 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:36 TNAS1 kernel: [ 1681] 0 1681 485024 35702 35252 450 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:36 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:36 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:36 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:36 TNAS1 kernel: [ 3516] 0 3516 21731 350 94 256 0 57344 0 -1000 auditd
Nov 9 01:01:36 TNAS1 kernel: [ 3540] 0 3540 27526 4813 4148 665 0 114688 0 0 truenas_audit_h
Nov 9 01:01:36 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:36 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:36 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:36 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:36 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:36 TNAS1 kernel: [ 3599] 0 3599 5605 390 320 70 0 86016 0 0 certmonger
Nov 9 01:01:36 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:36 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:36 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:36 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:36 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:36 TNAS1 kernel: [ 3681] 0 3681 280105 6798 6236 562 0 577536 0 0 syslog-ng
Nov 9 01:01:36 TNAS1 kernel: [ 3691] 0 3691 12416 560 224 336 0 94208 0 0 systemd-logind
Nov 9 01:01:36 TNAS1 kernel: [ 3695] 0 3695 60178 496 256 240 0 86016 0 0 zed
Nov 9 01:01:36 TNAS1 kernel: [ 3708] 0 3708 523983 4212 4140 72 0 327680 0 -999 containerd
Nov 9 01:01:36 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:36 TNAS1 kernel: [ 3809] 122 3809 3089 379 121 258 0 49152 0 0 usbhid-ups
Nov 9 01:01:36 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:36 TNAS1 kernel: [ 3962] 122 3962 1695 202 96 106 0 57344 0 0 upsd
Nov 9 01:01:36 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:36 TNAS1 kernel: [ 3995] 0 3995 74342 15060 14472 588 0 217088 0 0 cli
Nov 9 01:01:36 TNAS1 kernel: [ 4009] 122 4009 2127 305 96 209 0 53248 0 0 upsmon
Nov 9 01:01:36 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:36 TNAS1 kernel: [ 4132] 0 4132 4136 667 224 443 0 69632 0 0 systemd-machine
Nov 9 01:01:36 TNAS1 kernel: [ 4133] 0 4133 427247 3578 2880 698 0 339968 0 0 libvirtd
Nov 9 01:01:36 TNAS1 kernel: [ 5386] 105 5386 1781 231 96 135 0 49152 0 0 avahi-daemon
Nov 9 01:01:36 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:36 TNAS1 kernel: [ 5692] 1 5692 8070 4299 3840 459 0 98304 0 0 wsdd.py
Nov 9 01:01:36 TNAS1 kernel: [ 5694] 0 5694 17601 709 544 165 0 118784 0 0 nmbd
Nov 9 01:01:36 TNAS1 kernel: [ 6054] 0 6054 16985 813 448 365 0 114688 0 -900 virtlogd
Nov 9 01:01:36 TNAS1 kernel: [ 6171] 986 6171 802276 295139 294906 233 0 2904064 0 0 qemu-system-x86
Nov 9 01:01:36 TNAS1 kernel: [ 6672] 0 6672 13921 5761 5472 289 0 139264 0 -1000 websockify
Nov 9 01:01:36 TNAS1 kernel: [ 6757] 999 6757 116708 29612 29218 330 64 475136 0 -900 netdata
Nov 9 01:01:36 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:36 TNAS1 kernel: [ 7021] 999 7021 22248 5272 4846 426 0 147456 0 -900 python.d.plugin
Nov 9 01:01:36 TNAS1 kernel: [ 7210] 0 7210 18963 953 608 249 96 131072 0 0 winbindd
Nov 9 01:01:36 TNAS1 kernel: [ 7212] 0 7212 18982 870 640 192 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:36 TNAS1 kernel: [ 7230] 0 7230 13921 5697 5413 284 0 131072 0 -1000 websockify
Nov 9 01:01:36 TNAS1 kernel: [ 7371] 33 7371 7296 856 567 289 0 69632 0 0 nginx
Nov 9 01:01:36 TNAS1 kernel: [ 10337] 0 10337 4476 644 416 228 0 73728 0 0 sshd
Nov 9 01:01:36 TNAS1 kernel: [ 10339] 0 10339 4477 718 448 270 0 81920 0 0 sshd
Nov 9 01:01:36 TNAS1 kernel: [ 10342] 0 10342 4941 1069 608 461 0 86016 0 100 systemd
Nov 9 01:01:36 TNAS1 kernel: [ 10343] 0 10343 42782 1639 1482 157 0 94208 0 100 (sd-pam)
Nov 9 01:01:36 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:36 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:36 TNAS1 kernel: [ 11235] 0 11235 19579 892 640 177 75 135168 0 0 wb-idmap
Nov 9 01:01:36 TNAS1 kernel: [1629815] 0 1629815 20194 1495 608 162 725 147456 0 0 smbd
Nov 9 01:01:36 TNAS1 kernel: [1629817] 0 1629817 19634 760 630 130 0 131072 0 0 smbd-notifyd
Nov 9 01:01:36 TNAS1 kernel: [1629818] 0 1629818 19638 705 630 75 0 135168 0 0 smbd-cleanupd
Nov 9 01:01:36 TNAS1 kernel: [1632448] 0 1632448 34145 6351 2476 103 3772 212992 0 0 smbd[192.168.1.
Nov 9 01:01:36 TNAS1 kernel: [1636144] 0 1636144 33944 12502 8678 142 3682 262144 0 0 smbd[fe80::3b34
Nov 9 01:01:36 TNAS1 kernel: [1653950] 0 1653950 34260 7712 3997 105 3610 225280 0 0 smbd[192.168.1.
Nov 9 01:01:36 TNAS1 kernel: [2479756] 999 2479756 1113 417 160 257 0 49152 0 -900 bash
Nov 9 01:01:36 TNAS1 kernel: [2709452] 0 2709452 94214 221 32 189 0 86016 0 0 nscd
Nov 9 01:01:36 TNAS1 kernel: [2773557] 0 2773557 206757 12604 12274 330 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:36 TNAS1 kernel: [2773561] 0 2773561 207385 13362 12971 391 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:36 TNAS1 kernel: [2776151] 0 2776151 207108 13226 12886 340 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:36 TNAS1 kernel: [2777444] 0 2777444 206501 12625 12278 347 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:36 TNAS1 kernel: [2777446] 0 2777446 207155 12916 12561 355 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:36 TNAS1 kernel: [2782070] 999 2782070 623 223 0 223 0 40960 0 -900 timeout
Nov 9 01:01:36 TNAS1 kernel: [2782071] 999 2782071 1 0 0 0 0 12288 0 -900 upsc
Nov 9 01:01:36 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/user@0.service/init.scope,task=(sd-pam),pid=10343,uid=0
Nov 9 01:01:36 TNAS1 kernel: Out of memory: Killed process 10343 ((sd-pam)) total-vm:171128kB, anon-rss:5928kB, file-rss:628kB, shmem-rss:0kB, UID:0 pgtables:92kB oom_score_adj:100
Nov 9 01:01:36 TNAS1 systemd[10342]: init.scope: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:36 TNAS1 systemd[10342]: init.scope: Failed with result 'oom-kill'.
Nov 9 01:01:36 TNAS1 systemd[1]: user@0.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:39 TNAS1 kernel: bash invoked oom-killer: gfp_mask=0x500cc2(GFP_HIGHUSER|__GFP_ACCOUNT), order=0, oom_score_adj=-900
Nov 9 01:01:39 TNAS1 kernel: CPU: 1 UID: 999 PID: 2782114 Comm: bash Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:39 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:39 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:39 TNAS1 kernel: Call Trace:
Nov 9 01:01:39 TNAS1 kernel:
Nov 9 01:01:39 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:39 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:39 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:39 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:39 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:39 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:39 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:39 TNAS1 kernel: pipe_write+0x3a9/0x640
Nov 9 01:01:39 TNAS1 kernel: vfs_write+0x3ad/0x450
Nov 9 01:01:39 TNAS1 kernel: ksys_write+0xbb/0xf0
Nov 9 01:01:39 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:39 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:01:39 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:39 TNAS1 kernel: RIP: 0033:0x7fe3738c02c0
Nov 9 01:01:39 TNAS1 kernel: Code: 40 00 48 8b 15 41 9b 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 80 3d 21 23 0e 00 00 74 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 48 89
Nov 9 01:01:39 TNAS1 kernel: RSP: 002b:00007ffe09353928 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
Nov 9 01:01:39 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007fe3738c02c0
Nov 9 01:01:39 TNAS1 kernel: RDX: 0000000000000007 RSI: 000055cbcc0a24a0 RDI: 0000000000000001
Nov 9 01:01:39 TNAS1 kernel: RBP: 000055cbcc0a24a0 R08: 00007ffe093539c0 R09: 0000000000000073
Nov 9 01:01:39 TNAS1 kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000007
Nov 9 01:01:39 TNAS1 kernel: R13: 00007fe37399b760 R14: 0000000000000007 R15: 00007fe3739969e0
Nov 9 01:01:39 TNAS1 kernel:
Nov 9 01:01:39 TNAS1 kernel: Mem-Info:
Nov 9 01:01:39 TNAS1 kernel: active_anon:328145 inactive_anon:7408674 isolated_anon:0
active_file:59 inactive_file:45 isolated_file:0
unevictable:1000 dirty:1 writeback:47
slab_reclaimable:58772 slab_unreclaimable:82738
mapped:4453 shmem:92861 pagetables:17220
sec_pagetables:637 bounce:0
kernel_misc_reclaimable:0
free:30044 free_pcp:3 free_cma:95
Nov 9 01:01:39 TNAS1 kernel: Node 0 active_anon:1312580kB inactive_anon:29634696kB active_file:528kB inactive_file:156kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:18104kB dirty:296kB writeback:188kB shmem:371444kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:2072576kB writeback_tmp:0kB kernel_stack:12864kB pagetables:68880kB sec_pagetables:2548kB all_unreclaimable? no
Nov 9 01:01:39 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:39 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:39 TNAS1 kernel: Node 0 DMA32 free:6724kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3478676kB active_file:0kB inactive_file:4kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:40 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:40 TNAS1 kernel: Node 0 Normal free:110400kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:0KB active_anon:1312576kB inactive_anon:26143452kB active_file:852kB inactive_file:0kB unevictable:4000kB writepending:272kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:2280kB local_pcp:0kB free_cma:380kB
Nov 9 01:01:40 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:40 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:40 TNAS1 kernel: Node 0 DMA32: 37*4kB (UME) 34*8kB (UME) 55*16kB (UME) 35*32kB (UME) 41*64kB (UME) 9*128kB (ME) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6708kB
Nov 9 01:01:40 TNAS1 kernel: Node 0 Normal: 1*4kB (E) 0*8kB 2124*16kB (UE) 2359*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 109476kB
Nov 9 01:01:40 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:40 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:40 TNAS1 kernel: 93333 total pagecache pages
Nov 9 01:01:40 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:40 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:40 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:40 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:40 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:40 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:40 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:40 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:40 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:40 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:40 TNAS1 kernel: [ 728] 104 728 2091 321 256 65 0 65536 0 -900 dbus-daemon
Nov 9 01:01:40 TNAS1 kernel: [ 741] 0 741 11506 490 224 266 0 106496 0 -250 systemd-journal
Nov 9 01:01:40 TNAS1 kernel: [ 763] 0 763 6768 627 512 115 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:40 TNAS1 kernel: [ 1630] 0 1630 8000352 7137062 7136682 316 64 57962496 0 -1000 asyncio_loop
Nov 9 01:01:40 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:40 TNAS1 kernel: [ 1681] 0 1681 485024 35350 35252 98 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:40 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:40 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:40 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:40 TNAS1 kernel: [ 3516] 0 3516 21731 222 94 128 0 57344 0 -1000 auditd
Nov 9 01:01:40 TNAS1 kernel: [ 3540] 0 3540 27526 4301 4148 153 0 114688 0 0 truenas_audit_h
Nov 9 01:01:40 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:40 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:40 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:40 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:40 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:40 TNAS1 kernel: [ 3599] 0 3599 5605 358 320 38 0 86016 0 0 certmonger
Nov 9 01:01:40 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:40 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:40 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:40 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:40 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:40 TNAS1 kernel: [ 3681] 0 3681 282154 6350 6236 114 0 577536 0 0 syslog-ng
Nov 9 01:01:40 TNAS1 kernel: [ 3691] 0 3691 12416 336 224 112 0 94208 0 0 systemd-logind
Nov 9 01:01:40 TNAS1 kernel: [ 3695] 0 3695 60178 336 256 80 0 86016 0 0 zed
Nov 9 01:01:40 TNAS1 kernel: [ 3708] 0 3708 523983 4180 4140 40 0 327680 0 -999 containerd
Nov 9 01:01:40 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:40 TNAS1 kernel: [ 3809] 122 3809 3089 192 121 71 0 49152 0 0 usbhid-ups
Nov 9 01:01:40 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:40 TNAS1 kernel: [ 3962] 122 3962 1695 197 96 101 0 57344 0 0 upsd
Nov 9 01:01:40 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:40 TNAS1 kernel: [ 3995] 0 3995 74342 14699 14472 227 0 217088 0 0 cli
Nov 9 01:01:40 TNAS1 kernel: [ 4009] 122 4009 2127 241 96 145 0 53248 0 0 upsmon
Nov 9 01:01:40 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:40 TNAS1 kernel: [ 4132] 0 4132 4136 315 224 91 0 69632 0 0 systemd-machine
Nov 9 01:01:40 TNAS1 kernel: [ 4133] 0 4133 427247 2917 2880 37 0 339968 0 0 libvirtd
Nov 9 01:01:40 TNAS1 kernel: [ 5386] 105 5386 1781 135 96 39 0 49152 0 0 avahi-daemon
Nov 9 01:01:40 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:40 TNAS1 kernel: [ 5692] 1 5692 8070 3979 3840 139 0 98304 0 0 wsdd.py
Nov 9 01:01:40 TNAS1 kernel: [ 5694] 0 5694 17601 549 544 5 0 118784 0 0 nmbd
Nov 9 01:01:40 TNAS1 kernel: [ 6054] 0 6054 16985 525 448 77 0 114688 0 -900 virtlogd
Nov 9 01:01:40 TNAS1 kernel: [ 6171] 986 6171 812521 294963 294906 57 0 2912256 0 0 qemu-system-x86
Nov 9 01:01:40 TNAS1 kernel: [ 6672] 0 6672 13921 5597 5472 125 0 139264 0 -1000 websockify
Nov 9 01:01:40 TNAS1 kernel: [ 6757] 999 6757 116708 29359 29218 77 64 475136 0 -900 netdata
Nov 9 01:01:40 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:40 TNAS1 kernel: [ 7021] 999 7021 22248 4944 4846 98 0 147456 0 -900 python.d.plugin
Nov 9 01:01:40 TNAS1 kernel: [ 7210] 0 7210 18963 828 608 124 96 131072 0 0 winbindd
Nov 9 01:01:40 TNAS1 kernel: [ 7212] 0 7212 18982 774 640 96 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:40 TNAS1 kernel: [ 7230] 0 7230 13921 5535 5413 122 0 131072 0 -1000 websockify
Nov 9 01:01:40 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:40 TNAS1 kernel: [ 10337] 0 10337 4476 484 416 68 0 73728 0 0 sshd
Nov 9 01:01:40 TNAS1 kernel: [ 10339] 0 10339 4477 558 448 110 0 81920 0 0 sshd
Nov 9 01:01:40 TNAS1 kernel: [ 10342] 0 10342 4941 717 608 109 0 86016 0 100 systemd
Nov 9 01:01:40 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:40 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:40 TNAS1 kernel: [ 11235] 0 11235 19579 796 640 81 75 135168 0 0 wb-idmap
Nov 9 01:01:40 TNAS1 kernel: [1629815] 0 1629815 20194 1495 608 162 725 147456 0 0 smbd
Nov 9 01:01:40 TNAS1 kernel: [1629817] 0 1629817 19634 696 630 66 0 131072 0 0 smbd-notifyd
Nov 9 01:01:40 TNAS1 kernel: [1629818] 0 1629818 19638 705 630 75 0 135168 0 0 smbd-cleanupd
Nov 9 01:01:40 TNAS1 kernel: [1632448] 0 1632448 34145 6351 2476 103 3772 212992 0 0 smbd[192.168.1.
Nov 9 01:01:40 TNAS1 kernel: [1636144] 0 1636144 33944 12502 8678 142 3682 262144 0 0 smbd[fe80::3b34
Nov 9 01:01:40 TNAS1 kernel: [1653950] 0 1653950 34260 7712 3997 105 3610 225280 0 0 smbd[192.168.1.
Nov 9 01:01:40 TNAS1 kernel: [2479756] 999 2479756 1113 245 160 85 0 49152 0 -900 bash
Nov 9 01:01:40 TNAS1 kernel: [2709452] 0 2709452 94214 61 32 29 0 86016 0 0 nscd
Nov 9 01:01:40 TNAS1 kernel: [2773557] 0 2773557 206757 12412 12274 138 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:40 TNAS1 kernel: [2773561] 0 2773561 207385 13106 12971 135 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:40 TNAS1 kernel: [2776151] 0 2776151 207108 13034 12886 148 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:40 TNAS1 kernel: [2777444] 0 2777444 206501 12337 12278 59 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:40 TNAS1 kernel: [2777446] 0 2777446 207155 12692 12561 131 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:40 TNAS1 kernel: [2782113] 999 2782113 1113 278 204 74 0 49152 0 -900 bash
Nov 9 01:01:40 TNAS1 kernel: [2782114] 999 2782114 1113 280 204 76 0 49152 0 -900 bash
Nov 9 01:01:40 TNAS1 kernel: [2782115] 999 2782115 627 135 0 135 0 40960 0 -900 tr
Nov 9 01:01:40 TNAS1 kernel: [2782116] 999 2782116 851 150 0 150 0 45056 0 -900 sed
Nov 9 01:01:40 TNAS1 kernel: [2782117] 999 2782117 627 137 0 137 0 49152 0 -900 tr
Nov 9 01:01:40 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/user@0.service/init.scope,task=systemd,pid=10342,uid=0
Nov 9 01:01:40 TNAS1 kernel: Out of memory: Killed process 10342 (systemd) total-vm:19764kB, anon-rss:2432kB, file-rss:436kB, shmem-rss:0kB, UID:0 pgtables:84kB oom_score_adj:100
Nov 9 01:01:40 TNAS1 kernel: sed invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=-900
Nov 9 01:01:40 TNAS1 kernel: CPU: 2 UID: 999 PID: 2782116 Comm: sed Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:40 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:40 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:40 TNAS1 kernel: Call Trace:
Nov 9 01:01:40 TNAS1 kernel:
Nov 9 01:01:40 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:40 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:40 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:40 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:40 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:40 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:40 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:40 TNAS1 kernel: __pmd_alloc+0x34/0x210
Nov 9 01:01:40 TNAS1 kernel: __handle_mm_fault+0x497/0xfd0
Nov 9 01:01:40 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:01:40 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:01:40 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:01:40 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:01:40 TNAS1 kernel: RIP: 0033:0x7f184f67b0d6
Nov 9 01:01:40 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7f184f67b0ac.
Nov 9 01:01:40 TNAS1 kernel: RSP: 002b:00007ffe9ad4c250 EFLAGS: 00010202
Nov 9 01:01:40 TNAS1 kernel: RAX: ffffffffffffffff RBX: 00007f184f7b6c60 RCX: 0000556bc9c9d000
Nov 9 01:01:40 TNAS1 kernel: RDX: 0000000000021000 RSI: 0000000000000fff RDI: 0000000000000000
Nov 9 01:01:40 TNAS1 kernel: RBP: 0000000000000290 R08: 0000556bc9cbe000 R09: 0000000000021001
Nov 9 01:01:40 TNAS1 kernel: R10: 00007f184f80f7d0 R11: 00007f184f7b6cc0 R12: 0000000000000000
Nov 9 01:01:40 TNAS1 kernel: R13: 00007f184f7b6cc0 R14: 0000000000000000 R15: 00000000000002b0
Nov 9 01:01:40 TNAS1 kernel:
Nov 9 01:01:40 TNAS1 kernel: Mem-Info:
Nov 9 01:01:40 TNAS1 kernel: active_anon:328145 inactive_anon:7408210 isolated_anon:0
active_file:95 inactive_file:173 isolated_file:0
unevictable:1000 dirty:39 writeback:61
slab_reclaimable:58763 slab_unreclaimable:82700
mapped:4616 shmem:92861 pagetables:17211
sec_pagetables:637 bounce:0
kernel_misc_reclaimable:0
free:30311 free_pcp:0 free_cma:0
Nov 9 01:01:40 TNAS1 kernel: Node 0 active_anon:1312580kB inactive_anon:29632840kB active_file:380kB inactive_file:692kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:18464kB dirty:156kB writeback:244kB shmem:371444kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:2072576kB writeback_tmp:0kB kernel_stack:12832kB pagetables:68844kB sec_pagetables:2548kB all_unreclaimable? no
Nov 9 01:01:40 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:40 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:40 TNAS1 kernel: Node 0 DMA32 free:6452kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3478980kB active_file:16kB inactive_file:0kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:40 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:40 TNAS1 kernel: Node 0 Normal free:114776kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:0KB active_anon:1312576kB inactive_anon:26142276kB active_file:308kB inactive_file:1068kB unevictable:4000kB writepending:400kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:40 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:40 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:40 TNAS1 kernel: Node 0 DMA32: 38*4kB (UME) 34*8kB (UME) 55*16kB (UME) 35*32kB (UME) 41*64kB (UME) 9*128kB (ME) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6712kB
Nov 9 01:01:40 TNAS1 kernel: Node 0 Normal: 627*4kB (UMEC) 235*8kB (UMEC) 2211*16kB (UME) 2360*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 115284kB
Nov 9 01:01:40 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:40 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:40 TNAS1 kernel: 93350 total pagecache pages
Nov 9 01:01:40 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:40 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:40 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:40 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:40 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:40 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:40 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:40 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:40 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:40 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:40 TNAS1 kernel: [ 728] 104 728 2091 321 256 65 0 65536 0 -900 dbus-daemon
Nov 9 01:01:40 TNAS1 kernel: [ 741] 0 741 11506 394 224 170 0 106496 0 -250 systemd-journal
Nov 9 01:01:40 TNAS1 kernel: [ 763] 0 763 6768 627 512 115 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:40 TNAS1 kernel: [ 1630] 0 1630 8000608 7137063 7136778 221 64 57966592 0 -1000 asyncio_loop
Nov 9 01:01:40 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:40 TNAS1 kernel: [ 1681] 0 1681 485024 35391 35252 139 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:40 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:40 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:40 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:40 TNAS1 kernel: [ 3516] 0 3516 21731 222 94 128 0 57344 0 -1000 auditd
Nov 9 01:01:40 TNAS1 kernel: [ 3540] 0 3540 27526 4301 4148 153 0 114688 0 0 truenas_audit_h
Nov 9 01:01:40 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:40 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:40 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:40 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:40 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:40 TNAS1 kernel: [ 3599] 0 3599 5605 358 320 38 0 86016 0 0 certmonger
Nov 9 01:01:40 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:40 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:40 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:40 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:40 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:40 TNAS1 kernel: [ 3681] 0 3681 282154 6350 6236 114 0 577536 0 0 syslog-ng
Nov 9 01:01:40 TNAS1 kernel: [ 3691] 0 3691 12416 336 224 112 0 94208 0 0 systemd-logind
Nov 9 01:01:40 TNAS1 kernel: [ 3695] 0 3695 60178 336 256 80 0 86016 0 0 zed
Nov 9 01:01:40 TNAS1 kernel: [ 3708] 0 3708 523983 4180 4140 40 0 327680 0 -999 containerd
Nov 9 01:01:40 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:40 TNAS1 kernel: [ 3809] 122 3809 3089 192 121 71 0 49152 0 0 usbhid-ups
Nov 9 01:01:40 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:40 TNAS1 kernel: [ 3962] 122 3962 1695 197 96 101 0 57344 0 0 upsd
Nov 9 01:01:40 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:40 TNAS1 kernel: [ 3995] 0 3995 74342 14603 14472 131 0 217088 0 0 cli
Nov 9 01:01:40 TNAS1 kernel: [ 4009] 122 4009 2127 241 96 145 0 53248 0 0 upsmon
Nov 9 01:01:40 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:40 TNAS1 kernel: [ 4132] 0 4132 4136 315 224 91 0 69632 0 0 systemd-machine
Nov 9 01:01:40 TNAS1 kernel: [ 4133] 0 4133 427247 2917 2880 37 0 339968 0 0 libvirtd
Nov 9 01:01:40 TNAS1 kernel: [ 5386] 105 5386 1781 135 96 39 0 49152 0 0 avahi-daemon
Nov 9 01:01:40 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:40 TNAS1 kernel: [ 5692] 1 5692 8070 3979 3840 139 0 98304 0 0 wsdd.py
Nov 9 01:01:40 TNAS1 kernel: [ 5694] 0 5694 17601 614 544 70 0 118784 0 0 nmbd
Nov 9 01:01:40 TNAS1 kernel: [ 6054] 0 6054 16985 525 448 77 0 114688 0 -900 virtlogd
Nov 9 01:01:40 TNAS1 kernel: [ 6171] 986 6171 812521 295027 294906 121 0 2912256 0 0 qemu-system-x86
Nov 9 01:01:40 TNAS1 kernel: [ 6672] 0 6672 13921 5597 5472 125 0 139264 0 -1000 websockify
Nov 9 01:01:40 TNAS1 kernel: [ 6757] 999 6757 116708 29391 29218 109 64 475136 0 -900 netdata
Nov 9 01:01:40 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:40 TNAS1 kernel: [ 7021] 999 7021 22248 4944 4846 98 0 147456 0 -900 python.d.plugin
Nov 9 01:01:40 TNAS1 kernel: [ 7210] 0 7210 18963 828 608 124 96 131072 0 0 winbindd
Nov 9 01:01:40 TNAS1 kernel: [ 7212] 0 7212 18982 774 640 96 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:40 TNAS1 kernel: [ 7230] 0 7230 13921 5535 5413 122 0 131072 0 -1000 websockify
Nov 9 01:01:40 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:40 TNAS1 kernel: [ 10337] 0 10337 4476 484 416 68 0 73728 0 0 sshd
Nov 9 01:01:40 TNAS1 kernel: [ 10339] 0 10339 4477 558 448 110 0 81920 0 0 sshd
Nov 9 01:01:40 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:40 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:40 TNAS1 kernel: [ 11235] 0 11235 19579 796 640 81 75 135168 0 0 wb-idmap
Nov 9 01:01:40 TNAS1 kernel: [1629815] 0 1629815 20194 1495 608 162 725 147456 0 0 smbd
Nov 9 01:01:40 TNAS1 kernel: [1629817] 0 1629817 19634 696 630 66 0 131072 0 0 smbd-notifyd
Nov 9 01:01:40 TNAS1 kernel: [1629818] 0 1629818 19638 705 630 75 0 135168 0 0 smbd-cleanupd
Nov 9 01:01:40 TNAS1 kernel: [1632448] 0 1632448 34145 6351 2476 103 3772 212992 0 0 smbd[192.168.1.
Nov 9 01:01:40 TNAS1 kernel: [1636144] 0 1636144 33944 12502 8678 142 3682 262144 0 0 smbd[fe80::3b34
Nov 9 01:01:40 TNAS1 kernel: [1653950] 0 1653950 34260 7712 3997 105 3610 225280 0 0 smbd[192.168.1.
Nov 9 01:01:40 TNAS1 kernel: [2479756] 999 2479756 1113 245 160 85 0 49152 0 -900 bash
Nov 9 01:01:40 TNAS1 kernel: [2709452] 0 2709452 94214 61 32 29 0 86016 0 0 nscd
Nov 9 01:01:40 TNAS1 kernel: [2773557] 0 2773557 206757 12412 12274 138 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:40 TNAS1 kernel: [2773561] 0 2773561 207385 13106 12971 135 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:40 TNAS1 kernel: [2776151] 0 2776151 207108 13034 12886 148 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:40 TNAS1 kernel: [2777444] 0 2777444 206501 12337 12278 59 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:40 TNAS1 kernel: [2777446] 0 2777446 207155 12692 12561 131 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:40 TNAS1 kernel: [2782113] 999 2782113 1113 278 204 74 0 49152 0 -900 bash
Nov 9 01:01:40 TNAS1 kernel: [2782114] 999 2782114 1113 280 204 76 0 49152 0 -900 bash
Nov 9 01:01:40 TNAS1 kernel: [2782115] 999 2782115 627 135 0 135 0 40960 0 -900 tr
Nov 9 01:01:40 TNAS1 kernel: [2782116] 999 2782116 851 118 0 118 0 45056 0 -900 sed
Nov 9 01:01:40 TNAS1 kernel: [2782117] 999 2782117 627 73 0 73 0 49152 0 -900 tr
Nov 9 01:01:40 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/machine.slice/machine-qemu\x2d1\x2d2NixVM.scope/libvirt,task=qemu-system-x86,pid=6171,uid=986
Nov 9 01:01:40 TNAS1 kernel: Out of memory: Killed process 6171 (qemu-system-x86) total-vm:3250084kB, anon-rss:1179624kB, file-rss:484kB, shmem-rss:0kB, UID:986 pgtables:2844kB oom_score_adj:0
Nov 9 01:01:40 TNAS1 kernel: br0: port 2(vnet0) entered disabled state
Nov 9 01:01:40 TNAS1 kernel: vnet0 (unregistering): left allmulticast mode
Nov 9 01:01:40 TNAS1 kernel: vnet0 (unregistering): left promiscuous mode
Nov 9 01:01:40 TNAS1 kernel: br0: port 2(vnet0) entered disabled state
Nov 9 01:01:40 TNAS1 systemd[1]: user@0.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:40 TNAS1 systemd[1]: machine-qemu\x2d1\x2d2NixVM.scope: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:40 TNAS1 systemd[1]: user@0.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:40 TNAS1 systemd[1]: user@0.service: Failed with result 'signal'.
Nov 9 01:01:40 TNAS1 systemd[1]: machine-qemu\x2d1\x2d2NixVM.scope: Deactivated successfully.
Nov 9 01:01:40 TNAS1 systemd[1]: machine-qemu\x2d1\x2d2NixVM.scope: Consumed 33min 9.684s CPU time.
Nov 9 01:01:40 TNAS1 systemd[1]: machine.slice: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:40 TNAS1 nscd[2709452]: 2709452 monitoring file `/etc/nsswitch.conf` (4)
Nov 9 01:01:40 TNAS1 nscd[2709452]: 2709452 monitoring directory `/etc` (2)
Nov 9 01:01:40 TNAS1 nscd[2709452]: 2709452 monitoring file `/etc/resolv.conf` (3)
Nov 9 01:01:40 TNAS1 nscd[2709452]: 2709452 monitoring directory `/etc` (2)
Nov 9 01:01:50 TNAS1 kernel: IoThread invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-1000
Nov 9 01:01:50 TNAS1 kernel: CPU: 5 UID: 0 PID: 4147 Comm: IoThread Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:50 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:50 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:50 TNAS1 kernel: Call Trace:
Nov 9 01:01:50 TNAS1 kernel:
Nov 9 01:01:50 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:50 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:50 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:50 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:50 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:50 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:50 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:50 TNAS1 kernel: folio_alloc_noprof+0x5b/0xb0
Nov 9 01:01:50 TNAS1 kernel: __filemap_get_folio+0x1aa/0x2e0
Nov 9 01:01:50 TNAS1 kernel: filemap_fault+0x647/0xd40
Nov 9 01:01:50 TNAS1 kernel: __do_fault+0x33/0x170
Nov 9 01:01:50 TNAS1 kernel: do_fault+0x2a8/0x4d0
Nov 9 01:01:50 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:01:50 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:01:50 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:01:50 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:01:50 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:01:50 TNAS1 kernel: RIP: 0033:0x7ff07f3f2630
Nov 9 01:01:50 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7ff07f3f2606.
Nov 9 01:01:50 TNAS1 kernel: RSP: 002b:00007fefd57f9648 EFLAGS: 00010202
Nov 9 01:01:50 TNAS1 kernel: RAX: 000000000000000e RBX: 0000000000a4db08 RCX: 0000000000a4db28
Nov 9 01:01:50 TNAS1 kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000a4db38
Nov 9 01:01:50 TNAS1 kernel: RBP: 0000000000000000 R08: 0000000000000004 R09: 0000000000670958
Nov 9 01:01:50 TNAS1 kernel: R10: 00007fefd57f9740 R11: 0000000000000246 R12: 0000000000a4db38
Nov 9 01:01:50 TNAS1 kernel: R13: 000000000000006e R14: 0000000000000000 R15: 0000000000000000
Nov 9 01:01:50 TNAS1 kernel:
Nov 9 01:01:50 TNAS1 kernel: Mem-Info:
Nov 9 01:01:50 TNAS1 kernel: active_anon:64892 inactive_anon:7672984 isolated_anon:0
active_file:38 inactive_file:173 isolated_file:44
unevictable:1000 dirty:0 writeback:239
slab_reclaimable:58611 slab_unreclaimable:82283
mapped:4464 shmem:92850 pagetables:17062
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:32134 free_pcp:0 free_cma:0
Nov 9 01:01:50 TNAS1 kernel: Node 0 active_anon:259568kB inactive_anon:30691936kB active_file:152kB inactive_file:692kB unevictable:4000kB isolated(anon):0kB isolated(file):176kB mapped:17856kB dirty:0kB writeback:956kB shmem:371400kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:907264kB writeback_tmp:0kB kernel_stack:12464kB pagetables:68248kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:50 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA32 free:7812kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3479444kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:50 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:50 TNAS1 kernel: Node 0 Normal free:122048kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:0KB active_anon:259564kB inactive_anon:27201320kB active_file:56kB inactive_file:428kB unevictable:4000kB writepending:956kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:50 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA32: 24*4kB (UE) 32*8kB (UE) 52*16kB (UME) 36*32kB (UME) 41*64kB (UME) 9*128kB (ME) 2*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6624kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 Normal: 1864*4kB (UME) 420*8kB (UME) 2133*16kB (UME) 2396*32kB (UME) 13*64kB (UM) 6*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 123216kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:50 TNAS1 kernel: 93167 total pagecache pages
Nov 9 01:01:50 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:50 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:50 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:50 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:50 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:50 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:50 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:50 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:50 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:50 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:50 TNAS1 kernel: [ 728] 104 728 2091 300 256 44 0 65536 0 -900 dbus-daemon
Nov 9 01:01:50 TNAS1 kernel: [ 741] 0 741 11506 542 224 318 0 106496 0 -250 systemd-journal
Nov 9 01:01:50 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:50 TNAS1 kernel: [ 1630] 0 1630 8308576 7433501 7433354 83 64 60342272 0 -1000 asyncio_loop
Nov 9 01:01:50 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:50 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:50 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:50 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:50 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:50 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:50 TNAS1 kernel: [ 3540] 0 3540 27526 4256 4148 108 0 114688 0 0 truenas_audit_h
Nov 9 01:01:50 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:50 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:50 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:50 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:50 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:50 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:50 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:50 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:50 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:50 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:50 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:50 TNAS1 kernel: [ 3681] 0 3681 282154 6401 6236 165 0 577536 0 0 syslog-ng
Nov 9 01:01:50 TNAS1 kernel: [ 3691] 0 3691 12416 322 224 98 0 94208 0 0 systemd-logind
Nov 9 01:01:50 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:50 TNAS1 kernel: [ 3708] 0 3708 523983 4180 4140 40 0 327680 0 -999 containerd
Nov 9 01:01:50 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:50 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:50 TNAS1 kernel: [ 3962] 122 3962 1695 190 96 94 0 57344 0 0 upsd
Nov 9 01:01:50 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:50 TNAS1 kernel: [ 3995] 0 3995 74342 14536 14472 64 0 217088 0 0 cli
Nov 9 01:01:50 TNAS1 kernel: [ 4009] 122 4009 2127 251 96 155 0 53248 0 0 upsmon
Nov 9 01:01:50 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:50 TNAS1 kernel: [ 4132] 0 4132 4136 280 224 56 0 69632 0 0 systemd-machine
Nov 9 01:01:50 TNAS1 kernel: [ 4133] 0 4133 427247 2902 2880 22 0 339968 0 0 libvirtd
Nov 9 01:01:50 TNAS1 kernel: [ 5386] 105 5386 1781 152 96 56 0 49152 0 0 avahi-daemon
Nov 9 01:01:50 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:50 TNAS1 kernel: [ 5692] 1 5692 8070 3981 3840 141 0 98304 0 0 wsdd.py
Nov 9 01:01:50 TNAS1 kernel: [ 5694] 0 5694 17601 616 544 72 0 118784 0 0 nmbd
Nov 9 01:01:50 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:50 TNAS1 kernel: [ 6672] 0 6672 13921 5555 5472 83 0 139264 0 -1000 websockify
Nov 9 01:01:50 TNAS1 kernel: [ 6757] 999 6757 116708 29330 29218 48 64 475136 0 -900 netdata
Nov 9 01:01:50 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:50 TNAS1 kernel: [ 7021] 999 7021 22248 4872 4846 26 0 147456 0 -900 python.d.plugin
Nov 9 01:01:50 TNAS1 kernel: [ 7210] 0 7210 18963 765 608 61 96 131072 0 0 winbindd
Nov 9 01:01:50 TNAS1 kernel: [ 7212] 0 7212 18982 747 640 69 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:50 TNAS1 kernel: [ 7230] 0 7230 13921 5471 5413 58 0 131072 0 -1000 websockify
Nov 9 01:01:50 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:50 TNAS1 kernel: [ 10337] 0 10337 4476 501 416 85 0 73728 0 0 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 10339] 0 10339 4477 591 448 143 0 81920 0 0 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:50 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 11235] 0 11235 19579 797 640 82 75 135168 0 0 wb-idmap
Nov 9 01:01:50 TNAS1 kernel: [1629815] 0 1629815 20194 1495 608 162 725 147456 0 0 smbd
Nov 9 01:01:50 TNAS1 kernel: [1629817] 0 1629817 19634 696 630 66 0 131072 0 0 smbd-notifyd
Nov 9 01:01:50 TNAS1 kernel: [1629818] 0 1629818 19638 709 630 79 0 135168 0 0 smbd-cleanupd
Nov 9 01:01:50 TNAS1 kernel: [1632448] 0 1632448 34145 6326 2476 78 3772 212992 0 0 smbd[192.168.1.
Nov 9 01:01:50 TNAS1 kernel: [1636144] 0 1636144 33944 12502 8678 142 3682 262144 0 0 smbd[fe80::3b34
Nov 9 01:01:50 TNAS1 kernel: [1653950] 0 1653950 34260 7704 3997 97 3610 225280 0 0 smbd[192.168.1.
Nov 9 01:01:50 TNAS1 kernel: [2479756] 999 2479756 1113 333 160 173 0 49152 0 -900 bash
Nov 9 01:01:50 TNAS1 kernel: [2709452] 0 2709452 94214 54 32 22 0 86016 0 0 nscd
Nov 9 01:01:50 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2776151] 0 2776151 207108 13004 12886 118 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2777444] 0 2777444 206501 12405 12278 127 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2782291] 999 2782291 1113 310 204 106 0 49152 0 -900 bash
Nov 9 01:01:50 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=middlewared.service,mems_allowed=0,global_oom,task_memcg=/system.slice/system-getty.slice/getty@tty1.service,task=cli,pid=3995,uid=0
Nov 9 01:01:50 TNAS1 kernel: Out of memory: Killed process 3995 (cli) total-vm:297368kB, anon-rss:57888kB, file-rss:256kB, shmem-rss:0kB, UID:0 pgtables:212kB oom_score_adj:0
Nov 9 01:01:50 TNAS1 kernel: libvirtd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Nov 9 01:01:50 TNAS1 kernel: CPU: 0 UID: 0 PID: 4133 Comm: libvirtd Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:50 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:50 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:50 TNAS1 kernel: Call Trace:
Nov 9 01:01:50 TNAS1 kernel:
Nov 9 01:01:50 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:50 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:50 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:50 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:50 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:50 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:50 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:50 TNAS1 kernel: ? do_sys_poll+0x383/0x600
Nov 9 01:01:50 TNAS1 kernel: get_free_pages_noprof+0x11/0x40
Nov 9 01:01:50 TNAS1 kernel: __pollwait+0xa5/0x120
Nov 9 01:01:50 TNAS1 kernel: unix_poll+0x29/0x110
Nov 9 01:01:50 TNAS1 kernel: sock_poll+0x54/0xf0
Nov 9 01:01:50 TNAS1 kernel: do_sys_poll+0x2dd/0x600
Nov 9 01:01:50 TNAS1 kernel: ? _copy_to_iter+0xd1/0x6d0
Nov 9 01:01:50 TNAS1 kernel: ? __pfx___pollwait+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: ? __pfx_pollwake+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: ? __pfx_read_tsc+0x10/0x10
Nov 9 01:01:50 TNAS1 kernel: __x64_sys_poll+0xbb/0x140
Nov 9 01:01:50 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:50 TNAS1 kernel: ? __count_memcg_events+0x53/0xf0
Nov 9 01:01:50 TNAS1 kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 9 01:01:50 TNAS1 kernel: ? handle_mm_fault+0xae/0x2d0
Nov 9 01:01:50 TNAS1 kernel: ? do_user_addr_fault+0x17f/0x670
Nov 9 01:01:50 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:01:50 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:50 TNAS1 kernel: RIP: 0033:0x7feba2cb81df
Nov 9 01:01:50 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7feba2cb81b5.
Nov 9 01:01:50 TNAS1 kernel: RSP: 002b:00007ffcf7e8fcb0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
Nov 9 01:01:50 TNAS1 kernel: RAX: ffffffffffffffda RBX: 000000000000000b RCX: 00007feba2cb81df
Nov 9 01:01:50 TNAS1 kernel: RDX: 0000000000000127 RSI: 000000000000000b RDI: 000055a6886634e0
Nov 9 01:01:50 TNAS1 kernel: RBP: 000055a688680dc0 R08: 0000000000000000 R09: 0000000000000000
Nov 9 01:01:50 TNAS1 kernel: R10: 00007feba3612080 R11: 0000000000000293 R12: 000055a6886634e0
Nov 9 01:01:50 TNAS1 kernel: R13: 00007feba3090150 R14: 0000000000000127 R15: 000000000000000c
Nov 9 01:01:50 TNAS1 kernel:
Nov 9 01:01:50 TNAS1 kernel: Mem-Info:
Nov 9 01:01:50 TNAS1 kernel: active_anon:64891 inactive_anon:7658409 isolated_anon:0
active_file:236 inactive_file:186 isolated_file:29
unevictable:1000 dirty:0 writeback:239
slab_reclaimable:58598 slab_unreclaimable:82258
mapped:4477 shmem:92850 pagetables:17009
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:47331 free_pcp:0 free_cma:0
Nov 9 01:01:50 TNAS1 kernel: Node 0 active_anon:259564kB inactive_anon:30633636kB active_file:944kB inactive_file:744kB unevictable:4000kB isolated(anon):0kB isolated(file):116kB mapped:17908kB dirty:0kB writeback:956kB shmem:371400kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:868352kB writeback_tmp:0kB kernel_stack:12400kB pagetables:68036kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:50 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA32 free:9436kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3477332kB active_file:160kB inactive_file:716kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:50 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:50 TNAS1 kernel: Node 0 Normal free:179860kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:0KB active_anon:259560kB inactive_anon:27145068kB active_file:480kB inactive_file:132kB unevictable:4000kB writepending:956kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:50 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA32: 220*4kB (UME) 66*8kB (UME) 64*16kB (UME) 36*32kB (UME) 40*64kB (UME) 3*128kB (ME) 2*256kB (M) 0*512kB 0*1024kB 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 9088kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 Normal: 4593*4kB (UME) 832*8kB (UME) 2269*16kB (UME) 2424*32kB (UME) 24*64kB (UM) 12*128kB (UM) 4*256kB (M) 0*512kB 0*1024kB 0*2048kB 1*4096kB (M) 2*8192kB (M) 1*16384kB (M) 0*32768kB 0*65536kB = 179860kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:50 TNAS1 kernel: 93205 total pagecache pages
Nov 9 01:01:50 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:50 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:50 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:50 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:50 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:50 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:50 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:50 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:50 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:50 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:50 TNAS1 kernel: [ 728] 104 728 2091 300 256 44 0 65536 0 -900 dbus-daemon
Nov 9 01:01:50 TNAS1 kernel: [ 741] 0 741 11506 542 224 318 0 106496 0 -250 systemd-journal
Nov 9 01:01:50 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:50 TNAS1 kernel: [ 1630] 0 1630 8308576 7433469 7433354 51 64 60342272 0 -1000 asyncio_loop
Nov 9 01:01:50 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:50 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:50 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:50 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:50 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:50 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:50 TNAS1 kernel: [ 3540] 0 3540 27526 4256 4148 108 0 114688 0 0 truenas_audit_h
Nov 9 01:01:50 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:50 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:50 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:50 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:50 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:50 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:50 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:50 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:50 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:50 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:50 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:50 TNAS1 kernel: [ 3681] 0 3681 282154 6401 6236 165 0 577536 0 0 syslog-ng
Nov 9 01:01:50 TNAS1 kernel: [ 3691] 0 3691 12416 323 224 99 0 94208 0 0 systemd-logind
Nov 9 01:01:50 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:50 TNAS1 kernel: [ 3708] 0 3708 523983 4180 4140 40 0 327680 0 -999 containerd
Nov 9 01:01:50 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:50 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:50 TNAS1 kernel: [ 3962] 122 3962 1695 190 96 94 0 57344 0 0 upsd
Nov 9 01:01:50 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:50 TNAS1 kernel: [ 4009] 122 4009 2127 251 96 155 0 53248 0 0 upsmon
Nov 9 01:01:50 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:50 TNAS1 kernel: [ 4132] 0 4132 4136 317 224 93 0 69632 0 0 systemd-machine
Nov 9 01:01:50 TNAS1 kernel: [ 4133] 0 4133 427247 2934 2880 54 0 339968 0 0 libvirtd
Nov 9 01:01:50 TNAS1 kernel: [ 5386] 105 5386 1781 152 96 56 0 49152 0 0 avahi-daemon
Nov 9 01:01:50 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:50 TNAS1 kernel: [ 5692] 1 5692 8070 3981 3840 141 0 98304 0 0 wsdd.py
Nov 9 01:01:50 TNAS1 kernel: [ 5694] 0 5694 17601 616 544 72 0 118784 0 0 nmbd
Nov 9 01:01:50 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:50 TNAS1 kernel: [ 6672] 0 6672 13921 5555 5472 83 0 139264 0 -1000 websockify
Nov 9 01:01:50 TNAS1 kernel: [ 6757] 999 6757 116708 29300 29218 18 64 475136 0 -900 netdata
Nov 9 01:01:50 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:50 TNAS1 kernel: [ 7021] 999 7021 22248 4904 4846 58 0 147456 0 -900 python.d.plugin
Nov 9 01:01:50 TNAS1 kernel: [ 7210] 0 7210 18963 765 608 61 96 131072 0 0 winbindd
Nov 9 01:01:50 TNAS1 kernel: [ 7212] 0 7212 18982 747 640 69 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:50 TNAS1 kernel: [ 7230] 0 7230 13921 5439 5413 26 0 131072 0 -1000 websockify
Nov 9 01:01:50 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:50 TNAS1 kernel: [ 10337] 0 10337 4476 501 416 85 0 73728 0 0 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 10339] 0 10339 4477 591 448 143 0 81920 0 0 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:50 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 11235] 0 11235 19579 797 640 82 75 135168 0 0 wb-idmap
Nov 9 01:01:50 TNAS1 kernel: [1629815] 0 1629815 20194 1495 608 162 725 147456 0 0 smbd
Nov 9 01:01:50 TNAS1 kernel: [1629817] 0 1629817 19634 696 630 66 0 131072 0 0 smbd-notifyd
Nov 9 01:01:50 TNAS1 kernel: [1629818] 0 1629818 19638 709 630 79 0 135168 0 0 smbd-cleanupd
Nov 9 01:01:50 TNAS1 kernel: [1632448] 0 1632448 34145 6326 2476 78 3772 212992 0 0 smbd[192.168.1.
Nov 9 01:01:50 TNAS1 kernel: [1636144] 0 1636144 33944 12502 8678 142 3682 262144 0 0 smbd[fe80::3b34
Nov 9 01:01:50 TNAS1 kernel: [1653950] 0 1653950 34260 7704 3997 97 3610 225280 0 0 smbd[192.168.1.
Nov 9 01:01:50 TNAS1 kernel: [2479756] 999 2479756 1113 333 160 173 0 49152 0 -900 bash
Nov 9 01:01:50 TNAS1 kernel: [2709452] 0 2709452 94214 54 32 22 0 86016 0 0 nscd
Nov 9 01:01:50 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2776151] 0 2776151 207108 13004 12886 118 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2777444] 0 2777444 206501 12405 12278 127 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2782291] 999 2782291 1113 278 204 74 0 49152 0 -900 bash
Nov 9 01:01:50 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=libvirtd.service,mems_allowed=0,global_oom,task_memcg=/system.slice/smbd.service,task=smbd[fe80::3b34,pid=1636144,uid=0
Nov 9 01:01:50 TNAS1 kernel: Out of memory: Killed process 1636144 (smbd[fe80::3b34) total-vm:135776kB, anon-rss:34712kB, file-rss:568kB, shmem-rss:14728kB, UID:0 pgtables:256kB oom_score_adj:0
Nov 9 01:01:50 TNAS1 kernel: bash invoked oom-killer: gfp_mask=0x102dc2(GFP_HIGHUSER|__GFP_NOWARN|__GFP_ZERO), order=0, oom_score_adj=-900
Nov 9 01:01:50 TNAS1 kernel: CPU: 5 UID: 999 PID: 2782291 Comm: bash Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:50 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:50 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:50 TNAS1 kernel: Call Trace:
Nov 9 01:01:50 TNAS1 kernel:
Nov 9 01:01:50 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:50 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:50 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:50 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:50 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:50 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:50 TNAS1 kernel: alloc_pages_bulk_noprof+0x3df/0x720
Nov 9 01:01:50 TNAS1 kernel: __vmalloc_node_range_noprof+0x24f/0x850
Nov 9 01:01:50 TNAS1 kernel: ? kernel_clone+0xc2/0x430
Nov 9 01:01:50 TNAS1 kernel: copy_process+0xde5/0x26b0
Nov 9 01:01:50 TNAS1 kernel: ? kernel_clone+0xc2/0x430
Nov 9 01:01:50 TNAS1 kernel: ? __mod_memcg_lruvec_state+0x8d/0x140
Nov 9 01:01:50 TNAS1 kernel: ? __lruvec_stat_mod_folio+0x83/0xd0
Nov 9 01:01:50 TNAS1 kernel: ? rrw_exit+0x5e/0xc0 [zfs]
Nov 9 01:01:50 TNAS1 kernel: kernel_clone+0xc2/0x430
Nov 9 01:01:50 TNAS1 kernel: __do_sys_clone+0x66/0x90
Nov 9 01:01:50 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:50 TNAS1 kernel: ? do_fault+0x281/0x4d0
Nov 9 01:01:50 TNAS1 kernel: ? __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:01:50 TNAS1 kernel: ? __count_memcg_events+0x53/0xf0
Nov 9 01:01:50 TNAS1 kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 9 01:01:50 TNAS1 kernel: ? handle_mm_fault+0xae/0x2d0
Nov 9 01:01:50 TNAS1 kernel: ? do_user_addr_fault+0x17f/0x670
Nov 9 01:01:50 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:01:50 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:50 TNAS1 kernel: RIP: 0033:0x7fe37389c313
Nov 9 01:01:50 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7fe37389c2e9.
Nov 9 01:01:50 TNAS1 kernel: RSP: 002b:00007ffe093537f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Nov 9 01:01:50 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe37389c313
Nov 9 01:01:50 TNAS1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Nov 9 01:01:50 TNAS1 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00222a2422206f68
Nov 9 01:01:50 TNAS1 kernel: R10: 00007fe3737c5a10 R11: 0000000000000246 R12: 0000000000000001
Nov 9 01:01:50 TNAS1 kernel: R13: 00007ffe09353a30 R14: 000055cb8d949b08 R15: 0000000000000000
Nov 9 01:01:50 TNAS1 kernel:
Nov 9 01:01:50 TNAS1 kernel: Mem-Info:
Nov 9 01:01:50 TNAS1 systemd[1]: getty@tty1.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:50 TNAS1 systemd[1]: getty@tty1.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:50 TNAS1 systemd[1]: getty@tty1.service: Failed with result 'oom-kill'.
Nov 9 01:01:50 TNAS1 systemd[1]: getty@tty1.service: Consumed 2min 23.392s CPU time.
Nov 9 01:01:50 TNAS1 kernel: active_anon:64964 inactive_anon:7650231 isolated_anon:0
active_file:185 inactive_file:610 isolated_file:107
unevictable:1000 dirty:219 writeback:239
slab_reclaimable:58598 slab_unreclaimable:82258
mapped:5134 shmem:92850 pagetables:17009
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:51479 free_pcp:3444 free_cma:3072
Nov 9 01:01:50 TNAS1 kernel: Node 0 active_anon:259856kB inactive_anon:30601660kB active_file:620kB inactive_file:2920kB unevictable:4000kB isolated(anon):0kB isolated(file):428kB mapped:20896kB dirty:812kB writeback:436kB shmem:371400kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:837632kB writeback_tmp:0kB kernel_stack:12400kB pagetables:68036kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:50 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA32 free:20888kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3463176kB active_file:836kB inactive_file:720kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:1652kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:50 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:50 TNAS1 kernel: Node 0 Normal free:183176kB boost:0kB min:60392kB low:89652kB high:118912kB reserved_highatomic:0KB active_anon:260064kB inactive_anon:27127036kB active_file:316kB inactive_file:2008kB unevictable:4000kB writepending:1460kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:13456kB local_pcp:0kB free_cma:12288kB
Nov 9 01:01:50 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 DMA32: 24*4kB (UE) 33*8kB (UME) 52*16kB (UME) 37*32kB (UME) 38*64kB (UME) 5*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 7*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 20552kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 Normal: 2715*4kB (UME) 792*8kB (UM) 1845*16kB (UME) 2397*32kB (UME) 24*64kB (UM) 12*128kB (UM) 4*256kB (M) 0*512kB 0*1024kB 6*2048kB (MC) 2*4096kB (MC) 2*8192kB (M) 1*16384kB (M) 0*32768kB 0*65536kB = 180764kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:50 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:50 TNAS1 kernel: 93953 total pagecache pages
Nov 9 01:01:50 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:50 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:50 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:50 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:50 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:50 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:50 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:50 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:50 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:50 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:50 TNAS1 kernel: [ 728] 104 728 2091 400 256 144 0 65536 0 -900 dbus-daemon
Nov 9 01:01:50 TNAS1 kernel: [ 741] 0 741 11506 867 224 643 0 106496 0 -250 systemd-journal
Nov 9 01:01:50 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:50 TNAS1 kernel: [ 1630] 0 1630 8309088 7434219 7433834 321 64 60346368 0 -1000 asyncio_loop
Nov 9 01:01:50 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:50 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:50 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:50 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:50 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:50 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:50 TNAS1 kernel: [ 3540] 0 3540 27526 4256 4148 108 0 114688 0 0 truenas_audit_h
Nov 9 01:01:50 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:50 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:50 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:50 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:50 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:50 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:50 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:50 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:50 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:50 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:50 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:50 TNAS1 kernel: [ 3681] 0 3681 282154 6401 6236 165 0 577536 0 0 syslog-ng
Nov 9 01:01:50 TNAS1 kernel: [ 3691] 0 3691 12416 497 224 273 0 94208 0 0 systemd-logind
Nov 9 01:01:50 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:50 TNAS1 kernel: [ 3708] 0 3708 523983 4148 4140 8 0 327680 0 -999 containerd
Nov 9 01:01:50 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:50 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:50 TNAS1 kernel: [ 3962] 122 3962 1695 190 96 94 0 57344 0 0 upsd
Nov 9 01:01:50 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:50 TNAS1 kernel: [ 4009] 122 4009 2127 251 96 155 0 53248 0 0 upsmon
Nov 9 01:01:50 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:50 TNAS1 kernel: [ 4132] 0 4132 4136 420 224 196 0 69632 0 0 systemd-machine
Nov 9 01:01:50 TNAS1 kernel: [ 4133] 0 4133 427247 2934 2880 54 0 339968 0 0 libvirtd
Nov 9 01:01:50 TNAS1 kernel: [ 5386] 105 5386 1781 152 96 56 0 49152 0 0 avahi-daemon
Nov 9 01:01:50 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:50 TNAS1 kernel: [ 5692] 1 5692 8070 3981 3840 141 0 98304 0 0 wsdd.py
Nov 9 01:01:50 TNAS1 kernel: [ 5694] 0 5694 17601 616 544 72 0 118784 0 0 nmbd
Nov 9 01:01:50 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:50 TNAS1 kernel: [ 6672] 0 6672 13921 5633 5472 161 0 139264 0 -1000 websockify
Nov 9 01:01:50 TNAS1 kernel: [ 6757] 999 6757 116708 29364 29218 82 64 475136 0 -900 netdata
Nov 9 01:01:50 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:50 TNAS1 kernel: [ 7021] 999 7021 22248 5135 4846 289 0 147456 0 -900 python.d.plugin
Nov 9 01:01:50 TNAS1 kernel: [ 7210] 0 7210 18963 765 608 61 96 131072 0 0 winbindd
Nov 9 01:01:50 TNAS1 kernel: [ 7212] 0 7212 18982 747 640 69 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:50 TNAS1 kernel: [ 7230] 0 7230 13921 5439 5413 26 0 131072 0 -1000 websockify
Nov 9 01:01:50 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:50 TNAS1 kernel: [ 10337] 0 10337 4476 501 416 85 0 73728 0 0 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 10339] 0 10339 4477 591 448 143 0 81920 0 0 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:50 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:50 TNAS1 kernel: [ 11235] 0 11235 19579 797 640 82 75 135168 0 0 wb-idmap
Nov 9 01:01:50 TNAS1 kernel: [1629815] 0 1629815 20194 1399 608 66 725 147456 0 0 smbd
Nov 9 01:01:50 TNAS1 kernel: [1629817] 0 1629817 19634 696 630 66 0 131072 0 0 smbd-notifyd
Nov 9 01:01:50 TNAS1 kernel: [1629818] 0 1629818 19638 709 630 79 0 135168 0 0 smbd-cleanupd
Nov 9 01:01:50 TNAS1 kernel: [1632448] 0 1632448 34145 6326 2476 78 3772 212992 0 0 smbd[192.168.1.
Nov 9 01:01:50 TNAS1 kernel: [1653950] 0 1653950 34260 7704 3997 97 3610 225280 0 0 smbd[192.168.1.
Nov 9 01:01:50 TNAS1 kernel: [2479756] 999 2479756 1113 333 160 173 0 49152 0 -900 bash
Nov 9 01:01:50 TNAS1 kernel: [2709452] 0 2709452 94214 54 32 22 0 86016 0 0 nscd
Nov 9 01:01:50 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2776151] 0 2776151 207108 13004 12886 118 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2777444] 0 2777444 206501 12405 12278 127 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:50 TNAS1 kernel: [2782291] 999 2782291 1113 278 204 74 0 49152 0 -900 bash
Nov 9 01:01:50 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/system.slice/smbd.service,task=smbd[192.168.1.,pid=1653950,uid=0
Nov 9 01:01:50 TNAS1 kernel: Out of memory: Killed process 1653950 (smbd[192.168.1.) total-vm:137040kB, anon-rss:15988kB, file-rss:388kB, shmem-rss:14440kB, UID:0 pgtables:220kB oom_score_adj:0
Nov 9 01:01:50 TNAS1 systemd[1]: smbd.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:50 TNAS1 systemd[1]: system-getty.slice: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:50 TNAS1 systemd[1]: smbd.service: Failed with result 'oom-kill'.
Nov 9 01:01:50 TNAS1 systemd[1]: smbd.service: Consumed 20min 10.851s CPU time.
Nov 9 01:01:51 TNAS1 syslog-ng[2782311]: syslog-ng starting up; version='3.38.1'
Nov 9 01:01:51 TNAS1 kernel: IoThread invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=-1000
Nov 9 01:01:51 TNAS1 kernel: CPU: 4 UID: 0 PID: 4125 Comm: IoThread Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:51 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:51 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:51 TNAS1 kernel: Call Trace:
Nov 9 01:01:51 TNAS1 kernel:
Nov 9 01:01:51 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:51 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:51 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:51 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:51 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:51 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:51 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:51 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:01:51 TNAS1 kernel: __pte_alloc+0x2a/0xb0
Nov 9 01:01:51 TNAS1 kernel: do_anonymous_page+0x51a/0x7d0
Nov 9 01:01:51 TNAS1 kernel: ? reweight_entity+0x1c1/0x260
Nov 9 01:01:51 TNAS1 kernel: ? update_curr+0x9c/0x240
Nov 9 01:01:51 TNAS1 kernel: __handle_mm_fault+0xb6d/0xfd0
Nov 9 01:01:51 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:01:51 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:01:51 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:01:51 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:01:51 TNAS1 kernel: RIP: 0033:0x4dbdb7
Nov 9 01:01:51 TNAS1 kernel: Code: c0 6c ab 00 00 00 00 00 89 d0 4c 89 24 c5 c0 6c ab 00 4c 89 e0 48 2b 05 37 b1 5d 00 48 c1 f8 04 41 0f 11 44 24 20 48 0f af c1 46 24 ff ff 00 00 4c 89 25 03 b1 5d 00 89 46 20 48 8d 86 00 40
Nov 9 01:01:51 TNAS1 kernel: RSP: 002b:00007fefd5ffa630 EFLAGS: 00010a03
Nov 9 01:01:51 TNAS1 kernel: RAX: 00000000000057f2 RBX: 0000000000000018 RCX: aaaaaaaaaaaaaaab
Nov 9 01:01:51 TNAS1 kernel: RDX: 000000000000003f RSI: 00007fe895f00000 RDI: 00007fe895f00000
Nov 9 01:01:51 TNAS1 kernel: RBP: 0000000000000000 R08: 00000000ffffffff R09: 0000000000000000
Nov 9 01:01:51 TNAS1 kernel: R10: 0000000000000022 R11: 0000000000000246 R12: 00000000241ce270
Nov 9 01:01:51 TNAS1 kernel: R13: 00007fe895f00000 R14: 0000000000000040 R15: 0000000000000005
Nov 9 01:01:51 TNAS1 kernel:
Nov 9 01:01:51 TNAS1 kernel: Mem-Info:
Nov 9 01:01:51 TNAS1 kernel: active_anon:65320 inactive_anon:7670005 isolated_anon:0
active_file:394 inactive_file:483 isolated_file:0
unevictable:1000 dirty:162 writeback:273
slab_reclaimable:58485 slab_unreclaimable:81956
mapped:1137 shmem:92845 pagetables:16777
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:35258 free_pcp:87 free_cma:1558
Nov 9 01:01:51 TNAS1 kernel: Node 0 active_anon:261280kB inactive_anon:30680020kB active_file:1576kB inactive_file:1932kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:4548kB dirty:648kB writeback:1092kB shmem:371380kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:823296kB writeback_tmp:0kB kernel_stack:12192kB pagetables:67108kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:51 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA32 free:9676kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3475700kB active_file:508kB inactive_file:760kB unevictable:0kB writepending:76kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:316kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:51 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:51 TNAS1 kernel: Node 0 Normal free:131328kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:0KB active_anon:261276kB inactive_anon:27193084kB active_file:1068kB inactive_file:1172kB unevictable:4000kB writepending:1688kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:20kB local_pcp:0kB free_cma:6232kB
Nov 9 01:01:51 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA32: 25*4kB (UME) 136*8kB (UME) 103*16kB (UME) 64*32kB (UME) 40*64kB (UME) 4*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 8724kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 Normal: 3024*4kB (UME) 449*8kB (UMEC) 2062*16kB (UMEC) 2328*32kB (UMEC) 0*64kB 0*128kB 0*256kB 2*512kB (C) 1*1024kB (C) 3*2048kB (MC) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 131368kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:51 TNAS1 kernel: 93750 total pagecache pages
Nov 9 01:01:51 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:51 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:51 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:51 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:51 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:51 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:51 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:51 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:51 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:51 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:51 TNAS1 kernel: [ 728] 104 728 2091 336 256 80 0 65536 0 -900 dbus-daemon
Nov 9 01:01:51 TNAS1 kernel: [ 741] 0 741 11506 707 224 483 0 106496 0 -250 systemd-journal
Nov 9 01:01:51 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:51 TNAS1 kernel: [ 1630] 0 1630 8335456 7460432 7460138 230 64 60555264 0 -1000 asyncio_loop
Nov 9 01:01:51 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:51 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:51 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:51 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:51 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:51 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:51 TNAS1 kernel: [ 3540] 0 3540 27526 4256 4148 108 0 114688 0 0 truenas_audit_h
Nov 9 01:01:51 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:51 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:51 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:51 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:51 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:51 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:51 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:51 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:51 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:51 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:51 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:51 TNAS1 kernel: [ 3681] 0 3681 282154 6584 6236 348 0 577536 0 0 syslog-ng
Nov 9 01:01:51 TNAS1 kernel: [ 3691] 0 3691 12416 369 224 145 0 94208 0 0 systemd-logind
Nov 9 01:01:51 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:51 TNAS1 kernel: [ 3708] 0 3708 523983 4148 4140 8 0 327680 0 -999 containerd
Nov 9 01:01:51 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:51 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:51 TNAS1 kernel: [ 3962] 122 3962 1695 194 96 98 0 57344 0 0 upsd
Nov 9 01:01:51 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:51 TNAS1 kernel: [ 4009] 122 4009 2127 256 96 160 0 53248 0 0 upsmon
Nov 9 01:01:51 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:51 TNAS1 kernel: [ 4132] 0 4132 4136 292 224 68 0 69632 0 0 systemd-machine
Nov 9 01:01:51 TNAS1 kernel: [ 4133] 0 4133 427247 2943 2880 63 0 339968 0 0 libvirtd
Nov 9 01:01:51 TNAS1 kernel: [ 5386] 105 5386 1781 152 96 56 0 49152 0 0 avahi-daemon
Nov 9 01:01:51 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:51 TNAS1 kernel: [ 5692] 1 5692 8070 3981 3840 141 0 98304 0 0 wsdd.py
Nov 9 01:01:51 TNAS1 kernel: [ 5694] 0 5694 17601 616 544 72 0 118784 0 0 nmbd
Nov 9 01:01:51 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:51 TNAS1 kernel: [ 6672] 0 6672 13921 5601 5472 129 0 139264 0 -1000 websockify
Nov 9 01:01:51 TNAS1 kernel: [ 6757] 999 6757 116708 29365 29218 83 64 475136 0 -900 netdata
Nov 9 01:01:51 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:51 TNAS1 kernel: [ 7021] 999 7021 22248 4975 4846 129 0 147456 0 -900 python.d.plugin
Nov 9 01:01:51 TNAS1 kernel: [ 7210] 0 7210 18963 733 608 29 96 131072 0 0 winbindd
Nov 9 01:01:51 TNAS1 kernel: [ 7212] 0 7212 18982 747 640 69 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:51 TNAS1 kernel: [ 7230] 0 7230 13921 5565 5413 152 0 131072 0 -1000 websockify
Nov 9 01:01:51 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:51 TNAS1 kernel: [ 10337] 0 10337 4476 501 416 85 0 73728 0 0 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 10339] 0 10339 4477 591 448 143 0 81920 0 0 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:51 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 11235] 0 11235 19579 797 640 82 75 135168 0 0 wb-idmap
Nov 9 01:01:51 TNAS1 kernel: [2479756] 999 2479756 1113 273 160 113 0 49152 0 -900 bash
Nov 9 01:01:51 TNAS1 kernel: [2709452] 0 2709452 94214 86 32 54 0 86016 0 0 nscd
Nov 9 01:01:51 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2776151] 0 2776151 207108 13004 12886 118 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2777444] 0 2777444 206501 12405 12278 127 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=middlewared.service,mems_allowed=0,global_oom,task_memcg=/system.slice/syslog-ng.service,task=syslog-ng,pid=3681,uid=0
Nov 9 01:01:51 TNAS1 kernel: Out of memory: Killed process 3681 (syslog-ng) total-vm:1128616kB, anon-rss:24944kB, file-rss:1392kB, shmem-rss:0kB, UID:0 pgtables:564kB oom_score_adj:0
Nov 9 01:01:51 TNAS1 systemd[1]: syslog-ng.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:51 TNAS1 systemd[1]: syslog-ng.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:51 TNAS1 systemd[1]: syslog-ng.service: Failed with result 'oom-kill'.
Nov 9 01:01:51 TNAS1 systemd[1]: syslog-ng.service: Consumed 4min 55.147s CPU time.
Nov 9 01:01:51 TNAS1 kernel: IoThread invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=-1000
Nov 9 01:01:51 TNAS1 kernel: CPU: 4 UID: 0 PID: 4125 Comm: IoThread Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:51 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:51 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:51 TNAS1 kernel: Call Trace:
Nov 9 01:01:51 TNAS1 kernel:
Nov 9 01:01:51 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:51 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:51 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:51 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:51 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:51 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:51 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:51 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:01:51 TNAS1 kernel: __pte_alloc+0x2a/0xb0
Nov 9 01:01:51 TNAS1 kernel: do_anonymous_page+0x51a/0x7d0
Nov 9 01:01:51 TNAS1 kernel: ? reweight_entity+0x1c1/0x260
Nov 9 01:01:51 TNAS1 kernel: ? update_curr+0x9c/0x240
Nov 9 01:01:51 TNAS1 kernel: __handle_mm_fault+0xb6d/0xfd0
Nov 9 01:01:51 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:01:51 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:01:51 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:01:51 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:01:51 TNAS1 kernel: RIP: 0033:0x4dbdb7
Nov 9 01:01:51 TNAS1 kernel: Code: c0 6c ab 00 00 00 00 00 89 d0 4c 89 24 c5 c0 6c ab 00 4c 89 e0 48 2b 05 37 b1 5d 00 48 c1 f8 04 41 0f 11 44 24 20 48 0f af c1 46 24 ff ff 00 00 4c 89 25 03 b1 5d 00 89 46 20 48 8d 86 00 40
Nov 9 01:01:51 TNAS1 kernel: RSP: 002b:00007fefd5ffa630 EFLAGS: 00010a03
Nov 9 01:01:51 TNAS1 kernel: RAX: 00000000000057f2 RBX: 0000000000000018 RCX: aaaaaaaaaaaaaaab
Nov 9 01:01:51 TNAS1 kernel: RDX: 000000000000003f RSI: 00007fe895f00000 RDI: 00007fe895f00000
Nov 9 01:01:51 TNAS1 kernel: RBP: 0000000000000000 R08: 00000000ffffffff R09: 0000000000000000
Nov 9 01:01:51 TNAS1 kernel: R10: 0000000000000022 R11: 0000000000000246 R12: 00000000241ce270
Nov 9 01:01:51 TNAS1 kernel: R13: 00007fe895f00000 R14: 0000000000000040 R15: 0000000000000005
Nov 9 01:01:51 TNAS1 kernel:
Nov 9 01:01:51 TNAS1 kernel: Mem-Info:
Nov 9 01:01:51 TNAS1 kernel: active_anon:65320 inactive_anon:7663742 isolated_anon:0
active_file:467 inactive_file:614 isolated_file:0
unevictable:1000 dirty:144 writeback:309
slab_reclaimable:58485 slab_unreclaimable:81954
mapped:1310 shmem:92845 pagetables:16704
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:40298 free_pcp:1037 free_cma:3102
Nov 9 01:01:51 TNAS1 kernel: Node 0 active_anon:261280kB inactive_anon:30654968kB active_file:1868kB inactive_file:2456kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:5240kB dirty:576kB writeback:1236kB shmem:371380kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:802816kB writeback_tmp:0kB kernel_stack:12160kB pagetables:66816kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:51 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA32 free:15524kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3464980kB active_file:508kB inactive_file:1196kB unevictable:0kB writepending:88kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:4140kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:51 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:51 TNAS1 kernel: Node 0 Normal free:145640kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:0KB active_anon:261276kB inactive_anon:27178996kB active_file:1068kB inactive_file:1172kB unevictable:4000kB writepending:1724kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:12408kB
Nov 9 01:01:51 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 32kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA32: 24*4kB (UE) 32*8kB (UE) 51*16kB (UE) 31*32kB (UE) 33*64kB (UME) 3*128kB (ME) 1*256kB (M) 0*512kB 0*1024kB 5*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 15152kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 Normal: 3755*4kB (UME) 535*8kB (UMEC) 2083*16kB (UMEC) 2333*32kB (UMEC) 3*64kB (MC) 0*128kB 0*256kB 2*512kB (C) 1*1024kB (C) 8*2048kB (MC) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 145908kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:51 TNAS1 kernel: 93988 total pagecache pages
Nov 9 01:01:51 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:51 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:51 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:51 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:51 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:51 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:51 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:51 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:51 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:51 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:51 TNAS1 kernel: [ 728] 104 728 2091 400 256 144 0 65536 0 -900 dbus-daemon
Nov 9 01:01:51 TNAS1 kernel: [ 741] 0 741 11506 803 224 579 0 106496 0 -250 systemd-journal
Nov 9 01:01:51 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:51 TNAS1 kernel: [ 1630] 0 1630 8335456 7460400 7460138 198 64 60555264 0 -1000 asyncio_loop
Nov 9 01:01:51 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:51 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:51 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:51 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:51 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:51 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:51 TNAS1 kernel: [ 3540] 0 3540 27526 4256 4148 108 0 114688 0 0 truenas_audit_h
Nov 9 01:01:51 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:51 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:51 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:51 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:51 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:51 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:51 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:51 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:51 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:51 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:51 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:51 TNAS1 kernel: [ 3691] 0 3691 12416 510 224 286 0 94208 0 0 systemd-logind
Nov 9 01:01:51 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:51 TNAS1 kernel: [ 3708] 0 3708 523983 4148 4140 8 0 327680 0 -999 containerd
Nov 9 01:01:51 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:51 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:51 TNAS1 kernel: [ 3962] 122 3962 1695 194 96 98 0 57344 0 0 upsd
Nov 9 01:01:51 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:51 TNAS1 kernel: [ 4009] 122 4009 2127 256 96 160 0 53248 0 0 upsmon
Nov 9 01:01:51 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:51 TNAS1 kernel: [ 4132] 0 4132 4136 442 224 218 0 69632 0 0 systemd-machine
Nov 9 01:01:51 TNAS1 kernel: [ 4133] 0 4133 427247 2943 2880 63 0 339968 0 0 libvirtd
Nov 9 01:01:51 TNAS1 kernel: [ 5386] 105 5386 1781 152 96 56 0 49152 0 0 avahi-daemon
Nov 9 01:01:51 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:51 TNAS1 kernel: [ 5692] 1 5692 8070 3981 3840 141 0 98304 0 0 wsdd.py
Nov 9 01:01:51 TNAS1 kernel: [ 5694] 0 5694 17601 616 544 72 0 118784 0 0 nmbd
Nov 9 01:01:51 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:51 TNAS1 kernel: [ 6672] 0 6672 13921 5601 5472 129 0 139264 0 -1000 websockify
Nov 9 01:01:51 TNAS1 kernel: [ 6757] 999 6757 116708 29365 29218 83 64 475136 0 -900 netdata
Nov 9 01:01:51 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:51 TNAS1 kernel: [ 7021] 999 7021 22248 4975 4846 129 0 147456 0 -900 python.d.plugin
Nov 9 01:01:51 TNAS1 kernel: [ 7210] 0 7210 18963 733 608 29 96 131072 0 0 winbindd
Nov 9 01:01:51 TNAS1 kernel: [ 7212] 0 7212 18982 747 640 69 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:51 TNAS1 kernel: [ 7230] 0 7230 13921 5565 5413 152 0 131072 0 -1000 websockify
Nov 9 01:01:51 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:51 TNAS1 kernel: [ 10337] 0 10337 4476 501 416 85 0 73728 0 0 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 10339] 0 10339 4477 591 448 143 0 81920 0 0 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:51 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 11235] 0 11235 19579 797 640 82 75 135168 0 0 wb-idmap
Nov 9 01:01:51 TNAS1 kernel: [2479756] 999 2479756 1113 273 160 113 0 49152 0 -900 bash
Nov 9 01:01:51 TNAS1 kernel: [2709452] 0 2709452 94214 86 32 54 0 86016 0 0 nscd
Nov 9 01:01:51 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2776151] 0 2776151 207108 13004 12886 118 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2777444] 0 2777444 206501 12405 12278 127 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=middlewared.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tnaudit.service,task=truenas_audit_h,pid=3540,uid=0
Nov 9 01:01:51 TNAS1 kernel: Out of memory: Killed process 3540 (truenas_audit_h) total-vm:110104kB, anon-rss:16592kB, file-rss:432kB, shmem-rss:0kB, UID:0 pgtables:112kB oom_score_adj:0
Nov 9 01:01:51 TNAS1 systemd[1]: tnaudit.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:51 TNAS1 systemd[1]: tnaudit.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:51 TNAS1 systemd[1]: tnaudit.service: Failed with result 'oom-kill'.
Nov 9 01:01:51 TNAS1 kernel: IoThread invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=-1000
Nov 9 01:01:51 TNAS1 kernel: CPU: 3 UID: 0 PID: 4125 Comm: IoThread Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:51 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:51 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:51 TNAS1 kernel: Call Trace:
Nov 9 01:01:51 TNAS1 kernel:
Nov 9 01:01:51 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:51 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:51 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:51 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:51 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:51 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:51 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:51 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:01:51 TNAS1 kernel: __pte_alloc+0x2a/0xb0
Nov 9 01:01:51 TNAS1 kernel: do_anonymous_page+0x51a/0x7d0
Nov 9 01:01:51 TNAS1 kernel: ? reweight_entity+0x1c1/0x260
Nov 9 01:01:51 TNAS1 kernel: ? update_curr+0x9c/0x240
Nov 9 01:01:51 TNAS1 kernel: __handle_mm_fault+0xb6d/0xfd0
Nov 9 01:01:51 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:01:51 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:01:51 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:01:51 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:01:51 TNAS1 kernel: RIP: 0033:0x4dbdb7
Nov 9 01:01:51 TNAS1 kernel: Code: c0 6c ab 00 00 00 00 00 89 d0 4c 89 24 c5 c0 6c ab 00 4c 89 e0 48 2b 05 37 b1 5d 00 48 c1 f8 04 41 0f 11 44 24 20 48 0f af c1 46 24 ff ff 00 00 4c 89 25 03 b1 5d 00 89 46 20 48 8d 86 00 40
Nov 9 01:01:51 TNAS1 kernel: RSP: 002b:00007fefd5ffa630 EFLAGS: 00010a03
Nov 9 01:01:51 TNAS1 kernel: RAX: 00000000000057f2 RBX: 0000000000000018 RCX: aaaaaaaaaaaaaaab
Nov 9 01:01:51 TNAS1 kernel: RDX: 000000000000003f RSI: 00007fe895f00000 RDI: 00007fe895f00000
Nov 9 01:01:51 TNAS1 kernel: RBP: 0000000000000000 R08: 00000000ffffffff R09: 0000000000000000
Nov 9 01:01:51 TNAS1 kernel: R10: 0000000000000022 R11: 0000000000000246 R12: 00000000241ce270
Nov 9 01:01:51 TNAS1 kernel: R13: 00007fe895f00000 R14: 0000000000000040 R15: 0000000000000005
Nov 9 01:01:51 TNAS1 kernel:
Nov 9 01:01:51 TNAS1 kernel: Mem-Info:
Nov 9 01:01:51 TNAS1 kernel: active_anon:65320 inactive_anon:7659500 isolated_anon:0
active_file:540 inactive_file:719 isolated_file:0
unevictable:1000 dirty:208 writeback:318
slab_reclaimable:58484 slab_unreclaimable:81953
mapped:1437 shmem:92845 pagetables:16631
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:45000 free_pcp:258 free_cma:3102
Nov 9 01:01:51 TNAS1 kernel: Node 0 active_anon:261280kB inactive_anon:30638000kB active_file:2160kB inactive_file:2876kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:5748kB dirty:832kB writeback:1272kB shmem:371380kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:798720kB writeback_tmp:0kB kernel_stack:12128kB pagetables:66524kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:51 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA32 free:17384kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3464980kB active_file:944kB inactive_file:1796kB unevictable:0kB writepending:88kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:1044kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:51 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:51 TNAS1 kernel: Node 0 Normal free:162084kB boost:0kB min:60392kB low:89652kB high:118912kB reserved_highatomic:0KB active_anon:261276kB inactive_anon:27162160kB active_file:1068kB inactive_file:1172kB unevictable:4000kB writepending:1724kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:712kB local_pcp:0kB free_cma:12408kB
Nov 9 01:01:51 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 32kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 DMA32: 25*4kB (UME) 67*8kB (UME) 84*16kB (UME) 59*32kB (UME) 42*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 5*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 17948kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 Normal: 5349*4kB (UME) 952*8kB (UMEC) 2157*16kB (UMEC) 2345*32kB (UMEC) 7*64kB (MC) 0*128kB 0*256kB 2*512kB (C) 1*1024kB (C) 8*2048kB (MC) 1*4096kB (M) 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 161540kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:51 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:51 TNAS1 kernel: 94115 total pagecache pages
Nov 9 01:01:51 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:51 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:51 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:51 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:51 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:51 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:51 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:51 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:51 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:51 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:51 TNAS1 kernel: [ 728] 104 728 2091 400 256 144 0 65536 0 -900 dbus-daemon
Nov 9 01:01:51 TNAS1 kernel: [ 741] 0 741 11506 803 224 579 0 106496 0 -250 systemd-journal
Nov 9 01:01:51 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:51 TNAS1 kernel: [ 1630] 0 1630 8335456 7460368 7460138 166 64 60555264 0 -1000 asyncio_loop
Nov 9 01:01:51 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:51 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:51 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:51 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:51 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:51 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:51 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:51 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:51 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:51 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:51 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:51 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:51 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:51 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:51 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:51 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:51 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:51 TNAS1 kernel: [ 3691] 0 3691 12416 510 224 286 0 94208 0 0 systemd-logind
Nov 9 01:01:51 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:51 TNAS1 kernel: [ 3708] 0 3708 523983 4148 4140 8 0 327680 0 -999 containerd
Nov 9 01:01:51 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:51 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:51 TNAS1 kernel: [ 3962] 122 3962 1695 194 96 98 0 57344 0 0 upsd
Nov 9 01:01:51 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:51 TNAS1 kernel: [ 4009] 122 4009 2127 256 96 160 0 53248 0 0 upsmon
Nov 9 01:01:51 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:51 TNAS1 kernel: [ 4132] 0 4132 4136 442 224 218 0 69632 0 0 systemd-machine
Nov 9 01:01:51 TNAS1 kernel: [ 4133] 0 4133 427247 2943 2880 63 0 339968 0 0 libvirtd
Nov 9 01:01:51 TNAS1 kernel: [ 5386] 105 5386 1781 152 96 56 0 49152 0 0 avahi-daemon
Nov 9 01:01:51 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:51 TNAS1 kernel: [ 5692] 1 5692 8070 3981 3840 141 0 98304 0 0 wsdd.py
Nov 9 01:01:51 TNAS1 kernel: [ 5694] 0 5694 17601 616 544 72 0 118784 0 0 nmbd
Nov 9 01:01:51 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:51 TNAS1 kernel: [ 6672] 0 6672 13921 5601 5472 129 0 139264 0 -1000 websockify
Nov 9 01:01:51 TNAS1 kernel: [ 6757] 999 6757 116708 29365 29218 83 64 475136 0 -900 netdata
Nov 9 01:01:51 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:51 TNAS1 kernel: [ 7021] 999 7021 22248 4975 4846 129 0 147456 0 -900 python.d.plugin
Nov 9 01:01:51 TNAS1 kernel: [ 7210] 0 7210 18963 733 608 29 96 131072 0 0 winbindd
Nov 9 01:01:51 TNAS1 kernel: [ 7212] 0 7212 18982 747 640 69 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:51 TNAS1 kernel: [ 7230] 0 7230 13921 5565 5413 152 0 131072 0 -1000 websockify
Nov 9 01:01:51 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:51 TNAS1 kernel: [ 10337] 0 10337 4476 501 416 85 0 73728 0 0 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 10339] 0 10339 4477 591 448 143 0 81920 0 0 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:51 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:51 TNAS1 kernel: [ 11235] 0 11235 19579 797 640 82 75 135168 0 0 wb-idmap
Nov 9 01:01:51 TNAS1 kernel: [2479756] 999 2479756 1113 273 160 113 0 49152 0 -900 bash
Nov 9 01:01:51 TNAS1 kernel: [2709452] 0 2709452 94214 86 32 54 0 86016 0 0 nscd
Nov 9 01:01:51 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2776151] 0 2776151 207108 13004 12886 118 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2777444] 0 2777444 206501 12405 12278 127 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:51 TNAS1 kernel: [2782309] 0 2782309 42617 1729 1470 259 0 94208 0 0 (ndler.py)
Nov 9 01:01:51 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=middlewared.service,mems_allowed=0,global_oom,task_memcg=/system.slice/wsdd.service,task=wsdd.py,pid=5692,uid=1
Nov 9 01:01:51 TNAS1 kernel: Out of memory: Killed process 5692 (wsdd.py) total-vm:32280kB, anon-rss:15360kB, file-rss:564kB, shmem-rss:0kB, UID:1 pgtables:96kB oom_score_adj:0
Nov 9 01:01:51 TNAS1 systemd[1]: Starting tnaudit.service - TrueNAS Auditd Handler...
Nov 9 01:01:51 TNAS1 systemd[1]: wsdd.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:51 TNAS1 systemd[1]: wsdd.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:51 TNAS1 systemd[1]: wsdd.service: Failed with result 'oom-kill'.
Nov 9 01:01:51 TNAS1 systemd[1]: syslog-ng.service: Scheduled restart job, restart counter is at 1.
Nov 9 01:01:51 TNAS1 systemd[1]: Stopped syslog-ng.service - System Logger Daemon.
Nov 9 01:01:51 TNAS1 systemd[1]: syslog-ng.service: Consumed 4min 55.147s CPU time.
Nov 9 01:01:51 TNAS1 systemd[1]: Starting syslog-ng.service - System Logger Daemon...
Nov 9 01:01:51 TNAS1 systemd[1]: Started tnaudit.service - TrueNAS Auditd Handler.
Nov 9 01:01:51 TNAS1 systemd[1]: Started syslog-ng.service - System Logger Daemon.
Nov 9 01:01:52 TNAS1 kernel: bash invoked oom-killer: gfp_mask=0x102dc2(GFP_HIGHUSER|__GFP_NOWARN|__GFP_ZERO), order=0, oom_score_adj=-900
Nov 9 01:01:53 TNAS1 kernel: CPU: 4 UID: 999 PID: 2479756 Comm: bash Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:53 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:53 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:53 TNAS1 kernel: Call Trace:
Nov 9 01:01:53 TNAS1 kernel:
Nov 9 01:01:53 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:53 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:53 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:53 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:53 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:53 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:53 TNAS1 kernel: alloc_pages_bulk_noprof+0x3df/0x720
Nov 9 01:01:53 TNAS1 kernel: __vmalloc_node_range_noprof+0x24f/0x850
Nov 9 01:01:53 TNAS1 kernel: ? kernel_clone+0xc2/0x430
Nov 9 01:01:53 TNAS1 kernel: copy_process+0xde5/0x26b0
Nov 9 01:01:53 TNAS1 kernel: ? kernel_clone+0xc2/0x430
Nov 9 01:01:53 TNAS1 kernel: ? filemap_map_pages+0x521/0x650
Nov 9 01:01:53 TNAS1 kernel: kernel_clone+0xc2/0x430
Nov 9 01:01:53 TNAS1 kernel: ? __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:01:53 TNAS1 kernel: __do_sys_clone+0x66/0x90
Nov 9 01:01:53 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:53 TNAS1 kernel: ? do_user_addr_fault+0x17f/0x670
Nov 9 01:01:53 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:01:53 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:53 TNAS1 kernel: RIP: 0033:0x7fe37389c313
Nov 9 01:01:53 TNAS1 kernel: Code: 00 00 00 00 00 66 90 64 48 8b 04 25 10 00 00 00 45 31 c0 31 d2 31 f6 bf 11 00 20 01 4c 8d 90 d0 02 00 00 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 89 c2 85 c0 75 2c 64 48 8b 04 25 10 00 00
Nov 9 01:01:53 TNAS1 kernel: RSP: 002b:00007ffe09353958 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Nov 9 01:01:53 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe37389c313
Nov 9 01:01:53 TNAS1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Nov 9 01:01:53 TNAS1 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 000055cbcc0923e0
Nov 9 01:01:53 TNAS1 kernel: R10: 00007fe3737c5a10 R11: 0000000000000246 R12: 0000000000000001
Nov 9 01:01:53 TNAS1 kernel: R13: 00007ffe09353b90 R14: 000055cb8d949b08 R15: 00000000ffffffff
Nov 9 01:01:53 TNAS1 kernel:
Nov 9 01:01:53 TNAS1 kernel: Mem-Info:
Nov 9 01:01:53 TNAS1 kernel: active_anon:65319 inactive_anon:7670459 isolated_anon:0
active_file:522 inactive_file:564 isolated_file:0
unevictable:1000 dirty:243 writeback:380
slab_reclaimable:58439 slab_unreclaimable:81838
mapped:1253 shmem:92844 pagetables:16729
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:31922 free_pcp:64 free_cma:221
Nov 9 01:01:53 TNAS1 kernel: Node 0 active_anon:261276kB inactive_anon:30681836kB active_file:2088kB inactive_file:2256kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:5012kB dirty:972kB writeback:1520kB shmem:371376kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:792576kB writeback_tmp:0kB kernel_stack:12208kB pagetables:66916kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:53 TNAS1 kernel: Node 0 DMA free:32kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11232kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:53 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:53 TNAS1 kernel: Node 0 DMA32 free:11248kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3474012kB active_file:504kB inactive_file:924kB unevictable:0kB writepending:436kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:53 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:53 TNAS1 kernel: Node 0 Normal free:117584kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261272kB inactive_anon:27196592kB active_file:980kB inactive_file:1016kB unevictable:4000kB writepending:2056kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:884kB
Nov 9 01:01:53 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:53 TNAS1 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 32kB
Nov 9 01:01:53 TNAS1 kernel: Node 0 DMA32: 323*4kB (UME) 147*8kB (UME) 75*16kB (UME) 43*32kB (UME) 36*64kB (UME) 5*128kB (ME) 2*256kB (M) 1*512kB (M) 1*1024kB (M) 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 12084kB
Nov 9 01:01:53 TNAS1 kernel: Node 0 Normal: 2351*4kB (UMEC) 571*8kB (UME) 1842*16kB (UMEC) 2158*32kB (UMEC) 7*64kB (MC) 5*128kB (MC) 4*256kB (MC) 3*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 118196kB
Nov 9 01:01:53 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:53 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:53 TNAS1 kernel: 93534 total pagecache pages
Nov 9 01:01:53 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:53 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:53 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:53 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:53 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:53 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:53 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:53 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:53 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:53 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:53 TNAS1 kernel: [ 728] 104 728 2091 304 256 48 0 65536 0 -900 dbus-daemon
Nov 9 01:01:53 TNAS1 kernel: [ 741] 0 741 11506 650 224 426 0 106496 0 -250 systemd-journal
Nov 9 01:01:53 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:53 TNAS1 kernel: [ 1630] 0 1630 8346720 7471602 7471402 136 64 60645376 0 -1000 asyncio_loop
Nov 9 01:01:53 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:53 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:53 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:53 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:53 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:53 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:53 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:53 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:53 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:53 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:53 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:53 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:53 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:53 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:53 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:53 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:53 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:53 TNAS1 kernel: [ 3691] 0 3691 12416 350 224 126 0 94208 0 0 systemd-logind
Nov 9 01:01:53 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:53 TNAS1 kernel: [ 3708] 0 3708 523983 4140 4140 0 0 327680 0 -999 containerd
Nov 9 01:01:53 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:53 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:53 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:53 TNAS1 kernel: [ 3962] 122 3962 1695 162 96 66 0 57344 0 0 upsd
Nov 9 01:01:53 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:53 TNAS1 kernel: [ 4009] 122 4009 2127 192 96 96 0 53248 0 0 upsmon
Nov 9 01:01:53 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:53 TNAS1 kernel: [ 4132] 0 4132 4136 282 224 58 0 69632 0 0 systemd-machine
Nov 9 01:01:53 TNAS1 kernel: [ 4133] 0 4133 427247 2943 2880 63 0 339968 0 0 libvirtd
Nov 9 01:01:53 TNAS1 kernel: [ 5386] 105 5386 1781 152 96 56 0 49152 0 0 avahi-daemon
Nov 9 01:01:53 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:53 TNAS1 kernel: [ 5694] 0 5694 17601 616 544 72 0 118784 0 0 nmbd
Nov 9 01:01:53 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:53 TNAS1 kernel: [ 6672] 0 6672 13921 5584 5472 112 0 139264 0 -1000 websockify
Nov 9 01:01:53 TNAS1 kernel: [ 6757] 999 6757 116708 29301 29218 19 64 475136 0 -900 netdata
Nov 9 01:01:53 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:53 TNAS1 kernel: [ 7021] 999 7021 22248 5016 4846 170 0 147456 0 -900 python.d.plugin
Nov 9 01:01:53 TNAS1 kernel: [ 7210] 0 7210 18963 733 608 29 96 131072 0 0 winbindd
Nov 9 01:01:53 TNAS1 kernel: [ 7212] 0 7212 18982 747 640 69 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:53 TNAS1 kernel: [ 7230] 0 7230 13921 5545 5413 132 0 131072 0 -1000 websockify
Nov 9 01:01:53 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:53 TNAS1 kernel: [ 10337] 0 10337 4476 501 416 85 0 73728 0 0 sshd
Nov 9 01:01:53 TNAS1 kernel: [ 10339] 0 10339 4477 591 448 143 0 81920 0 0 sshd
Nov 9 01:01:53 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:53 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:53 TNAS1 kernel: [ 11235] 0 11235 19579 797 640 82 75 135168 0 0 wb-idmap
Nov 9 01:01:53 TNAS1 kernel: [2479756] 999 2479756 1113 241 160 81 0 49152 0 -900 bash
Nov 9 01:01:53 TNAS1 kernel: [2709452] 0 2709452 94214 93 32 61 0 86016 0 0 nscd
Nov 9 01:01:53 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:53 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:53 TNAS1 kernel: [2776151] 0 2776151 207108 13004 12886 118 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:53 TNAS1 kernel: [2777444] 0 2777444 206501 12405 12278 127 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:53 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:53 TNAS1 kernel: [2782309] 0 2782309 7027 3092 2944 148 0 90112 0 0 truenas_audit_h
Nov 9 01:01:53 TNAS1 kernel: [2782311] 0 2782311 187783 644 512 132 0 417792 0 0 syslog-ng
Nov 9 01:01:53 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tnaudit.service,task=truenas_audit_h,pid=2782309,uid=0
Nov 9 01:01:53 TNAS1 kernel: Out of memory: Killed process 2782309 (truenas_audit_h) total-vm:28108kB, anon-rss:11776kB, file-rss:592kB, shmem-rss:0kB, UID:0 pgtables:88kB oom_score_adj:0
Nov 9 01:01:53 TNAS1 systemd[1]: tnaudit.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:53 TNAS1 systemd[1]: tnaudit.service: Failed with result 'signal'.
Nov 9 01:01:53 TNAS1 systemd[1]: Starting tnaudit.service - TrueNAS Auditd Handler...
Nov 9 01:01:53 TNAS1 systemd[1]: Started tnaudit.service - TrueNAS Auditd Handler.
Nov 9 01:01:55 TNAS1 syslog-ng[2782363]: syslog-ng starting up; version='3.38.1'
Nov 9 01:01:54 TNAS1 kernel: IoThread invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-1000
Nov 9 01:01:54 TNAS1 kernel: CPU: 4 UID: 0 PID: 7229 Comm: IoThread Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:54 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:54 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:54 TNAS1 kernel: Call Trace:
Nov 9 01:01:54 TNAS1 kernel:
Nov 9 01:01:54 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:54 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:54 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:54 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:54 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:54 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:54 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:54 TNAS1 kernel: get_free_pages_noprof+0x11/0x40
Nov 9 01:01:54 TNAS1 kernel: proc_pid_cmdline_read+0x1c1/0x3e0
Nov 9 01:01:54 TNAS1 kernel: vfs_read+0xeb/0x370
Nov 9 01:01:54 TNAS1 kernel: ? syscall_exit_to_user_mode+0x1c5/0x1f0
Nov 9 01:01:54 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:54 TNAS1 kernel: ? auditd_test_task+0x3d/0x50
Nov 9 01:01:54 TNAS1 kernel: ? mutex_lock+0x12/0x30
Nov 9 01:01:54 TNAS1 kernel: ksys_read+0x6f/0xf0
Nov 9 01:01:54 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:54 TNAS1 kernel: ? switch_fpu_return+0x4f/0xd0
Nov 9 01:01:54 TNAS1 kernel: ? syscall_exit_to_user_mode+0x1c5/0x1f0
Nov 9 01:01:54 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:54 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:01:54 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:54 TNAS1 kernel: RIP: 0033:0x7ff07f45f25c
Nov 9 01:01:54 TNAS1 kernel: Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 d9 d5 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 2f d6 f8 ff 48
Nov 9 01:01:54 TNAS1 kernel: RSP: 002b:00007fefb5ffa700 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Nov 9 01:01:54 TNAS1 kernel: RAX: ffffffffffffffda RBX: 00007fefb5ffb640 RCX: 00007ff07f45f25c
Nov 9 01:01:54 TNAS1 kernel: RDX: 0000000000002000 RSI: 00007fefa400d950 RDI: 000000000000004d
Nov 9 01:01:54 TNAS1 kernel: RBP: 00007fefc0000ba0 R08: 0000000000000000 R09: 0000000000a4db28
Nov 9 01:01:54 TNAS1 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007fe893835fc0
Nov 9 01:01:54 TNAS1 kernel: R13: 00007fefc0000ba0 R14: 00007fefa400d950 R15: 000000000000004d
Nov 9 01:01:54 TNAS1 kernel:
Nov 9 01:01:54 TNAS1 kernel: Mem-Info:
Nov 9 01:01:54 TNAS1 kernel: active_anon:65321 inactive_anon:7673049 isolated_anon:0
active_file:1015 inactive_file:1015 isolated_file:0
unevictable:1000 dirty:169 writeback:293
slab_reclaimable:58278 slab_unreclaimable:81734
mapped:2504 shmem:92845 pagetables:16757
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:29039 free_pcp:239 free_cma:0
Nov 9 01:01:54 TNAS1 kernel: Node 0 active_anon:261284kB inactive_anon:30692196kB active_file:4060kB inactive_file:4060kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:10016kB dirty:676kB writeback:1172kB shmem:371380kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:792576kB writeback_tmp:0kB kernel_stack:12272kB pagetables:67028kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:54 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA32 free:9788kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3475604kB active_file:1200kB inactive_file:652kB unevictable:0kB writepending:192kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:436kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:54 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:54 TNAS1 kernel: Node 0 Normal free:106340kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261280kB inactive_anon:27205064kB active_file:3384kB inactive_file:3368kB unevictable:4000kB writepending:1656kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:508kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:54 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA32: 158*4kB (UME) 63*8kB (UME) 57*16kB (UME) 42*32kB (UME) 33*64kB (UME) 4*128kB (ME) 4*256kB (M) 1*512kB (M) 0*1024kB 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 9600kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 Normal: 1257*4kB (UME) 445*8kB (UME) 1768*16kB (UME) 2147*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 105580kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:54 TNAS1 kernel: 95176 total pagecache pages
Nov 9 01:01:54 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:54 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:54 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:54 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:54 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:54 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:54 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:54 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:54 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:54 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:54 TNAS1 kernel: [ 728] 104 728 2091 376 256 120 0 65536 0 -900 dbus-daemon
Nov 9 01:01:54 TNAS1 kernel: [ 741] 0 741 11506 852 224 628 0 106496 0 -250 systemd-journal
Nov 9 01:01:54 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:54 TNAS1 kernel: [ 1630] 0 1630 8351328 7476423 7476074 285 64 60686336 0 -1000 asyncio_loop
Nov 9 01:01:54 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:54 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:54 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:54 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:54 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:54 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:54 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:54 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:54 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:54 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:54 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:54 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:54 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:54 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:54 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:54 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:54 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:54 TNAS1 kernel: [ 3691] 0 3691 12416 471 224 247 0 94208 0 0 systemd-logind
Nov 9 01:01:54 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:54 TNAS1 kernel: [ 3708] 0 3708 523983 4212 4140 72 0 327680 0 -999 containerd
Nov 9 01:01:54 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:54 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:54 TNAS1 kernel: [ 3962] 122 3962 1695 236 96 140 0 57344 0 0 upsd
Nov 9 01:01:54 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:54 TNAS1 kernel: [ 4009] 122 4009 2127 192 96 96 0 53248 0 0 upsmon
Nov 9 01:01:54 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:54 TNAS1 kernel: [ 4132] 0 4132 4136 460 224 236 0 69632 0 0 systemd-machine
Nov 9 01:01:54 TNAS1 kernel: [ 4133] 0 4133 427247 2943 2880 63 0 339968 0 0 libvirtd
Nov 9 01:01:54 TNAS1 kernel: [ 5386] 105 5386 1781 152 96 56 0 49152 0 0 avahi-daemon
Nov 9 01:01:54 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:54 TNAS1 kernel: [ 5694] 0 5694 17601 616 544 72 0 118784 0 0 nmbd
Nov 9 01:01:54 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:54 TNAS1 kernel: [ 6672] 0 6672 13921 5825 5472 353 0 139264 0 -1000 websockify
Nov 9 01:01:54 TNAS1 kernel: [ 6757] 999 6757 116708 29501 29218 219 64 475136 0 -900 netdata
Nov 9 01:01:54 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:54 TNAS1 kernel: [ 7021] 999 7021 22248 5460 4846 614 0 147456 0 -900 python.d.plugin
Nov 9 01:01:54 TNAS1 kernel: [ 7210] 0 7210 18963 733 608 29 96 131072 0 0 winbindd
Nov 9 01:01:54 TNAS1 kernel: [ 7212] 0 7212 18982 747 640 69 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:54 TNAS1 kernel: [ 7230] 0 7230 13921 5833 5413 420 0 131072 0 -1000 websockify
Nov 9 01:01:54 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:54 TNAS1 kernel: [ 10337] 0 10337 4476 501 416 85 0 73728 0 0 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 10339] 0 10339 4477 591 448 143 0 81920 0 0 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:54 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 11235] 0 11235 19579 797 640 82 75 135168 0 0 wb-idmap
Nov 9 01:01:54 TNAS1 kernel: [2479756] 999 2479756 1113 390 160 230 0 49152 0 -900 bash
Nov 9 01:01:54 TNAS1 kernel: [2709452] 0 2709452 94214 93 32 61 0 86016 0 0 nscd
Nov 9 01:01:54 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2776151] 0 2776151 207108 13004 12886 118 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2777444] 0 2777444 206501 12405 12278 127 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2782311] 0 2782311 187783 842 512 330 0 417792 0 0 syslog-ng
Nov 9 01:01:54 TNAS1 kernel: [2782328] 0 2782328 3546 1548 768 780 0 61440 0 0 truenas_audit_h
Nov 9 01:01:54 TNAS1 kernel: [2782335] 999 2782335 623 182 0 182 0 45056 0 -900 timeout
Nov 9 01:01:54 TNAS1 kernel: [2782336] 999 2782336 2119 701 96 605 0 57344 0 -900 upsc
Nov 9 01:01:54 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=middlewared.service,mems_allowed=0,global_oom,task_memcg=/system.slice/libvirtd.service,task=libvirtd,pid=4133,uid=0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 4133 (libvirtd) total-vm:1708988kB, anon-rss:11520kB, file-rss:252kB, shmem-rss:0kB, UID:0 pgtables:332kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 systemd[1]: libvirtd.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:54 TNAS1 systemd[1]: libvirtd.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:54 TNAS1 systemd[1]: libvirtd.service: Failed with result 'oom-kill'.
Nov 9 01:01:54 TNAS1 systemd[1]: libvirtd.service: Consumed 1min 7.585s CPU time.
Nov 9 01:01:54 TNAS1 kernel: IoThread invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-1000
Nov 9 01:01:54 TNAS1 kernel: CPU: 1 UID: 0 PID: 7229 Comm: IoThread Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:54 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:54 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:54 TNAS1 kernel: Call Trace:
Nov 9 01:01:54 TNAS1 kernel:
Nov 9 01:01:54 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:54 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:54 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:54 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:54 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:54 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:54 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:54 TNAS1 kernel: get_free_pages_noprof+0x11/0x40
Nov 9 01:01:54 TNAS1 kernel: proc_pid_cmdline_read+0x1c1/0x3e0
Nov 9 01:01:54 TNAS1 kernel: vfs_read+0xeb/0x370
Nov 9 01:01:54 TNAS1 kernel: ? syscall_exit_to_user_mode+0x1c5/0x1f0
Nov 9 01:01:54 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:54 TNAS1 kernel: ? auditd_test_task+0x3d/0x50
Nov 9 01:01:54 TNAS1 kernel: ? mutex_lock+0x12/0x30
Nov 9 01:01:54 TNAS1 kernel: ksys_read+0x6f/0xf0
Nov 9 01:01:54 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:54 TNAS1 kernel: ? switch_fpu_return+0x4f/0xd0
Nov 9 01:01:54 TNAS1 kernel: ? syscall_exit_to_user_mode+0x1c5/0x1f0
Nov 9 01:01:54 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:54 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:01:54 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:54 TNAS1 kernel: RIP: 0033:0x7ff07f45f25c
Nov 9 01:01:54 TNAS1 kernel: Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 d9 d5 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 2f d6 f8 ff 48
Nov 9 01:01:54 TNAS1 kernel: RSP: 002b:00007fefb5ffa700 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Nov 9 01:01:54 TNAS1 kernel: RAX: ffffffffffffffda RBX: 00007fefb5ffb640 RCX: 00007ff07f45f25c
Nov 9 01:01:54 TNAS1 kernel: RDX: 0000000000002000 RSI: 00007fefa400d950 RDI: 000000000000004d
Nov 9 01:01:54 TNAS1 kernel: RBP: 00007fefc0000ba0 R08: 0000000000000000 R09: 0000000000a4db28
Nov 9 01:01:54 TNAS1 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007fe893835fc0
Nov 9 01:01:54 TNAS1 kernel: R13: 00007fefc0000ba0 R14: 00007fefa400d950 R15: 000000000000004d
Nov 9 01:01:54 TNAS1 kernel:
Nov 9 01:01:54 TNAS1 kernel: Mem-Info:
Nov 9 01:01:54 TNAS1 kernel: active_anon:65321 inactive_anon:7670725 isolated_anon:0
active_file:1532 inactive_file:220 isolated_file:0
unevictable:1000 dirty:202 writeback:334
slab_reclaimable:58266 slab_unreclaimable:81663
mapped:2098 shmem:92844 pagetables:16685
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:31500 free_pcp:254 free_cma:0
Nov 9 01:01:54 TNAS1 kernel: Node 0 active_anon:261284kB inactive_anon:30682900kB active_file:5836kB inactive_file:1172kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:8392kB dirty:808kB writeback:1336kB shmem:371376kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:792576kB writeback_tmp:0kB kernel_stack:12048kB pagetables:66740kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:54 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA32 free:7176kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3477680kB active_file:1108kB inactive_file:984kB unevictable:0kB writepending:408kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:54 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:54 TNAS1 kernel: Node 0 Normal free:119712kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261280kB inactive_anon:27194104kB active_file:2736kB inactive_file:1616kB unevictable:4000kB writepending:1736kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:172kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:54 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA32: 41*4kB (UME) 34*8kB (UME) 56*16kB (UME) 34*32kB (UME) 33*64kB (UME) 4*128kB (ME) 1*256kB (M) 0*512kB 0*1024kB 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7348kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 Normal: 3526*4kB (UME) 706*8kB (UME) 1808*16kB (UME) 2163*32kB (UME) 8*64kB (M) 5*128kB (M) 4*256kB (M) 1*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 121608kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:54 TNAS1 kernel: 94673 total pagecache pages
Nov 9 01:01:54 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:54 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:54 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:54 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:54 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:54 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:54 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:54 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:54 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:54 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:54 TNAS1 kernel: [ 728] 104 728 2091 409 256 153 0 65536 0 -900 dbus-daemon
Nov 9 01:01:54 TNAS1 kernel: [ 741] 0 741 11506 916 224 692 0 106496 0 -250 systemd-journal
Nov 9 01:01:54 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:54 TNAS1 kernel: [ 1630] 0 1630 8351840 7476871 7476522 285 64 60686336 0 -1000 asyncio_loop
Nov 9 01:01:54 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:54 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:54 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:54 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:54 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:54 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:54 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:54 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:54 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:54 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:54 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:54 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:54 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:54 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:54 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:54 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:54 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:54 TNAS1 kernel: [ 3691] 0 3691 12416 471 224 247 0 94208 0 0 systemd-logind
Nov 9 01:01:54 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:54 TNAS1 kernel: [ 3708] 0 3708 523983 4244 4140 104 0 327680 0 -999 containerd
Nov 9 01:01:54 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:54 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:54 TNAS1 kernel: [ 3962] 122 3962 1695 268 96 172 0 57344 0 0 upsd
Nov 9 01:01:54 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:54 TNAS1 kernel: [ 4009] 122 4009 2127 192 96 96 0 53248 0 0 upsmon
Nov 9 01:01:54 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:54 TNAS1 kernel: [ 4132] 0 4132 4136 460 224 236 0 69632 0 0 systemd-machine
Nov 9 01:01:54 TNAS1 kernel: [ 5386] 105 5386 1781 267 96 171 0 49152 0 0 avahi-daemon
Nov 9 01:01:54 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:54 TNAS1 kernel: [ 5694] 0 5694 17601 616 544 72 0 118784 0 0 nmbd
Nov 9 01:01:54 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:54 TNAS1 kernel: [ 6672] 0 6672 13921 5825 5472 353 0 139264 0 -1000 websockify
Nov 9 01:01:54 TNAS1 kernel: [ 6757] 999 6757 116708 29469 29218 187 64 475136 0 -900 netdata
Nov 9 01:01:54 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:54 TNAS1 kernel: [ 7021] 999 7021 22248 5364 4846 518 0 147456 0 -900 python.d.plugin
Nov 9 01:01:54 TNAS1 kernel: [ 7210] 0 7210 18963 733 608 29 96 131072 0 0 winbindd
Nov 9 01:01:54 TNAS1 kernel: [ 7212] 0 7212 18982 747 640 69 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:54 TNAS1 kernel: [ 7230] 0 7230 13921 5769 5413 356 0 131072 0 -1000 websockify
Nov 9 01:01:54 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:54 TNAS1 kernel: [ 10337] 0 10337 4476 501 416 85 0 73728 0 0 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 10339] 0 10339 4477 591 448 143 0 81920 0 0 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:54 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 11235] 0 11235 19579 797 640 82 75 135168 0 0 wb-idmap
Nov 9 01:01:54 TNAS1 kernel: [2479756] 999 2479756 1113 390 160 230 0 49152 0 -900 bash
Nov 9 01:01:54 TNAS1 kernel: [2709452] 0 2709452 94214 93 32 61 0 86016 0 0 nscd
Nov 9 01:01:54 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2776151] 0 2776151 207108 13004 12886 118 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2777444] 0 2777444 206501 12405 12278 127 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2782311] 0 2782311 187783 778 512 266 0 417792 0 0 syslog-ng
Nov 9 01:01:54 TNAS1 kernel: [2782328] 0 2782328 3739 1644 992 652 0 61440 0 0 truenas_audit_h
Nov 9 01:01:54 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=middlewared.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tnaudit.service,task=truenas_audit_h,pid=2782328,uid=0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 2782328 (truenas_audit_h) total-vm:14956kB, anon-rss:3968kB, file-rss:2608kB, shmem-rss:0kB, UID:0 pgtables:60kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 systemd[1]: tnaudit.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:54 TNAS1 kernel: systemd-journal invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-250
Nov 9 01:01:54 TNAS1 kernel: CPU: 3 UID: 0 PID: 741 Comm: systemd-journal Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:54 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:54 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:54 TNAS1 kernel: Call Trace:
Nov 9 01:01:54 TNAS1 kernel:
Nov 9 01:01:54 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:54 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:54 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:54 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:54 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:54 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:54 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:54 TNAS1 kernel: get_free_pages_noprof+0x11/0x40
Nov 9 01:01:54 TNAS1 kernel: proc_pid_cmdline_read+0x1c1/0x3e0
Nov 9 01:01:54 TNAS1 kernel: vfs_read+0xeb/0x370
Nov 9 01:01:54 TNAS1 kernel: ? __seccomp_filter+0x32c/0x500
Nov 9 01:01:54 TNAS1 kernel: ksys_read+0x6f/0xf0
Nov 9 01:01:54 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:54 TNAS1 kernel: ? syscall_exit_to_user_mode+0x10/0x1f0
Nov 9 01:01:54 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:54 TNAS1 kernel: ? do_readlinkat+0xc2/0x180
Nov 9 01:01:54 TNAS1 kernel: ? syscall_exit_to_user_mode+0x10/0x1f0
Nov 9 01:01:54 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:01:54 TNAS1 kernel: ? __count_memcg_events+0x53/0xf0
Nov 9 01:01:54 TNAS1 kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 9 01:01:54 TNAS1 kernel: ? handle_mm_fault+0xae/0x2d0
Nov 9 01:01:54 TNAS1 kernel: ? do_user_addr_fault+0x379/0x670
Nov 9 01:01:54 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:01:54 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:54 TNAS1 kernel: RIP: 0033:0x7f2fdb3f125c
Nov 9 01:01:54 TNAS1 kernel: Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 d9 d5 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 2f d6 f8 ff 48
Nov 9 01:01:54 TNAS1 kernel: RSP: 002b:00007ffe05039770 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Nov 9 01:01:54 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000563c0c0525b0 RCX: 00007f2fdb3f125c
Nov 9 01:01:54 TNAS1 kernel: RDX: 0000000000001008 RSI: 0000563c0c0525b0 RDI: 0000000000000016
Nov 9 01:01:54 TNAS1 kernel: RBP: 0000000000000016 R08: 0000000000000000 R09: 00007f2fdb4cc350
Nov 9 01:01:54 TNAS1 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001008
Nov 9 01:01:54 TNAS1 kernel: R13: 0000000000001007 R14: ffffffffffffffff R15: 0000000000000001
Nov 9 01:01:54 TNAS1 kernel:
Nov 9 01:01:54 TNAS1 kernel: Mem-Info:
Nov 9 01:01:54 TNAS1 kernel: active_anon:65321 inactive_anon:7669751 isolated_anon:0
active_file:1145 inactive_file:301 isolated_file:0
unevictable:1000 dirty:202 writeback:334
slab_reclaimable:58266 slab_unreclaimable:81663
mapped:1764 shmem:92844 pagetables:16685
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:32869 free_pcp:0 free_cma:0
Nov 9 01:01:54 TNAS1 kernel: Node 0 active_anon:261284kB inactive_anon:30679004kB active_file:4580kB inactive_file:1204kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:6764kB dirty:808kB writeback:1336kB shmem:371376kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:792576kB writeback_tmp:0kB kernel_stack:12048kB pagetables:66740kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:54 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA32 free:9920kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3475216kB active_file:1136kB inactive_file:476kB unevictable:0kB writepending:408kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:54 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:54 TNAS1 kernel: Node 0 Normal free:121964kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261280kB inactive_anon:27193096kB active_file:2840kB inactive_file:1396kB unevictable:4000kB writepending:1736kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:54 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 32kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 DMA32: 320*4kB (UME) 127*8kB (UME) 110*16kB (UME) 48*32kB (UME) 34*64kB (UME) 5*128kB (ME) 1*256kB (M) 0*512kB 0*1024kB 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 10712kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 Normal: 3854*4kB (UMEC) 782*8kB (UME) 1813*16kB (UME) 2167*32kB (UMEC) 8*64kB (M) 5*128kB (M) 4*256kB (M) 1*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 123736kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:54 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:54 TNAS1 kernel: 94495 total pagecache pages
Nov 9 01:01:54 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:54 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:54 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:54 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:54 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:54 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:54 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:54 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:54 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:54 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:54 TNAS1 kernel: [ 728] 104 728 2091 409 256 153 0 65536 0 -900 dbus-daemon
Nov 9 01:01:54 TNAS1 kernel: [ 741] 0 741 11506 884 224 660 0 106496 0 -250 systemd-journal
Nov 9 01:01:54 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:54 TNAS1 kernel: [ 1630] 0 1630 8351840 7476807 7476522 221 64 60686336 0 -1000 asyncio_loop
Nov 9 01:01:54 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:54 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:54 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:54 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:54 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:54 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:54 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:54 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:54 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:54 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:54 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:54 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:54 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:54 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:54 TNAS1 kernel: [ 3635] 131 3635 4715 183 138 45 0 61440 0 0 chronyd
Nov 9 01:01:54 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:54 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:54 TNAS1 kernel: [ 3691] 0 3691 12416 439 224 215 0 94208 0 0 systemd-logind
Nov 9 01:01:54 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:54 TNAS1 kernel: [ 3708] 0 3708 523983 4212 4140 72 0 327680 0 -999 containerd
Nov 9 01:01:54 TNAS1 kernel: [ 3783] 0 3783 3857 442 288 154 0 73728 0 -1000 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:54 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:54 TNAS1 kernel: [ 3962] 122 3962 1695 268 96 172 0 57344 0 0 upsd
Nov 9 01:01:54 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:54 TNAS1 kernel: [ 4009] 122 4009 2127 192 96 96 0 53248 0 0 upsmon
Nov 9 01:01:54 TNAS1 kernel: [ 4063] 0 4063 7296 564 436 128 0 73728 0 0 nginx
Nov 9 01:01:54 TNAS1 kernel: [ 4132] 0 4132 4136 460 224 236 0 69632 0 0 systemd-machine
Nov 9 01:01:54 TNAS1 kernel: [ 5386] 105 5386 1781 235 96 139 0 49152 0 0 avahi-daemon
Nov 9 01:01:54 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:54 TNAS1 kernel: [ 5694] 0 5694 17601 616 544 72 0 118784 0 0 nmbd
Nov 9 01:01:54 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:54 TNAS1 kernel: [ 6672] 0 6672 13921 5793 5472 321 0 139264 0 -1000 websockify
Nov 9 01:01:54 TNAS1 kernel: [ 6757] 999 6757 116708 29469 29218 187 64 475136 0 -900 netdata
Nov 9 01:01:54 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:54 TNAS1 kernel: [ 7021] 999 7021 22248 5204 4846 358 0 147456 0 -900 python.d.plugin
Nov 9 01:01:54 TNAS1 kernel: [ 7210] 0 7210 18963 733 608 29 96 131072 0 0 winbindd
Nov 9 01:01:54 TNAS1 kernel: [ 7212] 0 7212 18982 747 640 69 38 131072 0 0 wb[TNAS1]
Nov 9 01:01:54 TNAS1 kernel: [ 7230] 0 7230 13921 5673 5413 260 0 131072 0 -1000 websockify
Nov 9 01:01:54 TNAS1 kernel: [ 7371] 33 7371 7296 664 567 97 0 69632 0 0 nginx
Nov 9 01:01:54 TNAS1 kernel: [ 10337] 0 10337 4476 501 416 85 0 73728 0 0 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 10339] 0 10339 4477 591 448 143 0 81920 0 0 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 10366] 0 10366 2075 866 699 167 0 61440 0 0 zsh
Nov 9 01:01:54 TNAS1 kernel: [ 10378] 0 10378 4412 526 397 129 0 81920 0 0 sshd
Nov 9 01:01:54 TNAS1 kernel: [ 11235] 0 11235 19579 797 640 82 75 135168 0 0 wb-idmap
Nov 9 01:01:54 TNAS1 kernel: [2479756] 999 2479756 1113 390 160 230 0 49152 0 -900 bash
Nov 9 01:01:54 TNAS1 kernel: [2709452] 0 2709452 94214 93 32 61 0 86016 0 0 nscd
Nov 9 01:01:54 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2776151] 0 2776151 207108 13004 12886 118 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2777444] 0 2777444 206501 12405 12278 127 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:54 TNAS1 kernel: [2782311] 0 2782311 187783 778 512 266 0 417792 0 0 syslog-ng
Nov 9 01:01:54 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=systemd-journald.service,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-1.scope,task=zsh,pid=10366,uid=0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 10366 (zsh) total-vm:8300kB, anon-rss:2796kB, file-rss:668kB, shmem-rss:0kB, UID:0 pgtables:60kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 2782311 (syslog-ng) total-vm:751132kB, anon-rss:2048kB, file-rss:1064kB, shmem-rss:0kB, UID:0 pgtables:408kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 11235 (wb-idmap) total-vm:78316kB, anon-rss:2560kB, file-rss:328kB, shmem-rss:300kB, UID:0 pgtables:132kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 7210 (winbindd) total-vm:75852kB, anon-rss:2432kB, file-rss:396kB, shmem-rss:384kB, UID:0 pgtables:128kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 10339 (sshd) total-vm:17908kB, anon-rss:1792kB, file-rss:1148kB, shmem-rss:0kB, UID:0 pgtables:80kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 7371 (nginx) total-vm:29184kB, anon-rss:2268kB, file-rss:388kB, shmem-rss:0kB, UID:33 pgtables:68kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 10337 (sshd) total-vm:17904kB, anon-rss:1664kB, file-rss:860kB, shmem-rss:0kB, UID:0 pgtables:72kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 5694 (nmbd) total-vm:70404kB, anon-rss:2176kB, file-rss:688kB, shmem-rss:0kB, UID:0 pgtables:116kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 kernel: Out of memory: Killed process 4063 (nginx) total-vm:29184kB, anon-rss:1744kB, file-rss:780kB, shmem-rss:0kB, UID:0 pgtables:72kB oom_score_adj:0
Nov 9 01:01:54 TNAS1 nginx[4063]: 2025/11/09 01:01:54 [alert] 4063#4063: worker process 7371 exited on signal 9
Nov 9 01:01:54 TNAS1 systemd[1]: tnaudit.service: Failed with result 'signal'.
Nov 9 01:01:54 TNAS1 systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 1.
Nov 9 01:01:54 TNAS1 systemd[1]: Stopped libvirtd.service - Virtualization daemon.
Nov 9 01:01:54 TNAS1 systemd[1]: libvirtd.service: Consumed 1min 7.585s CPU time.
Nov 9 01:01:54 TNAS1 systemd[1]: Starting libvirtd.service - Virtualization daemon...
Nov 9 01:01:54 TNAS1 systemd[1]: Starting tnaudit.service - TrueNAS Auditd Handler...
Nov 9 01:01:54 TNAS1 systemd[1]: nmbd.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:54 TNAS1 systemd[1]: nginx.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:54 TNAS1 systemd[1]: session-3.scope: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:54 TNAS1 systemd[1]: winbind.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:54 TNAS1 systemd[1]: session-1.scope: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:54 TNAS1 systemd[1]: nginx.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:54 TNAS1 systemd[1]: nmbd.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:54 TNAS1 systemd[1]: nmbd.service: Failed with result 'oom-kill'.
Nov 9 01:01:54 TNAS1 start-stop-daemon[2782350]: /sbin/start-stop-daemon: warning: failed to kill 4063: No such process
Nov 9 01:01:54 TNAS1 systemd[1]: nmbd.service: Consumed 3.789s CPU time.
Nov 9 01:01:54 TNAS1 systemd[1]: winbind.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:54 TNAS1 systemd[1]: winbind.service: Failed with result 'oom-kill'.
Nov 9 01:01:54 TNAS1 systemd[1]: winbind.service: Consumed 3.607s CPU time.
Nov 9 01:01:54 TNAS1 systemd[1]: syslog-ng.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:54 TNAS1 systemd[1]: syslog-ng.service: Failed with result 'signal'.
Nov 9 01:01:54 TNAS1 systemd[1]: system.slice: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:54 TNAS1 systemd[1]: nginx.service: Failed with result 'oom-kill'.
Nov 9 01:01:54 TNAS1 systemd[1]: nginx.service: Consumed 6.896s CPU time.
Nov 9 01:01:54 TNAS1 systemd[1]: session-1.scope: Deactivated successfully.
Nov 9 01:01:54 TNAS1 systemd[1]: session-1.scope: Consumed 36min 26.620s CPU time.
Nov 9 01:01:54 TNAS1 systemd[1]: user-0.slice: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:54 TNAS1 systemd[1]: user.slice: A process of this unit has been killed by the OOM killer.
Nov 9 01:01:54 TNAS1 systemd[1]: session-3.scope: Deactivated successfully.
Nov 9 01:01:54 TNAS1 systemd[1]: session-3.scope: Consumed 1.070s CPU time.
Nov 9 01:01:54 TNAS1 systemd[1]: Stopping user-runtime-dir@0.service - User Runtime Directory /run/user/0...
Nov 9 01:01:54 TNAS1 systemd[1]: Started tnaudit.service - TrueNAS Auditd Handler.
Nov 9 01:01:54 TNAS1 systemd[1]: run-user-0.mount: Deactivated successfully.
Nov 9 01:01:54 TNAS1 systemd[1]: user-runtime-dir@0.service: Deactivated successfully.
Nov 9 01:01:54 TNAS1 systemd[1]: Stopped user-runtime-dir@0.service - User Runtime Directory /run/user/0.
Nov 9 01:01:54 TNAS1 systemd[1]: Removed slice user-0.slice - User Slice of UID 0.
Nov 9 01:01:54 TNAS1 systemd[1]: user-0.slice: Consumed 36min 28.107s CPU time.
Nov 9 01:01:55 TNAS1 systemd[1]: syslog-ng.service: Scheduled restart job, restart counter is at 2.
Nov 9 01:01:55 TNAS1 systemd[1]: Stopped syslog-ng.service - System Logger Daemon.
Nov 9 01:01:55 TNAS1 systemd[1]: Starting syslog-ng.service - System Logger Daemon...
Nov 9 01:01:55 TNAS1 systemd[1]: Started libvirtd.service - Virtualization daemon.
Nov 9 01:01:55 TNAS1 kernel: oom_kill_process: 8 callbacks suppressed
Nov 9 01:01:55 TNAS1 kernel: dnsmasq invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:01:55 TNAS1 kernel: CPU: 2 UID: 0 PID: 2782372 Comm: dnsmasq Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:55 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:55 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:55 TNAS1 kernel: Call Trace:
Nov 9 01:01:55 TNAS1 kernel:
Nov 9 01:01:55 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:55 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:55 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:55 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:55 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:55 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:55 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:55 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:01:55 TNAS1 kernel: __do_fault+0xa3/0x170
Nov 9 01:01:55 TNAS1 kernel: do_fault+0x3de/0x4d0
Nov 9 01:01:55 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:01:55 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:01:55 TNAS1 kernel: do_user_addr_fault+0x15d/0x670
Nov 9 01:01:55 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:01:55 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:01:55 TNAS1 kernel: RIP: 0010:rep_stos_alternative+0x40/0x80
Nov 9 01:01:55 TNAS1 kernel: Code: ff c7 48 ff c9 75 f6 c3 cc cc cc cc 48 89 07 48 83 c7 08 83 e9 08 74 ef 83 f9 08 73 ef eb de 66 66 2e 0f 1f 84 00 00 00 00 00 <48> 89 07 48 89 47 08 48 89 47 10 48 89 47 18 48 89 47 20 48 89 47
Nov 9 01:01:55 TNAS1 kernel: RSP: 0018:ffffac8092e8fd18 EFLAGS: 00010202
Nov 9 01:01:55 TNAS1 kernel: RAX: 0000000000000000 RBX: 000055fd60df5bf0 RCX: 0000000000000410
Nov 9 01:01:55 TNAS1 kernel: RDX: 000055fd60df61d8 RSI: 0000000000000000 RDI: 000055fd60df5bf0
Nov 9 01:01:55 TNAS1 kernel: RBP: ffff8ac4489ba918 R08: 000055fd60df6000 R09: 000000055fd60d80
Nov 9 01:01:55 TNAS1 kernel: R10: 000055fd60df1000 R11: 0000000000000001 R12: 0000000000000003
Nov 9 01:01:55 TNAS1 kernel: R13: 000055fd60df1000 R14: ffff8ac14be40000 R15: 0000000000000000
Nov 9 01:01:55 TNAS1 kernel: elf_load+0x236/0x260
Nov 9 01:01:55 TNAS1 kernel: load_elf_binary+0x606/0x1720
Nov 9 01:01:55 TNAS1 kernel: ? load_misc_binary+0x20d/0x350 [binfmt_misc]
Nov 9 01:01:55 TNAS1 kernel: bprm_execve+0x23c/0x610
Nov 9 01:01:55 TNAS1 kernel: do_execveat_common.isra.0+0x17f/0x1e0
Nov 9 01:01:55 TNAS1 kernel: __x64_sys_execve+0x36/0x40
Nov 9 01:01:55 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:55 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:01:55 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:55 TNAS1 kernel: RIP: 0033:0x7f447d951a97
Nov 9 01:01:55 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7f447d951a6d.
Nov 9 01:01:55 TNAS1 kernel: RSP: 002b:00007f4451ffa7f8 EFLAGS: 00000206 ORIG_RAX: 000000000000003b
Nov 9 01:01:55 TNAS1 kernel: RAX: ffffffffffffffda RBX: 00007f442c0206b0 RCX: 00007f447d951a97
Nov 9 01:01:55 TNAS1 kernel: RDX: 00007f4424008f90 RSI: 00007f442c01bf10 RDI: 00007f442c01bd80
Nov 9 01:01:55 TNAS1 kernel: RBP: 00007f4451ffa8c0 R08: 00007f4451ffb5e0 R09: 0000000000000000
Nov 9 01:01:55 TNAS1 kernel: R10: 0000000000000000 R11: 0000000000000206 R12: 00000000ffffffff
Nov 9 01:01:55 TNAS1 kernel: R13: 00007f4451ffa860 R14: 00007f44240091d0 R15: 00000000002a74a4
Nov 9 01:01:55 TNAS1 kernel:
Nov 9 01:01:55 TNAS1 kernel: Mem-Info:
Nov 9 01:01:55 TNAS1 kernel: active_anon:65305 inactive_anon:7670961 isolated_anon:0
active_file:1276 inactive_file:40 isolated_file:0
unevictable:1000 dirty:298 writeback:425
slab_reclaimable:57978 slab_unreclaimable:81054
mapped:1722 shmem:92803 pagetables:16426
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:31504 free_pcp:63 free_cma:565
Nov 9 01:01:55 TNAS1 kernel: Node 0 active_anon:261220kB inactive_anon:30683844kB active_file:5104kB inactive_file:160kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:6888kB dirty:1192kB writeback:1700kB shmem:371212kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11852kB pagetables:65704kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:55 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11232kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:55 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:55 TNAS1 kernel: Node 0 DMA32 free:7052kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3478772kB active_file:392kB inactive_file:680kB unevictable:0kB writepending:720kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:28kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:55 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:55 TNAS1 kernel: Node 0 Normal free:118936kB boost:77824kB min:138216kB low:167476kB high:196736kB reserved_highatomic:4096KB active_anon:261216kB inactive_anon:27193172kB active_file:3152kB inactive_file:612kB unevictable:4000kB writepending:2172kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:224kB local_pcp:0kB free_cma:2260kB
Nov 9 01:01:55 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:55 TNAS1 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 32kB
Nov 9 01:01:55 TNAS1 kernel: Node 0 DMA32: 100*4kB (UME) 54*8kB (UME) 58*16kB (UME) 35*32kB (UME) 35*64kB (UME) 3*128kB (ME) 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7552kB
Nov 9 01:01:55 TNAS1 kernel: Node 0 Normal: 3274*4kB (UMEC) 664*8kB (UMEC) 1743*16kB (UMEC) 2140*32kB (UMEC) 7*64kB (C) 6*128kB (MC) 0*256kB 1*512kB (M) 1*1024kB (M) 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 119576kB
Nov 9 01:01:55 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:55 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:55 TNAS1 kernel: 94351 total pagecache pages
Nov 9 01:01:55 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:55 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:55 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:55 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:55 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:55 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:55 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:55 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:55 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:55 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:55 TNAS1 kernel: [ 728] 104 728 2091 461 256 205 0 65536 0 -900 dbus-daemon
Nov 9 01:01:55 TNAS1 kernel: [ 741] 0 741 11506 904 224 680 0 106496 0 -250 systemd-journal
Nov 9 01:01:55 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:55 TNAS1 kernel: [ 1630] 0 1630 8359520 7484557 7484298 195 64 60751872 0 -1000 asyncio_loop
Nov 9 01:01:55 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:55 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:55 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:55 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:55 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:55 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:55 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:55 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:55 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:55 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:55 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:55 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:55 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:55 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:55 TNAS1 kernel: [ 3635] 131 3635 4715 224 138 86 0 61440 0 0 chronyd
Nov 9 01:01:55 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:55 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:55 TNAS1 kernel: [ 3691] 0 3691 12416 454 224 230 0 94208 0 0 systemd-logind
Nov 9 01:01:55 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:55 TNAS1 kernel: [ 3708] 0 3708 523983 4212 4140 72 0 327680 0 -999 containerd
Nov 9 01:01:55 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:01:55 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:55 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:55 TNAS1 kernel: [ 3962] 122 3962 1695 208 96 112 0 57344 0 0 upsd
Nov 9 01:01:55 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:55 TNAS1 kernel: [ 4009] 122 4009 2127 192 96 96 0 53248 0 0 upsmon
Nov 9 01:01:55 TNAS1 kernel: [ 4132] 0 4132 4136 465 224 241 0 69632 0 0 systemd-machine
Nov 9 01:01:55 TNAS1 kernel: [ 5386] 105 5386 1781 280 96 184 0 49152 0 0 avahi-daemon
Nov 9 01:01:55 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:55 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:55 TNAS1 kernel: [ 6672] 0 6672 13921 5695 5472 223 0 139264 0 -1000 websockify
Nov 9 01:01:55 TNAS1 kernel: [ 6757] 999 6757 116708 29347 29218 65 64 475136 0 -900 netdata
Nov 9 01:01:55 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:55 TNAS1 kernel: [ 7021] 999 7021 22248 5128 4846 282 0 147456 0 -900 python.d.plugin
Nov 9 01:01:55 TNAS1 kernel: [2479756] 999 2479756 1113 265 160 105 0 49152 0 -900 bash
Nov 9 01:01:55 TNAS1 kernel: [2709452] 0 2709452 94214 61 32 29 0 86016 0 0 nscd
Nov 9 01:01:55 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:55 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:55 TNAS1 kernel: [2776151] 0 2776151 207108 13269 12886 383 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:55 TNAS1 kernel: [2777444] 0 2777444 206501 12598 12278 320 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:55 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:55 TNAS1 kernel: [2782338] 0 2782338 351207 954 672 282 0 290816 0 0 libvirtd
Nov 9 01:01:55 TNAS1 kernel: [2782349] 0 2782349 6660 3292 2720 572 0 90112 0 0 truenas_audit_h
Nov 9 01:01:55 TNAS1 kernel: [2782363] 0 2782363 3949 507 192 315 0 69632 0 0 syslog-ng
Nov 9 01:01:55 TNAS1 kernel: [2782372] 0 2782372 151 0 0 0 0 20480 0 0 dnsmasq
Nov 9 01:01:55 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=libvirtd.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tnaudit.service,task=truenas_audit_h,pid=2782349,uid=0
Nov 9 01:01:55 TNAS1 kernel: Out of memory: Killed process 2782349 (truenas_audit_h) total-vm:26640kB, anon-rss:10880kB, file-rss:2288kB, shmem-rss:0kB, UID:0 pgtables:88kB oom_score_adj:0
Nov 9 01:01:55 TNAS1 kernel: dnsmasq invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:01:55 TNAS1 kernel: CPU: 7 UID: 0 PID: 2782372 Comm: dnsmasq Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:55 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:55 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:55 TNAS1 kernel: Call Trace:
Nov 9 01:01:55 TNAS1 kernel:
Nov 9 01:01:55 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:55 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:55 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:55 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:55 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:55 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:55 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:55 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:01:55 TNAS1 kernel: __do_fault+0xa3/0x170
Nov 9 01:01:55 TNAS1 kernel: do_fault+0x3de/0x4d0
Nov 9 01:01:55 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:01:55 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:01:55 TNAS1 kernel: do_user_addr_fault+0x15d/0x670
Nov 9 01:01:55 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:01:55 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:01:55 TNAS1 kernel: RIP: 0010:rep_stos_alternative+0x40/0x80
Nov 9 01:01:55 TNAS1 kernel: Code: ff c7 48 ff c9 75 f6 c3 cc cc cc cc 48 89 07 48 83 c7 08 83 e9 08 74 ef 83 f9 08 73 ef eb de 66 66 2e 0f 1f 84 00 00 00 00 00 <48> 89 07 48 89 47 08 48 89 47 10 48 89 47 18 48 89 47 20 48 89 47
Nov 9 01:01:55 TNAS1 kernel: RSP: 0018:ffffac8092e8fd18 EFLAGS: 00010202
Nov 9 01:01:55 TNAS1 kernel: RAX: 0000000000000000 RBX: 000055fd60df5bf0 RCX: 0000000000000410
Nov 9 01:01:55 TNAS1 kernel: RDX: 000055fd60df61d8 RSI: 0000000000000000 RDI: 000055fd60df5bf0
Nov 9 01:01:55 TNAS1 kernel: RBP: ffff8ac4489ba918 R08: 000055fd60df6000 R09: 000000055fd60d80
Nov 9 01:01:55 TNAS1 kernel: R10: 000055fd60df1000 R11: 0000000000000001 R12: 0000000000000003
Nov 9 01:01:55 TNAS1 kernel: R13: 000055fd60df1000 R14: ffff8ac14be40000 R15: 0000000000000000
Nov 9 01:01:55 TNAS1 kernel: elf_load+0x236/0x260
Nov 9 01:01:55 TNAS1 kernel: load_elf_binary+0x606/0x1720
Nov 9 01:01:55 TNAS1 kernel: ? load_misc_binary+0x20d/0x350 [binfmt_misc]
Nov 9 01:01:55 TNAS1 kernel: bprm_execve+0x23c/0x610
Nov 9 01:01:55 TNAS1 kernel: do_execveat_common.isra.0+0x17f/0x1e0
Nov 9 01:01:55 TNAS1 kernel: __x64_sys_execve+0x36/0x40
Nov 9 01:01:55 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:01:55 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:01:55 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:01:55 TNAS1 kernel: RIP: 0033:0x7f447d951a97
Nov 9 01:01:55 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7f447d951a6d.
Nov 9 01:01:55 TNAS1 kernel: RSP: 002b:00007f4451ffa7f8 EFLAGS: 00000206 ORIG_RAX: 000000000000003b
Nov 9 01:01:55 TNAS1 kernel: RAX: ffffffffffffffda RBX: 00007f442c0206b0 RCX: 00007f447d951a97
Nov 9 01:01:55 TNAS1 kernel: RDX: 00007f4424008f90 RSI: 00007f442c01bf10 RDI: 00007f442c01bd80
Nov 9 01:01:55 TNAS1 kernel: RBP: 00007f4451ffa8c0 R08: 00007f4451ffb5e0 R09: 0000000000000000
Nov 9 01:01:55 TNAS1 kernel: R10: 0000000000000000 R11: 0000000000000206 R12: 00000000ffffffff
Nov 9 01:01:55 TNAS1 kernel: R13: 00007f4451ffa860 R14: 00007f44240091d0 R15: 00000000002a74a4
Nov 9 01:01:55 TNAS1 kernel:
Nov 9 01:01:55 TNAS1 kernel: Mem-Info:
Nov 9 01:01:55 TNAS1 kernel: active_anon:65307 inactive_anon:7668394 isolated_anon:0
active_file:917 inactive_file:291 isolated_file:0
unevictable:1000 dirty:302 writeback:426
slab_reclaimable:57970 slab_unreclaimable:81035
mapped:1519 shmem:92805 pagetables:16371
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:34528 free_pcp:141 free_cma:1376
Nov 9 01:01:55 TNAS1 kernel: Node 0 active_anon:261228kB inactive_anon:30673576kB active_file:3668kB inactive_file:1164kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:6076kB dirty:1208kB writeback:1704kB shmem:371220kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11920kB pagetables:65484kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:55 TNAS1 kernel: Node 0 DMA free:32kB boost:2048kB min:2076kB low:2088kB high:2100kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11232kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:55 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:55 TNAS1 kernel: Node 0 DMA32 free:7172kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3478004kB active_file:1056kB inactive_file:900kB unevictable:0kB writepending:724kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:356kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:55 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:55 TNAS1 kernel: Node 0 Normal free:130908kB boost:77824kB min:138216kB low:167476kB high:196736kB reserved_highatomic:4096KB active_anon:261224kB inactive_anon:27184284kB active_file:2740kB inactive_file:672kB unevictable:4000kB writepending:2188kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:268kB local_pcp:0kB free_cma:5504kB
Nov 9 01:01:55 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:55 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:55 TNAS1 kernel: Node 0 DMA32: 24*4kB (UE) 32*8kB (UE) 51*16kB (UE) 33*32kB (UME) 34*64kB (UME) 3*128kB (ME) 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6832kB
Nov 9 01:01:55 TNAS1 kernel: Node 0 Normal: 4838*4kB (UMEC) 961*8kB (UMEC) 1806*16kB (UMEC) 2172*32kB (UMEC) 19*64kB (MEC) 7*128kB (MC) 1*256kB (C) 1*512kB (M) 1*1024kB (M) 1*2048kB (M) 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 131392kB
Nov 9 01:01:55 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:55 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:55 TNAS1 kernel: 94179 total pagecache pages
Nov 9 01:01:55 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:55 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:55 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:55 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:55 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:55 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:55 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:55 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:55 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:55 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:55 TNAS1 kernel: [ 728] 104 728 2091 461 256 205 0 65536 0 -900 dbus-daemon
Nov 9 01:01:55 TNAS1 kernel: [ 741] 0 741 11506 872 224 648 0 106496 0 -250 systemd-journal
Nov 9 01:01:55 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:55 TNAS1 kernel: [ 1630] 0 1630 8359520 7484625 7484330 231 64 60751872 0 -1000 asyncio_loop
Nov 9 01:01:55 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:55 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:55 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:55 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:55 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:55 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:55 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:55 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:55 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:55 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:55 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:55 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:55 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:55 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:55 TNAS1 kernel: [ 3635] 131 3635 4715 224 138 86 0 61440 0 0 chronyd
Nov 9 01:01:55 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:55 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:55 TNAS1 kernel: [ 3691] 0 3691 12416 422 224 198 0 94208 0 0 systemd-logind
Nov 9 01:01:55 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:55 TNAS1 kernel: [ 3708] 0 3708 523983 4212 4140 72 0 327680 0 -999 containerd
Nov 9 01:01:55 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:01:55 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:55 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:55 TNAS1 kernel: [ 3962] 122 3962 1695 208 96 112 0 57344 0 0 upsd
Nov 9 01:01:55 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:55 TNAS1 kernel: [ 4009] 122 4009 2127 192 96 96 0 53248 0 0 upsmon
Nov 9 01:01:55 TNAS1 kernel: [ 4132] 0 4132 4136 401 224 177 0 69632 0 0 systemd-machine
Nov 9 01:01:55 TNAS1 kernel: [ 5386] 105 5386 1781 280 96 184 0 49152 0 0 avahi-daemon
Nov 9 01:01:55 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:55 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:55 TNAS1 kernel: [ 6672] 0 6672 13921 5695 5472 223 0 139264 0 -1000 websockify
Nov 9 01:01:55 TNAS1 kernel: [ 6757] 999 6757 116708 29379 29218 97 64 475136 0 -900 netdata
Nov 9 01:01:55 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:55 TNAS1 kernel: [ 7021] 999 7021 22248 5096 4846 250 0 147456 0 -900 python.d.plugin
Nov 9 01:01:55 TNAS1 kernel: [2479756] 999 2479756 1113 265 160 105 0 49152 0 -900 bash
Nov 9 01:01:55 TNAS1 kernel: [2709452] 0 2709452 94214 61 32 29 0 86016 0 0 nscd
Nov 9 01:01:55 TNAS1 kernel: [2773557] 0 2773557 206757 12350 12274 76 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:55 TNAS1 kernel: [2773561] 0 2773561 207385 13111 12971 140 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:55 TNAS1 kernel: [2776151] 0 2776151 207108 13173 12886 287 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:55 TNAS1 kernel: [2777444] 0 2777444 206501 12534 12278 256 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:55 TNAS1 kernel: [2777446] 0 2777446 207155 12705 12561 144 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:55 TNAS1 kernel: [2782338] 0 2782338 351207 922 672 250 0 290816 0 0 libvirtd
Nov 9 01:01:55 TNAS1 kernel: [2782363] 0 2782363 3988 667 224 443 0 69632 0 0 syslog-ng
Nov 9 01:01:55 TNAS1 kernel: [2782372] 0 2782372 151 0 0 0 0 20480 0 0 dnsmasq
Nov 9 01:01:55 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=libvirtd.service,mems_allowed=0,global_oom,task_memcg=/system.slice/libvirtd.service,task=libvirtd,pid=2782338,uid=0
Nov 9 01:01:55 TNAS1 kernel: Out of memory: Killed process 2782338 (libvirtd) total-vm:1404828kB, anon-rss:2688kB, file-rss:1000kB, shmem-rss:0kB, UID:0 pgtables:284kB oom_score_adj:0
Nov 9 01:01:55 TNAS1 systemd[1]: libvirtd.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:55 TNAS1 systemd[1]: libvirtd.service: Failed with result 'signal'.
Nov 9 01:01:55 TNAS1 systemd[1]: libvirtd.service: Unit process 2782372 (dnsmasq) remains running after unit stopped.
Nov 9 01:01:55 TNAS1 systemd[1]: tnaudit.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:55 TNAS1 systemd[1]: tnaudit.service: Failed with result 'signal'.
Nov 9 01:01:55 TNAS1 systemd[1]: Starting tnaudit.service - TrueNAS Auditd Handler...
Nov 9 01:01:55 TNAS1 systemd[1]: Started tnaudit.service - TrueNAS Auditd Handler.
Nov 9 01:01:55 TNAS1 systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 2.
Nov 9 01:01:55 TNAS1 systemd[1]: Stopped libvirtd.service - Virtualization daemon.
Nov 9 01:01:57 TNAS1 systemd[1]: Starting libvirtd.service - Virtualization daemon...
Nov 9 01:01:57 TNAS1 systemd[1]: Started syslog-ng.service - System Logger Daemon.
Nov 9 01:01:57 TNAS1 systemd[1]: getty@tty1.service: Scheduled restart job, restart counter is at 1.
Nov 9 01:01:57 TNAS1 systemd[1]: Stopped getty@tty1.service - Getty on tty1.
Nov 9 01:01:57 TNAS1 systemd[1]: getty@tty1.service: Consumed 2min 23.392s CPU time.
Nov 9 01:01:58 TNAS1 kernel: python.d.plugin invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-900
Nov 9 01:01:58 TNAS1 kernel: CPU: 4 UID: 999 PID: 7182 Comm: python.d.plugin Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:01:58 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:01:58 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:01:58 TNAS1 kernel: Call Trace:
Nov 9 01:01:58 TNAS1 kernel:
Nov 9 01:01:58 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:01:58 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:01:58 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:01:58 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:01:58 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:01:58 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:01:58 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:01:58 TNAS1 kernel: folio_alloc_noprof+0x5b/0xb0
Nov 9 01:01:58 TNAS1 kernel: __filemap_get_folio+0x1aa/0x2e0
Nov 9 01:01:58 TNAS1 kernel: filemap_fault+0x647/0xd40
Nov 9 01:01:58 TNAS1 kernel: __do_fault+0x33/0x170
Nov 9 01:01:58 TNAS1 kernel: do_fault+0x2a8/0x4d0
Nov 9 01:01:58 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:01:58 TNAS1 kernel: ? switch_fpu_return+0x4f/0xd0
Nov 9 01:01:58 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:01:58 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:01:58 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:01:58 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:01:58 TNAS1 kernel: RIP: 0033:0x7f018495ef16
Nov 9 01:01:58 TNAS1 kernel: Code: 74 24 08 e8 2c f9 ff ff 4c 8b 54 24 18 8b 74 24 08 89 ea 89 c3 48 8b 7c 24 10 41 b9 ff ff ff ff 45 31 c0 b8 ca 00 00 00 0f 05 <89> df 48 89 44 24 08 e8 7e f9 ff ff 48 8b 44 24 08 e9 6a ff ff ff
Nov 9 01:01:58 TNAS1 kernel: RSP: 002b:00007f0183349690 EFLAGS: 00010246
Nov 9 01:01:58 TNAS1 kernel: RAX: ffffffffffffff92 RBX: 0000000000000000 RCX: 00007f018495ef16
Nov 9 01:01:58 TNAS1 kernel: RDX: 0000000000000000 RSI: 0000000000000089 RDI: 0000000000a4db30
Nov 9 01:01:58 TNAS1 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
Nov 9 01:01:58 TNAS1 kernel: R10: 00007f01833497c0 R11: 0000000000000246 R12: 0000000000000000
Nov 9 01:01:58 TNAS1 kernel: R13: 0000000000a4db38 R14: 0000000000a4db30 R15: 0000000000000000
Nov 9 01:01:58 TNAS1 kernel:
Nov 9 01:01:58 TNAS1 kernel: Mem-Info:
Nov 9 01:01:58 TNAS1 kernel: active_anon:65307 inactive_anon:7676959 isolated_anon:0
active_file:525 inactive_file:415 isolated_file:59
unevictable:1000 dirty:0 writeback:52
slab_reclaimable:57961 slab_unreclaimable:81021
mapped:1017 shmem:92803 pagetables:16475
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27246 free_pcp:0 free_cma:56
Nov 9 01:01:58 TNAS1 kernel: Node 0 active_anon:261228kB inactive_anon:30707836kB active_file:2100kB inactive_file:1660kB unevictable:4000kB isolated(anon):0kB isolated(file):236kB mapped:4068kB dirty:0kB writeback:208kB shmem:371212kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11824kB pagetables:65900kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:01:58 TNAS1 kernel: Node 0 DMA free:24kB boost:2048kB min:2076kB low:2088kB high:2100kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11232kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:58 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:01:58 TNAS1 kernel: Node 0 DMA32 free:6956kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3479348kB active_file:0kB inactive_file:228kB unevictable:0kB writepending:60kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:01:58 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:01:58 TNAS1 kernel: Node 0 Normal free:102004kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261224kB inactive_anon:27216884kB active_file:2000kB inactive_file:1292kB unevictable:4000kB writepending:144kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:224kB
Nov 9 01:01:58 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:01:58 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:01:58 TNAS1 kernel: Node 0 DMA32: 141*4kB (UME) 62*8kB (UME) 61*16kB (UME) 37*32kB (UME) 33*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7508kB
Nov 9 01:01:58 TNAS1 kernel: Node 0 Normal: 1266*4kB (UMEC) 552*8kB (UMEC) 1631*16kB (UMEC) 2078*32kB (UEC) 5*64kB (C) 2*128kB (C) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 102648kB
Nov 9 01:01:58 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:01:58 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:01:58 TNAS1 kernel: 93736 total pagecache pages
Nov 9 01:01:58 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:01:58 TNAS1 kernel: Free swap = 0kB
Nov 9 01:01:58 TNAS1 kernel: Total swap = 0kB
Nov 9 01:01:58 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:01:58 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:01:58 TNAS1 kernel: 170227 pages reserved
Nov 9 01:01:58 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:01:58 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:01:58 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:01:58 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:01:58 TNAS1 kernel: [ 728] 104 728 2091 336 256 80 0 65536 0 -900 dbus-daemon
Nov 9 01:01:58 TNAS1 kernel: [ 741] 0 741 11506 569 224 345 0 106496 0 -250 systemd-journal
Nov 9 01:01:58 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:01:58 TNAS1 kernel: [ 1630] 0 1630 8366432 7491271 7491114 93 64 60805120 0 -1000 asyncio_loop
Nov 9 01:01:58 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:01:58 TNAS1 kernel: [ 1681] 0 1681 485024 35427 35252 175 0 651264 0 -1000 middlewared (ze
Nov 9 01:01:58 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:01:58 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:01:58 TNAS1 kernel: [ 3515] 0 3515 1302 238 97 141 0 49152 0 0 nfsdcld
Nov 9 01:01:58 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:01:58 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:01:58 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:01:58 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:01:58 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:01:58 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:01:58 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:01:58 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:01:58 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:01:58 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:01:58 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:01:58 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:01:58 TNAS1 kernel: [ 3691] 0 3691 12416 406 224 182 0 94208 0 0 systemd-logind
Nov 9 01:01:58 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:01:58 TNAS1 kernel: [ 3708] 0 3708 523983 4148 4140 8 0 327680 0 -999 containerd
Nov 9 01:01:58 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:01:58 TNAS1 kernel: [ 3809] 122 3809 3089 134 121 13 0 49152 0 0 usbhid-ups
Nov 9 01:01:58 TNAS1 kernel: [ 3891] 0 3891 1005 98 32 66 0 45056 0 0 cron
Nov 9 01:01:58 TNAS1 kernel: [ 3962] 122 3962 1695 144 96 48 0 57344 0 0 upsd
Nov 9 01:01:58 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:01:58 TNAS1 kernel: [ 4009] 122 4009 2127 192 96 96 0 53248 0 0 upsmon
Nov 9 01:01:58 TNAS1 kernel: [ 4132] 0 4132 4136 464 224 240 0 69632 0 0 systemd-machine
Nov 9 01:01:58 TNAS1 kernel: [ 5386] 105 5386 1781 248 96 152 0 49152 0 0 avahi-daemon
Nov 9 01:01:58 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:01:58 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:01:58 TNAS1 kernel: [ 6672] 0 6672 13921 5695 5472 223 0 139264 0 -1000 websockify
Nov 9 01:01:58 TNAS1 kernel: [ 6757] 999 6757 116708 29352 29218 70 64 475136 0 -900 netdata
Nov 9 01:01:58 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:01:58 TNAS1 kernel: [ 7021] 999 7021 22248 4927 4846 81 0 147456 0 -900 python.d.plugin
Nov 9 01:01:58 TNAS1 kernel: [2479756] 999 2479756 1113 350 160 190 0 49152 0 -900 bash
Nov 9 01:01:58 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:01:58 TNAS1 kernel: [2773557] 0 2773557 206757 12351 12274 77 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:58 TNAS1 kernel: [2773561] 0 2773561 207385 13059 12971 88 0 315392 0 -1000 middlewared (wo
Nov 9 01:01:58 TNAS1 kernel: [2776151] 0 2776151 207108 12981 12886 95 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:58 TNAS1 kernel: [2777444] 0 2777444 206501 12406 12278 128 0 299008 0 -1000 middlewared (wo
Nov 9 01:01:58 TNAS1 kernel: [2777446] 0 2777446 207155 12634 12561 73 0 303104 0 -1000 middlewared (wo
Nov 9 01:01:58 TNAS1 kernel: [2782363] 0 2782363 187783 671 448 223 0 405504 0 0 syslog-ng
Nov 9 01:01:58 TNAS1 kernel: [2782373] 0 2782373 4497 1820 1728 92 0 73728 0 0 truenas_audit_h
Nov 9 01:01:58 TNAS1 kernel: [2782383] 0 2782383 42617 1630 1470 160 0 94208 0 0 (libvirtd)
Nov 9 01:01:58 TNAS1 kernel: [2782387] 999 2782387 1113 384 204 180 0 49152 0 -900 bash
Nov 9 01:01:58 TNAS1 kernel: [2782388] 999 2782388 596 32 0 32 0 32768 0 -900 timeout
Nov 9 01:01:58 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tnaudit.service,task=truenas_audit_h,pid=2782373,uid=0
Nov 9 01:01:58 TNAS1 kernel: Out of memory: Killed process 2782373 (truenas_audit_h) total-vm:17988kB, anon-rss:6912kB, file-rss:368kB, shmem-rss:0kB, UID:0 pgtables:72kB oom_score_adj:0
Nov 9 01:01:59 TNAS1 systemd[1]: Started getty@tty1.service - Getty on tty1.
Nov 9 01:01:59 TNAS1 systemd[1]: tnaudit.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:01:59 TNAS1 systemd[1]: tnaudit.service: Failed with result 'signal'.
Nov 9 01:02:01 TNAS1 systemd[1]: Starting tnaudit.service - TrueNAS Auditd Handler...
Nov 9 01:02:03 TNAS1 systemd[1]: Started tnaudit.service - TrueNAS Auditd Handler.
Nov 9 01:02:03 TNAS1 kernel: truenas_audit_h invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:02:03 TNAS1 kernel: CPU: 5 UID: 0 PID: 2782391 Comm: truenas_audit_h Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:03 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:03 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:03 TNAS1 kernel: Call Trace:
Nov 9 01:02:03 TNAS1 kernel:
Nov 9 01:02:03 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:03 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:03 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:03 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:03 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:03 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:03 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:03 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:02:03 TNAS1 kernel: do_fault+0x240/0x4d0
Nov 9 01:02:03 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:03 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:02:03 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:02:03 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:02:03 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:02:03 TNAS1 kernel: RIP: 0033:0x7fb411a2c588
Nov 9 01:02:03 TNAS1 kernel: Code: 58 fd ff ff 00 45 31 db 48 8d 15 d3 a9 00 00 4c 8d 05 0c 6d 01 00 49 89 c2 48 8d 78 ff 4c 89 e0 49 f7 da 0f 1f 80 00 00 00 00 <8b> 08 83 f9 07 77 19 85 c9 74 45 83 f9 07 77 40 48 63 0c 8a 48 01
Nov 9 01:02:03 TNAS1 kernel: RSP: 002b:00007ffcb06c5810 EFLAGS: 00010287
Nov 9 01:02:03 TNAS1 kernel: RAX: 0000000000400040 RBX: 0000000000400318 RCX: 000000000000000d
Nov 9 01:02:03 TNAS1 kernel: RDX: 00007fb411a36f40 RSI: 00007fb411a432e0 RDI: 0000000000000fff
Nov 9 01:02:03 TNAS1 kernel: RBP: 00007ffcb06c5ac0 R08: 00007fb411a43280 R09: 0000000000000000
Nov 9 01:02:03 TNAS1 kernel: R10: fffffffffffff000 R11: 0000000000000000 R12: 0000000000400040
Nov 9 01:02:03 TNAS1 kernel: R13: 00007fb411a36efc R14: 00007fb411a10000 R15: 00007fb411a38c99
Nov 9 01:02:03 TNAS1 kernel:
Nov 9 01:02:05 TNAS1 kernel: Mem-Info:
Nov 9 01:02:05 TNAS1 kernel: active_anon:65308 inactive_anon:7676413 isolated_anon:0
active_file:1248 inactive_file:69 isolated_file:0
unevictable:1000 dirty:34 writeback:106
slab_reclaimable:57957 slab_unreclaimable:81017
mapped:1426 shmem:92804 pagetables:16500
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27383 free_pcp:0 free_cma:211
Nov 9 01:02:05 TNAS1 kernel: Node 0 active_anon:261232kB inactive_anon:30705652kB active_file:4992kB inactive_file:276kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:5704kB dirty:136kB writeback:424kB shmem:371216kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11808kB pagetables:66000kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:05 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:05 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:05 TNAS1 kernel: Node 0 DMA32 free:7124kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480036kB active_file:180kB inactive_file:96kB unevictable:0kB writepending:28kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:05 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:05 TNAS1 kernel: Node 0 Normal free:102380kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261228kB inactive_anon:27215044kB active_file:4200kB inactive_file:396kB unevictable:4000kB writepending:532kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:844kB
Nov 9 01:02:05 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:05 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:05 TNAS1 kernel: Node 0 DMA32: 197*4kB (UME) 56*8kB (UME) 63*16kB (UME) 39*32kB (UME) 35*64kB (UME) 3*128kB (ME) 3*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7396kB
Nov 9 01:02:05 TNAS1 kernel: Node 0 Normal: 1237*4kB (UMEC) 534*8kB (UMEC) 1643*16kB (UEC) 2078*32kB (UEC) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 102004kB
Nov 9 01:02:05 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:05 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:05 TNAS1 kernel: 94226 total pagecache pages
Nov 9 01:02:05 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:05 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:05 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:05 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:05 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:05 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:05 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:05 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:05 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:05 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:05 TNAS1 kernel: [ 728] 104 728 2091 336 256 80 0 65536 0 -900 dbus-daemon
Nov 9 01:02:05 TNAS1 kernel: [ 741] 0 741 11506 585 224 361 0 106496 0 -250 systemd-journal
Nov 9 01:02:05 TNAS1 kernel: [ 763] 0 763 6768 609 512 97 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:05 TNAS1 kernel: [ 1630] 0 1630 8367456 7492527 7492170 293 64 60813312 0 -1000 asyncio_loop
Nov 9 01:02:05 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:05 TNAS1 kernel: [ 1681] 0 1681 485024 35318 35252 66 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:05 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:05 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:05 TNAS1 kernel: [ 3515] 0 3515 1302 410 97 313 0 49152 0 0 nfsdcld
Nov 9 01:02:05 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:05 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:05 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:05 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:05 TNAS1 kernel: [ 3597] 0 3597 1258 160 131 29 0 57344 0 0 rpc.mountd
Nov 9 01:02:05 TNAS1 kernel: [ 3598] 0 3598 1258 187 99 88 0 57344 0 0 rpc.mountd
Nov 9 01:02:05 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:05 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:05 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:05 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:05 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:05 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:02:05 TNAS1 kernel: [ 3691] 0 3691 12416 322 224 98 0 94208 0 0 systemd-logind
Nov 9 01:02:05 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:05 TNAS1 kernel: [ 3708] 0 3708 523983 4212 4140 72 0 327680 0 -999 containerd
Nov 9 01:02:05 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:05 TNAS1 kernel: [ 3809] 122 3809 3089 184 121 63 0 49152 0 0 usbhid-ups
Nov 9 01:02:05 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:05 TNAS1 kernel: [ 3962] 122 3962 1695 188 96 92 0 57344 0 0 upsd
Nov 9 01:02:05 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:05 TNAS1 kernel: [ 4009] 122 4009 2127 160 96 64 0 53248 0 0 upsmon
Nov 9 01:02:05 TNAS1 kernel: [ 4132] 0 4132 4136 335 224 111 0 69632 0 0 systemd-machine
Nov 9 01:02:05 TNAS1 kernel: [ 5386] 105 5386 1781 248 96 152 0 49152 0 0 avahi-daemon
Nov 9 01:02:05 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:05 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:05 TNAS1 kernel: [ 6672] 0 6672 13921 5648 5472 176 0 139264 0 -1000 websockify
Nov 9 01:02:05 TNAS1 kernel: [ 6757] 999 6757 116708 29392 29218 110 64 475136 0 -900 netdata
Nov 9 01:02:05 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:05 TNAS1 kernel: [ 7021] 999 7021 22248 5267 4846 421 0 147456 0 -900 python.d.plugin
Nov 9 01:02:05 TNAS1 kernel: [2479756] 999 2479756 1113 385 160 225 0 49152 0 -900 bash
Nov 9 01:02:05 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:02:05 TNAS1 kernel: [2773557] 0 2773557 206757 12351 12274 77 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:05 TNAS1 kernel: [2773561] 0 2773561 207385 13059 12971 88 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:05 TNAS1 kernel: [2776151] 0 2776151 207108 12981 12886 95 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:05 TNAS1 kernel: [2777444] 0 2777444 206501 12406 12278 128 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:05 TNAS1 kernel: [2777446] 0 2777446 207155 12634 12561 73 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:05 TNAS1 kernel: [2782363] 0 2782363 187783 823 512 311 0 405504 0 0 syslog-ng
Nov 9 01:02:05 TNAS1 kernel: [2782383] 0 2782383 16875 650 320 330 0 102400 0 0 libvirtd
Nov 9 01:02:05 TNAS1 kernel: [2782390] 0 2782390 42617 1703 1470 233 0 94208 0 0 (cli)
Nov 9 01:02:05 TNAS1 kernel: [2782391] 0 2782391 1815 40 0 40 0 32768 0 0 truenas_audit_h
Nov 9 01:02:05 TNAS1 kernel: [2782404] 999 2782404 1113 204 204 0 0 49152 0 -900 bash
Nov 9 01:02:05 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=tnaudit.service,mems_allowed=0,global_oom,task_memcg=/system.slice/system-getty.slice/getty@tty1.service,task=(cli),pid=2782390,uid=0
Nov 9 01:02:05 TNAS1 kernel: Out of memory: Killed process 2782390 ((cli)) total-vm:170468kB, anon-rss:5880kB, file-rss:932kB, shmem-rss:0kB, UID:0 pgtables:92kB oom_score_adj:0
Nov 9 01:02:05 TNAS1 systemd[1]: getty@tty1.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:07 TNAS1 systemd[1]: getty@tty1.service: Failed with result 'signal'.
Nov 9 01:02:07 TNAS1 systemd[1]: system-getty.slice: A process of this unit has been killed by the OOM killer.
Nov 9 01:02:10 TNAS1 systemd[1]: getty@tty1.service: Scheduled restart job, restart counter is at 2.
Nov 9 01:02:13 TNAS1 systemd[1]: Stopped getty@tty1.service - Getty on tty1.
Nov 9 01:02:15 TNAS1 kernel: upsc invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=-900
Nov 9 01:02:15 TNAS1 kernel: CPU: 6 UID: 999 PID: 2782419 Comm: upsc Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:15 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:15 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:15 TNAS1 kernel: Call Trace:
Nov 9 01:02:15 TNAS1 kernel:
Nov 9 01:02:15 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:15 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:15 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:15 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:15 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:15 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:15 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:15 TNAS1 kernel: __pmd_alloc+0x34/0x210
Nov 9 01:02:15 TNAS1 kernel: move_page_tables+0xc70/0x1180
Nov 9 01:02:15 TNAS1 kernel: relocate_vma_down+0x11c/0x210
Nov 9 01:02:15 TNAS1 kernel: setup_arg_pages+0x1fb/0x3a0
Nov 9 01:02:15 TNAS1 kernel: load_elf_binary+0x37f/0x1720
Nov 9 01:02:15 TNAS1 kernel: ? load_misc_binary+0x20d/0x350 [binfmt_misc]
Nov 9 01:02:15 TNAS1 kernel: bprm_execve+0x23c/0x610
Nov 9 01:02:15 TNAS1 kernel: do_execveat_common.isra.0+0x17f/0x1e0
Nov 9 01:02:15 TNAS1 kernel: __x64_sys_execve+0x36/0x40
Nov 9 01:02:15 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:02:15 TNAS1 kernel: ? syscall_exit_to_user_mode+0x10/0x1f0
Nov 9 01:02:15 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:02:15 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:02:15 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:02:15 TNAS1 kernel: RIP: 0033:0x7f0481e1da97
Nov 9 01:02:15 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7f0481e1da6d.
Nov 9 01:02:15 TNAS1 kernel: RSP: 002b:00007ffe14f77fc8 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
Nov 9 01:02:15 TNAS1 kernel: RAX: ffffffffffffffda RBX: 00007ffe14f7a510 RCX: 00007f0481e1da97
Nov 9 01:02:15 TNAS1 kernel: RDX: 00007ffe14f783b0 RSI: 00007ffe14f78398 RDI: 00007ffe14f77fd0
Nov 9 01:02:15 TNAS1 kernel: RBP: 00007ffe14f780f0 R08: 00007ffe14f7aeef R09: 00007f0481d46740
Nov 9 01:02:15 TNAS1 kernel: R10: 00007f0481d5ae70 R11: 0000000000000202 R12: 00007ffe14f78398
Nov 9 01:02:15 TNAS1 kernel: R13: 00007ffe14f783b0 R14: 00007ffe14f7aee7 R15: 0000000000000001
Nov 9 01:02:15 TNAS1 kernel:
Nov 9 01:02:15 TNAS1 kernel: Mem-Info:
Nov 9 01:02:15 TNAS1 kernel: active_anon:65308 inactive_anon:7677394 isolated_anon:0
active_file:54 inactive_file:35 isolated_file:0
unevictable:1000 dirty:33 writeback:40
slab_reclaimable:57920 slab_unreclaimable:80931
mapped:266 shmem:92804 pagetables:16494
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:28047 free_pcp:0 free_cma:189
Nov 9 01:02:15 TNAS1 kernel: Node 0 active_anon:261232kB inactive_anon:30709576kB active_file:216kB inactive_file:140kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:1064kB dirty:132kB writeback:160kB shmem:371216kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11840kB pagetables:65976kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:15 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:15 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:15 TNAS1 kernel: Node 0 DMA32 free:7068kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480616kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:15 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:15 TNAS1 kernel: Node 0 Normal free:105092kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261228kB inactive_anon:27217724kB active_file:428kB inactive_file:0kB unevictable:4000kB writepending:292kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:756kB
Nov 9 01:02:15 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:15 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:15 TNAS1 kernel: Node 0 DMA32: 175*4kB (UME) 75*8kB (UME) 54*16kB (UME) 40*32kB (UME) 35*64kB (UME) 2*128kB (E) 3*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7220kB
Nov 9 01:02:15 TNAS1 kernel: Node 0 Normal: 2165*4kB (UMEC) 558*8kB (UMEC) 1620*16kB (UMEC) 2082*32kB (UMEC) 2*64kB (C) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 105796kB
Nov 9 01:02:15 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:15 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:15 TNAS1 kernel: 92893 total pagecache pages
Nov 9 01:02:15 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:15 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:15 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:15 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:15 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:15 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:15 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:15 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:15 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:15 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:15 TNAS1 kernel: [ 728] 104 728 2091 314 256 58 0 65536 0 -900 dbus-daemon
Nov 9 01:02:15 TNAS1 kernel: [ 741] 0 741 11506 366 224 142 0 106496 0 -250 systemd-journal
Nov 9 01:02:15 TNAS1 kernel: [ 763] 0 763 6768 623 512 111 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:15 TNAS1 kernel: [ 1630] 0 1630 8367712 7492743 7492618 61 64 60817408 0 -1000 asyncio_loop
Nov 9 01:02:15 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:15 TNAS1 kernel: [ 1681] 0 1681 485024 35349 35252 97 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:15 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:15 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:15 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:15 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:15 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:15 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:15 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:15 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:15 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:15 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:15 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:15 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:15 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:15 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:15 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:02:15 TNAS1 kernel: [ 3691] 0 3691 12416 319 224 95 0 94208 0 0 systemd-logind
Nov 9 01:02:15 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:15 TNAS1 kernel: [ 3708] 0 3708 523983 4180 4140 40 0 327680 0 -999 containerd
Nov 9 01:02:15 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:15 TNAS1 kernel: [ 3809] 122 3809 3089 240 121 119 0 49152 0 0 usbhid-ups
Nov 9 01:02:15 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:15 TNAS1 kernel: [ 3962] 122 3962 1695 159 96 63 0 57344 0 0 upsd
Nov 9 01:02:15 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:15 TNAS1 kernel: [ 4009] 122 4009 2127 173 96 77 0 53248 0 0 upsmon
Nov 9 01:02:15 TNAS1 kernel: [ 4132] 0 4132 4136 283 224 59 0 69632 0 0 systemd-machine
Nov 9 01:02:15 TNAS1 kernel: [ 5386] 105 5386 1781 248 96 152 0 49152 0 0 avahi-daemon
Nov 9 01:02:15 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:15 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:15 TNAS1 kernel: [ 6672] 0 6672 13921 5593 5472 121 0 139264 0 -1000 websockify
Nov 9 01:02:15 TNAS1 kernel: [ 6757] 999 6757 116708 29282 29218 0 64 475136 0 -900 netdata
Nov 9 01:02:15 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:15 TNAS1 kernel: [ 7021] 999 7021 22248 4876 4846 30 0 147456 0 -900 python.d.plugin
Nov 9 01:02:15 TNAS1 kernel: [2479756] 999 2479756 1113 269 160 109 0 49152 0 -900 bash
Nov 9 01:02:15 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:02:15 TNAS1 kernel: [2773557] 0 2773557 206757 12351 12274 77 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:15 TNAS1 kernel: [2773561] 0 2773561 207385 13098 12971 127 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:15 TNAS1 kernel: [2776151] 0 2776151 207108 13021 12886 135 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:15 TNAS1 kernel: [2777444] 0 2777444 206501 12379 12278 101 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:15 TNAS1 kernel: [2777446] 0 2777446 207155 12652 12561 91 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:15 TNAS1 kernel: [2782363] 0 2782363 243082 659 544 115 0 430080 0 0 syslog-ng
Nov 9 01:02:15 TNAS1 kernel: [2782383] 0 2782383 16875 320 320 0 0 102400 0 0 libvirtd
Nov 9 01:02:16 TNAS1 kernel: [2782391] 0 2782391 3115 565 448 117 0 61440 0 0 truenas_audit_h
Nov 9 01:02:18 TNAS1 kernel: [2782416] 999 2782416 1113 309 204 105 0 49152 0 -900 bash
Nov 9 01:02:18 TNAS1 kernel: [2782417] 999 2782417 623 8 0 8 0 49152 0 -900 timeout
Nov 9 01:02:18 TNAS1 kernel: [2782418] 0 2782418 42617 1474 1470 4 0 94208 0 0 (cli)
Nov 9 01:02:18 TNAS1 kernel: [2782419] 999 2782419 1 0 0 0 0 12288 0 -900 upsc
Nov 9 01:02:18 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/system.slice/system-getty.slice/getty@tty1.service,task=(cli),pid=2782418,uid=0
Nov 9 01:02:18 TNAS1 kernel: Out of memory: Killed process 2782418 ((cli)) total-vm:170468kB, anon-rss:5880kB, file-rss:16kB, shmem-rss:0kB, UID:0 pgtables:92kB oom_score_adj:0
Nov 9 01:02:18 TNAS1 kernel: upsc invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=-900
Nov 9 01:02:18 TNAS1 kernel: CPU: 5 UID: 999 PID: 2782419 Comm: upsc Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:18 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:18 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:18 TNAS1 kernel: Call Trace:
Nov 9 01:02:18 TNAS1 kernel:
Nov 9 01:02:18 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:18 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:18 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:18 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:18 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:18 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:18 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:18 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:02:18 TNAS1 kernel: do_fault+0x240/0x4d0
Nov 9 01:02:18 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:18 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:02:18 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:02:18 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:02:18 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:02:18 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:02:18 TNAS1 kernel: RIP: 0033:0x7fadd7e7d262
Nov 9 01:02:18 TNAS1 kernel: Code: 0f 1f 84 00 00 00 00 00 3d 53 e5 74 64 75 0e 48 89 da 44 89 e6 4c 89 ef e8 cb f2 ff ff 49 39 9d c0 02 00 00 0f 84 ae 01 00 00 <8b> 43 c8 48 83 eb 38 83 f8 04 75 d2 48 8b 53 10 49 8b 45 00 4c 8b
Nov 9 01:02:18 TNAS1 kernel: RSP: 002b:00007ffe9823c750 EFLAGS: 00010293
Nov 9 01:02:18 TNAS1 kernel: RAX: 000000000000003f RBX: 00007fadd7b1b238 RCX: 0000000000000009
Nov 9 01:02:18 TNAS1 kernel: RDX: 00007fadd7b1b040 RSI: 00007fadd7b2c158 RDI: 000000006fffffff
Nov 9 01:02:18 TNAS1 kernel: RBP: 00007ffe9823c8a0 R08: 0000000000000006 R09: 0000000070000025
Nov 9 01:02:18 TNAS1 kernel: R10: 0000000000000035 R11: 000000006ffffdff R12: 0000000000000003
Nov 9 01:02:18 TNAS1 kernel: R13: 00007fadd7e6ebd0 R14: 00007ffe9823c940 R15: 00007fadd7c72000
Nov 9 01:02:18 TNAS1 kernel:
Nov 9 01:02:47 TNAS1 syslog-ng[2782519]: syslog-ng starting up; version='3.38.1'
Nov 9 01:02:18 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:02:18 TNAS1 kernel: RIP: 0033:0x7fadd7e7d262
Nov 9 01:02:18 TNAS1 kernel: Code: 0f 1f 84 00 00 00 00 00 3d 53 e5 74 64 75 0e 48 89 da 44 89 e6 4c 89 ef e8 cb f2 ff ff 49 39 9d c0 02 00 00 0f 84 ae 01 00 00 <8b> 43 c8 48 83 eb 38 83 f8 04 75 d2 48 8b 53 10 49 8b 45 00 4c 8b
Nov 9 01:02:18 TNAS1 kernel: RSP: 002b:00007ffe9823c750 EFLAGS: 00010293
Nov 9 01:02:18 TNAS1 kernel: RAX: 000000000000003f RBX: 00007fadd7b1b238 RCX: 0000000000000009
Nov 9 01:02:18 TNAS1 kernel: RDX: 00007fadd7b1b040 RSI: 00007fadd7b2c158 RDI: 000000006fffffff
Nov 9 01:02:18 TNAS1 kernel: RBP: 00007ffe9823c8a0 R08: 0000000000000006 R09: 0000000070000025
Nov 9 01:02:18 TNAS1 kernel: R10: 0000000000000035 R11: 000000006ffffdff R12: 0000000000000003
Nov 9 01:02:18 TNAS1 kernel: R13: 00007fadd7e6ebd0 R14: 00007ffe9823c940 R15: 00007fadd7c72000
Nov 9 01:02:18 TNAS1 kernel:
Nov 9 01:02:18 TNAS1 kernel: Mem-Info:
Nov 9 01:02:18 TNAS1 kernel: active_anon:65308 inactive_anon:7677394 isolated_anon:0
active_file:318 inactive_file:440 isolated_file:0
unevictable:1000 dirty:11 writeback:38
slab_reclaimable:57920 slab_unreclaimable:80931
mapped:985 shmem:92804 pagetables:16494
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27604 free_pcp:86 free_cma:271
Nov 9 01:02:18 TNAS1 kernel: Node 0 active_anon:261232kB inactive_anon:30709576kB active_file:1272kB inactive_file:1760kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:3940kB dirty:44kB writeback:152kB shmem:371216kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11840kB pagetables:65976kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:18 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:18 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:18 TNAS1 kernel: Node 0 DMA32 free:6960kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480616kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:344kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:18 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:18 TNAS1 kernel: Node 0 Normal free:103428kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261228kB inactive_anon:27217724kB active_file:624kB inactive_file:1232kB unevictable:4000kB writepending:196kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:1084kB
Nov 9 01:02:18 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:18 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:18 TNAS1 kernel: Node 0 DMA32: 24*4kB (UE) 48*8kB (UME) 54*16kB (UME) 40*32kB (UME) 35*64kB (UME) 2*128kB (E) 3*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6400kB
Nov 9 01:02:18 TNAS1 kernel: Node 0 Normal: 1400*4kB (UMEC) 533*8kB (UMEC) 1614*16kB (UMEC) 2080*32kB (UMEC) 2*64kB (C) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 102376kB
Nov 9 01:02:18 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:18 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:18 TNAS1 kernel: 93620 total pagecache pages
Nov 9 01:02:18 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:18 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:18 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:18 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:18 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:18 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:18 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:18 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:18 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:18 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:18 TNAS1 kernel: [ 728] 104 728 2091 386 256 130 0 65536 0 -900 dbus-daemon
Nov 9 01:02:18 TNAS1 kernel: [ 741] 0 741 11506 573 224 349 0 106496 0 -250 systemd-journal
Nov 9 01:02:18 TNAS1 kernel: [ 763] 0 763 6768 623 512 111 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:18 TNAS1 kernel: [ 1630] 0 1630 8367712 7493002 7492618 320 64 60817408 0 -1000 asyncio_loop
Nov 9 01:02:18 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:18 TNAS1 kernel: [ 1681] 0 1681 485024 35349 35252 97 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:18 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:18 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:18 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:18 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:18 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:18 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:18 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:18 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:18 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:18 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:18 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:18 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:18 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:18 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:18 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:02:18 TNAS1 kernel: [ 3691] 0 3691 12416 319 224 95 0 94208 0 0 systemd-logind
Nov 9 01:02:18 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:18 TNAS1 kernel: [ 3708] 0 3708 523983 4148 4140 8 0 327680 0 -999 containerd
Nov 9 01:02:18 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:18 TNAS1 kernel: [ 3809] 122 3809 3089 240 121 119 0 49152 0 0 usbhid-ups
Nov 9 01:02:18 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:18 TNAS1 kernel: [ 3962] 122 3962 1695 159 96 63 0 57344 0 0 upsd
Nov 9 01:02:18 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:18 TNAS1 kernel: [ 4009] 122 4009 2127 173 96 77 0 53248 0 0 upsmon
Nov 9 01:02:18 TNAS1 kernel: [ 4132] 0 4132 4136 283 224 59 0 69632 0 0 systemd-machine
Nov 9 01:02:18 TNAS1 kernel: [ 5386] 105 5386 1781 248 96 152 0 49152 0 0 avahi-daemon
Nov 9 01:02:18 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:18 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:18 TNAS1 kernel: [ 6672] 0 6672 13921 5735 5472 263 0 139264 0 -1000 websockify
Nov 9 01:02:18 TNAS1 kernel: [ 6757] 999 6757 116708 29373 29218 91 64 475136 0 -900 netdata
Nov 9 01:02:18 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:18 TNAS1 kernel: [ 7021] 999 7021 22248 5175 4846 329 0 147456 0 -900 python.d.plugin
Nov 9 01:02:18 TNAS1 kernel: [2479756] 999 2479756 1113 269 160 109 0 49152 0 -900 bash
Nov 9 01:02:18 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:02:18 TNAS1 kernel: [2773557] 0 2773557 206757 12351 12274 77 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:18 TNAS1 kernel: [2773561] 0 2773561 207385 13098 12971 127 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:18 TNAS1 kernel: [2776151] 0 2776151 207108 13021 12886 135 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:18 TNAS1 kernel: [2777444] 0 2777444 206501 12379 12278 101 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:18 TNAS1 kernel: [2777446] 0 2777446 207155 12652 12561 91 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:18 TNAS1 kernel: [2782363] 0 2782363 243082 832 544 288 0 430080 0 0 syslog-ng
Nov 9 01:02:18 TNAS1 kernel: [2782383] 0 2782383 16875 391 320 71 0 102400 0 0 libvirtd
Nov 9 01:02:18 TNAS1 kernel: [2782391] 0 2782391 3159 893 480 413 0 61440 0 0 truenas_audit_h
Nov 9 01:02:18 TNAS1 kernel: [2782416] 999 2782416 1113 309 204 105 0 49152 0 -900 bash
Nov 9 01:02:18 TNAS1 kernel: [2782417] 999 2782417 623 8 0 8 0 49152 0 -900 timeout
Nov 9 01:02:18 TNAS1 kernel: [2782419] 999 2782419 953 32 0 32 0 36864 0 -900 upsc
Nov 9 01:02:18 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tnaudit.service,task=truenas_audit_h,pid=2782391,uid=0
Nov 9 01:02:18 TNAS1 kernel: Out of memory: Killed process 2782391 (truenas_audit_h) total-vm:12636kB, anon-rss:1920kB, file-rss:1652kB, shmem-rss:0kB, UID:0 pgtables:60kB oom_score_adj:0
Nov 9 01:02:18 TNAS1 systemd[1]: Started getty@tty1.service - Getty on tty1.
Nov 9 01:02:18 TNAS1 systemd[1]: tnaudit.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:18 TNAS1 systemd[1]: tnaudit.service: Failed with result 'signal'.
Nov 9 01:02:18 TNAS1 systemd[1]: getty@tty1.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:18 TNAS1 systemd[1]: getty@tty1.service: Failed with result 'signal'.
Nov 9 01:02:19 TNAS1 systemd[1]: system-getty.slice: A process of this unit has been killed by the OOM killer.
Nov 9 01:02:21 TNAS1 systemd[1]: Starting tnaudit.service - TrueNAS Auditd Handler...
Nov 9 01:02:23 TNAS1 systemd[1]: Started tnaudit.service - TrueNAS Auditd Handler.
Nov 9 01:02:26 TNAS1 kernel: syslog-ng invoked oom-killer: gfp_mask=0x500cc2(GFP_HIGHUSER|__GFP_ACCOUNT), order=0, oom_score_adj=0
Nov 9 01:02:26 TNAS1 kernel: CPU: 4 UID: 0 PID: 2782415 Comm: syslog-ng Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:26 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:26 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:26 TNAS1 kernel: Call Trace:
Nov 9 01:02:26 TNAS1 kernel:
Nov 9 01:02:26 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:26 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:26 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:26 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:26 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:26 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:26 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:26 TNAS1 kernel: pipe_write+0x3a9/0x640
Nov 9 01:02:26 TNAS1 kernel: vfs_write+0x3ad/0x450
Nov 9 01:02:26 TNAS1 kernel: ksys_write+0xbb/0xf0
Nov 9 01:02:26 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:02:26 TNAS1 kernel: ? __audit_filter_op+0xaf/0x110
Nov 9 01:02:26 TNAS1 kernel: ? audit_reset_context+0x232/0x300
Nov 9 01:02:26 TNAS1 kernel: ? syscall_exit_to_user_mode_prepare+0x148/0x170
Nov 9 01:02:26 TNAS1 kernel: ? syscall_exit_to_user_mode+0x10/0x1f0
Nov 9 01:02:26 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:02:26 TNAS1 kernel: ? audit_reset_context+0x232/0x300
Nov 9 01:02:26 TNAS1 kernel: ? syscall_exit_to_user_mode_prepare+0x148/0x170
Nov 9 01:02:26 TNAS1 kernel: ? syscall_exit_to_user_mode+0x10/0x1f0
Nov 9 01:02:26 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:02:26 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:02:26 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:02:26 TNAS1 kernel: ? __irq_exit_rcu+0x38/0xb0
Nov 9 01:02:26 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:02:26 TNAS1 kernel: RIP: 0033:0x7f894583e2ff
Nov 9 01:02:26 TNAS1 kernel: Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 39 d5 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 8c d5 f8 ff 48
Nov 9 01:02:26 TNAS1 kernel: RSP: 002b:00007f891a7f9860 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
Nov 9 01:02:26 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000000000000044 RCX: 00007f894583e2ff
Nov 9 01:02:26 TNAS1 kernel: RDX: 0000000000000044 RSI: 0000562b11589dd0 RDI: 000000000000003f
Nov 9 01:02:26 TNAS1 kernel: RBP: 0000562b11589dd0 R08: 0000000000000000 R09: 0000000000000000
Nov 9 01:02:26 TNAS1 kernel: R10: 0000000000000002 R11: 0000000000000293 R12: 0000562b115896b0
Nov 9 01:02:26 TNAS1 kernel: R13: 0000000000000000 R14: 00007f8945b55444 R15: 0000000000000001
Nov 9 01:02:26 TNAS1 kernel:
Nov 9 01:02:26 TNAS1 kernel: Mem-Info:
Nov 9 01:02:26 TNAS1 kernel: active_anon:65307 inactive_anon:7677641 isolated_anon:0
active_file:396 inactive_file:630 isolated_file:0
unevictable:1000 dirty:35 writeback:147
slab_reclaimable:57924 slab_unreclaimable:80898
mapped:1306 shmem:92803 pagetables:16480
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27816 free_pcp:35 free_cma:0
Nov 9 01:02:26 TNAS1 kernel: Node 0 active_anon:261228kB inactive_anon:30710564kB active_file:1584kB inactive_file:2212kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:5224kB dirty:140kB writeback:588kB shmem:371212kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11904kB pagetables:65920kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:26 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:26 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:26 TNAS1 kernel: Node 0 DMA32 free:6944kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480620kB active_file:68kB inactive_file:56kB unevictable:0kB writepending:60kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:26 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:26 TNAS1 kernel: Node 0 Normal free:104292kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261224kB inactive_anon:27218708kB active_file:1648kB inactive_file:2532kB unevictable:4000kB writepending:668kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:26 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:26 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:26 TNAS1 kernel: Node 0 DMA32: 182*4kB (UME) 72*8kB (UME) 69*16kB (UME) 44*32kB (UME) 34*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7144kB
Nov 9 01:02:26 TNAS1 kernel: Node 0 Normal: 1788*4kB (UMEC) 547*8kB (UMEC) 1614*16kB (UMEC) 2078*32kB (UEC) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 103848kB
Nov 9 01:02:26 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:26 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:26 TNAS1 kernel: 93938 total pagecache pages
Nov 9 01:02:26 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:26 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:26 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:26 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:26 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:26 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:26 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:26 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:26 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:26 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:26 TNAS1 kernel: [ 728] 104 728 2091 310 256 54 0 65536 0 -900 dbus-daemon
Nov 9 01:02:26 TNAS1 kernel: [ 741] 0 741 11506 452 224 228 0 106496 0 -250 systemd-journal
Nov 9 01:02:26 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:26 TNAS1 kernel: [ 1630] 0 1630 8368224 7493216 7493066 86 64 60821504 0 -1000 asyncio_loop
Nov 9 01:02:26 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:26 TNAS1 kernel: [ 1681] 0 1681 485024 35365 35252 113 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:26 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:26 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:26 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:26 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:26 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:26 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:26 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:26 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:26 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:26 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:26 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:26 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:26 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:26 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:26 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:02:26 TNAS1 kernel: [ 3691] 0 3691 12416 386 224 162 0 94208 0 0 systemd-logind
Nov 9 01:02:26 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:26 TNAS1 kernel: [ 3708] 0 3708 523983 4212 4140 72 0 327680 0 -999 containerd
Nov 9 01:02:26 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:26 TNAS1 kernel: [ 3809] 122 3809 3089 181 121 60 0 49152 0 0 usbhid-ups
Nov 9 01:02:26 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:26 TNAS1 kernel: [ 3962] 122 3962 1695 153 96 57 0 57344 0 0 upsd
Nov 9 01:02:26 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:26 TNAS1 kernel: [ 4009] 122 4009 2127 205 96 109 0 53248 0 0 upsmon
Nov 9 01:02:26 TNAS1 kernel: [ 4132] 0 4132 4136 355 224 131 0 69632 0 0 systemd-machine
Nov 9 01:02:26 TNAS1 kernel: [ 5386] 105 5386 1781 248 96 152 0 49152 0 0 avahi-daemon
Nov 9 01:02:26 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:26 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:26 TNAS1 kernel: [ 6672] 0 6672 13921 5544 5472 72 0 139264 0 -1000 websockify
Nov 9 01:02:26 TNAS1 kernel: [ 6757] 999 6757 116708 29400 29218 118 64 475136 0 -900 netdata
Nov 9 01:02:26 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:26 TNAS1 kernel: [ 7021] 999 7021 22248 5049 4846 203 0 147456 0 -900 python.d.plugin
Nov 9 01:02:26 TNAS1 kernel: [2479756] 999 2479756 1113 260 160 100 0 49152 0 -900 bash
Nov 9 01:02:26 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:02:26 TNAS1 kernel: [2773557] 0 2773557 206757 12334 12274 60 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:26 TNAS1 kernel: [2773561] 0 2773561 207385 13075 12971 104 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:26 TNAS1 kernel: [2776151] 0 2776151 207108 12958 12886 72 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:26 TNAS1 kernel: [2777444] 0 2777444 206501 12381 12278 103 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:26 TNAS1 kernel: [2777446] 0 2777446 207155 12675 12561 114 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:26 TNAS1 kernel: [2782363] 0 2782363 243082 771 544 227 0 430080 0 0 syslog-ng
Nov 9 01:02:26 TNAS1 kernel: [2782383] 0 2782383 203061 903 512 391 0 212992 0 0 libvirtd
Nov 9 01:02:26 TNAS1 kernel: [2782440] 999 2782440 1511 128 64 64 0 45056 0 -900 awk
Nov 9 01:02:26 TNAS1 kernel: [2782441] 0 2782441 2630 306 0 306 0 57344 0 0 truenas_audit_h
Nov 9 01:02:26 TNAS1 kernel: [2782444] 0 2782444 42617 1576 1470 106 0 94208 0 0 (cli)
Nov 9 01:02:26 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=syslog-ng.service,mems_allowed=0,global_oom,task_memcg=/init.scope,task=(cli),pid=2782444,uid=0
Nov 9 01:02:26 TNAS1 kernel: Out of memory: Killed process 2782444 ((cli)) total-vm:170468kB, anon-rss:5880kB, file-rss:424kB, shmem-rss:0kB, UID:0 pgtables:92kB oom_score_adj:0
Nov 9 01:02:26 TNAS1 kernel: systemd invoked oom-killer: gfp_mask=0x102dc2(GFP_HIGHUSER|__GFP_NOWARN|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:02:26 TNAS1 kernel: CPU: 5 UID: 0 PID: 1 Comm: systemd Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:26 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:26 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:26 TNAS1 kernel: Call Trace:
Nov 9 01:02:26 TNAS1 kernel:
Nov 9 01:02:26 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:26 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:26 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:26 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:26 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:26 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:26 TNAS1 kernel: alloc_pages_bulk_noprof+0x3df/0x720
Nov 9 01:02:26 TNAS1 kernel: __vmalloc_node_range_noprof+0x24f/0x850
Nov 9 01:02:26 TNAS1 kernel: ? n_tty_open+0x18/0xa0
Nov 9 01:02:26 TNAS1 kernel: __vmalloc_node_noprof+0x4e/0x70
Nov 9 01:02:26 TNAS1 kernel: ? n_tty_open+0x18/0xa0
Nov 9 01:02:26 TNAS1 kernel: n_tty_open+0x18/0xa0
Nov 9 01:02:26 TNAS1 kernel: tty_ldisc_open+0x29/0x50
Nov 9 01:02:26 TNAS1 kernel: tty_ldisc_reinit+0x97/0xe0
Nov 9 01:02:26 TNAS1 kernel: tty_ldisc_hangup+0x100/0x200
Nov 9 01:02:26 TNAS1 kernel: __tty_hangup.part.0+0x1e4/0x360
Nov 9 01:02:26 TNAS1 kernel: tty_ioctl+0x7a3/0x8e0
Nov 9 01:02:26 TNAS1 kernel: __x64_sys_ioctl+0x94/0xd0
Nov 9 01:02:26 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:02:26 TNAS1 kernel: ? do_fault+0x281/0x4d0
Nov 9 01:02:26 TNAS1 kernel: ? __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:26 TNAS1 kernel: ? __count_memcg_events+0x53/0xf0
Nov 9 01:02:26 TNAS1 kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 9 01:02:26 TNAS1 kernel: ? handle_mm_fault+0xae/0x2d0
Nov 9 01:02:26 TNAS1 kernel: ? do_user_addr_fault+0x17f/0x670
Nov 9 01:02:26 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:02:26 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:02:26 TNAS1 kernel: RIP: 0033:0x7f15adcb3cdb
Nov 9 01:02:26 TNAS1 kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
Nov 9 01:02:26 TNAS1 kernel: RSP: 002b:00007fffc1aae690 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 9 01:02:26 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000000000000027 RCX: 00007f15adcb3cdb
Nov 9 01:02:26 TNAS1 kernel: RDX: 0000000000000000 RSI: 0000000000005437 RDI: 0000000000000027
Nov 9 01:02:26 TNAS1 kernel: RBP: 0000000000000027 R08: 00007fffc1aae670 R09: 0000564cb2755c90
Nov 9 01:02:26 TNAS1 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000564cb249a9f0
Nov 9 01:02:26 TNAS1 kernel: R13: 0000000000000009 R14: 0000000000000005 R15: 0000000000000000
Nov 9 01:02:26 TNAS1 kernel:
Nov 9 01:02:26 TNAS1 kernel: Mem-Info:
Nov 9 01:02:26 TNAS1 kernel: active_anon:65307 inactive_anon:7677663 isolated_anon:0
active_file:249 inactive_file:149 isolated_file:0
unevictable:1000 dirty:35 writeback:147
slab_reclaimable:57924 slab_unreclaimable:80898
mapped:742 shmem:92803 pagetables:16482
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:28169 free_pcp:70 free_cma:255
Nov 9 01:02:26 TNAS1 kernel: Node 0 active_anon:261228kB inactive_anon:30710652kB active_file:996kB inactive_file:284kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:2968kB dirty:140kB writeback:588kB shmem:371212kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11888kB pagetables:65928kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:26 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:26 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:26 TNAS1 kernel: Node 0 DMA32 free:7144kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480620kB active_file:44kB inactive_file:12kB unevictable:0kB writepending:60kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:26 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:26 TNAS1 kernel: Node 0 Normal free:106008kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261224kB inactive_anon:27218796kB active_file:384kB inactive_file:620kB unevictable:4000kB writepending:668kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:1020kB
Nov 9 01:02:26 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:26 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:26 TNAS1 kernel: Node 0 DMA32: 186*4kB (UME) 72*8kB (UME) 69*16kB (UME) 44*32kB (UME) 34*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7160kB
Nov 9 01:02:26 TNAS1 kernel: Node 0 Normal: 2242*4kB (UMEC) 567*8kB (UMEC) 1627*16kB (UMEC) 2075*32kB (UEC) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 105936kB
Nov 9 01:02:26 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:26 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:26 TNAS1 kernel: 93301 total pagecache pages
Nov 9 01:02:26 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:26 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:26 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:26 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:26 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:26 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:26 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:26 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:26 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:26 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:26 TNAS1 kernel: [ 728] 104 728 2091 310 256 54 0 65536 0 -900 dbus-daemon
Nov 9 01:02:26 TNAS1 kernel: [ 741] 0 741 11506 420 224 196 0 106496 0 -250 systemd-journal
Nov 9 01:02:26 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:26 TNAS1 kernel: [ 1630] 0 1630 8368224 7493221 7493066 91 64 60821504 0 -1000 asyncio_loop
Nov 9 01:02:26 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:26 TNAS1 kernel: [ 1681] 0 1681 485024 35365 35252 113 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:26 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:26 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:26 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:26 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:26 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:26 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:26 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:26 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:26 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:26 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:26 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:26 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:26 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:26 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:26 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:02:26 TNAS1 kernel: [ 3691] 0 3691 12416 332 224 108 0 94208 0 0 systemd-logind
Nov 9 01:02:26 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:26 TNAS1 kernel: [ 3708] 0 3708 523983 4212 4140 72 0 327680 0 -999 containerd
Nov 9 01:02:26 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:26 TNAS1 kernel: [ 3809] 122 3809 3089 181 121 60 0 49152 0 0 usbhid-ups
Nov 9 01:02:26 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:26 TNAS1 kernel: [ 3962] 122 3962 1695 121 96 25 0 57344 0 0 upsd
Nov 9 01:02:26 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:26 TNAS1 kernel: [ 4009] 122 4009 2127 173 96 77 0 53248 0 0 upsmon
Nov 9 01:02:26 TNAS1 kernel: [ 4132] 0 4132 4136 330 224 106 0 69632 0 0 systemd-machine
Nov 9 01:02:26 TNAS1 kernel: [ 5386] 105 5386 1781 152 96 56 0 49152 0 0 avahi-daemon
Nov 9 01:02:26 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:26 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:26 TNAS1 kernel: [ 6672] 0 6672 13921 5544 5472 72 0 139264 0 -1000 websockify
Nov 9 01:02:26 TNAS1 kernel: [ 6757] 999 6757 116708 29304 29218 22 64 475136 0 -900 netdata
Nov 9 01:02:26 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:26 TNAS1 kernel: [ 7021] 999 7021 22248 4929 4846 83 0 147456 0 -900 python.d.plugin
Nov 9 01:02:26 TNAS1 kernel: [2479756] 999 2479756 1113 260 160 100 0 49152 0 -900 bash
Nov 9 01:02:26 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:02:26 TNAS1 kernel: [2773557] 0 2773557 206757 12334 12274 60 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:26 TNAS1 kernel: [2773561] 0 2773561 207385 13075 12971 104 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:26 TNAS1 kernel: [2776151] 0 2776151 207108 12958 12886 72 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:26 TNAS1 kernel: [2777444] 0 2777444 206501 12381 12278 103 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:26 TNAS1 kernel: [2777446] 0 2777446 207155 12675 12561 114 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:26 TNAS1 kernel: [2782363] 0 2782363 243082 707 544 163 0 430080 0 0 syslog-ng
Nov 9 01:02:26 TNAS1 kernel: [2782383] 0 2782383 203061 551 512 39 0 212992 0 0 libvirtd
Nov 9 01:02:26 TNAS1 kernel: [2782440] 999 2782440 1511 128 64 64 0 45056 0 -900 awk
Nov 9 01:02:26 TNAS1 kernel: [2782441] 0 2782441 2630 146 0 146 0 57344 0 0 truenas_audit_h
Nov 9 01:02:26 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/syslog-ng.service,task=syslog-ng,pid=2782363,uid=0
Nov 9 01:02:26 TNAS1 kernel: Out of memory: Killed process 2782363 (syslog-ng) total-vm:972328kB, anon-rss:2176kB, file-rss:652kB, shmem-rss:0kB, UID:0 pgtables:420kB oom_score_adj:0
Nov 9 01:02:26 TNAS1 systemd[1]: getty@tty1.service: Scheduled restart job, restart counter is at 3.
Nov 9 01:02:26 TNAS1 systemd[1]: Stopped getty@tty1.service - Getty on tty1.
Nov 9 01:02:26 TNAS1 systemd[1]: Started getty@tty1.service - Getty on tty1.
Nov 9 01:02:26 TNAS1 systemd[1]: init.scope: A process of this unit has been killed by the OOM killer.
Nov 9 01:02:26 TNAS1 systemd[1]: init.scope: Failed with result 'oom-kill'.
Nov 9 01:02:26 TNAS1 systemd[1]: init.scope: Consumed 15.354s CPU time.
Nov 9 01:02:26 TNAS1 systemd[1]: getty@tty1.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:26 TNAS1 systemd[1]: getty@tty1.service: Failed with result 'signal'.
Nov 9 01:02:27 TNAS1 kernel: libvirtd invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:02:27 TNAS1 kernel: CPU: 1 UID: 0 PID: 2782383 Comm: libvirtd Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:27 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:27 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:27 TNAS1 kernel: Call Trace:
Nov 9 01:02:27 TNAS1 kernel:
Nov 9 01:02:27 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:27 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:27 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:27 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:27 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:27 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:27 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:27 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:02:27 TNAS1 kernel: do_fault+0x240/0x4d0
Nov 9 01:02:27 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:27 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:02:27 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:02:27 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:02:27 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:02:27 TNAS1 kernel: RIP: 0033:0x7fbd25522262
Nov 9 01:02:27 TNAS1 kernel: Code: 0f 1f 84 00 00 00 00 00 3d 53 e5 74 64 75 0e 48 89 da 44 89 e6 4c 89 ef e8 cb f2 ff ff 49 39 9d c0 02 00 00 0f 84 ae 01 00 00 <8b> 43 c8 48 83 eb 38 83 f8 04 75 d2 48 8b 53 10 49 8b 45 00 4c 8b
Nov 9 01:02:27 TNAS1 kernel: RSP: 002b:00007ffd3b5b7510 EFLAGS: 00010283
Nov 9 01:02:27 TNAS1 kernel: RAX: 0000000000000046 RBX: 00007fbd1834e270 RCX: 000000000000000a
Nov 9 01:02:27 TNAS1 kernel: RDX: 00007fbd1834e040 RSI: 00007fbd18378cd8 RDI: 000000006fffffff
Nov 9 01:02:27 TNAS1 kernel: RBP: 00007ffd3b5b7660 R08: 0000000000000006 R09: 0000000070000025
Nov 9 01:02:27 TNAS1 kernel: R10: 0000000000000035 R11: 000000006ffffdff R12: 0000000000000005
Nov 9 01:02:27 TNAS1 kernel: R13: 0000565147d5d020 R14: 00007ffd3b5b7700 R15: 00007fbd18535ba8
Nov 9 01:02:27 TNAS1 kernel:
Nov 9 01:02:27 TNAS1 kernel: Mem-Info:
Nov 9 01:02:27 TNAS1 kernel: active_anon:65306 inactive_anon:7677468 isolated_anon:0
active_file:981 inactive_file:47 isolated_file:0
unevictable:1000 dirty:180 writeback:291
slab_reclaimable:57924 slab_unreclaimable:80893
mapped:1238 shmem:92802 pagetables:16378
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27918 free_pcp:0 free_cma:254
Nov 9 01:02:27 TNAS1 kernel: Node 0 active_anon:261224kB inactive_anon:30709872kB active_file:3924kB inactive_file:188kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:4952kB dirty:720kB writeback:1164kB shmem:371208kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11744kB pagetables:65512kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:27 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:27 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:27 TNAS1 kernel: Node 0 DMA32 free:7100kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480444kB active_file:52kB inactive_file:100kB unevictable:0kB writepending:8kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:27 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:27 TNAS1 kernel: Node 0 Normal free:104544kB boost:78016kB min:138408kB low:167668kB high:196928kB reserved_highatomic:4096KB active_anon:261220kB inactive_anon:27218192kB active_file:3872kB inactive_file:88kB unevictable:4000kB writepending:1876kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:1016kB
Nov 9 01:02:27 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:27 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:27 TNAS1 kernel: Node 0 DMA32: 188*4kB (UME) 81*8kB (UME) 70*16kB (UME) 44*32kB (UME) 34*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7256kB
Nov 9 01:02:27 TNAS1 kernel: Node 0 Normal: 2371*4kB (UMEC) 425*8kB (UMEC) 1595*16kB (UMEC) 2077*32kB (UMEC) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 104932kB
Nov 9 01:02:27 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:27 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:27 TNAS1 kernel: 93655 total pagecache pages
Nov 9 01:02:27 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:27 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:27 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:27 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:27 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:27 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:27 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:27 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:27 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:27 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:27 TNAS1 kernel: [ 728] 104 728 2091 310 256 54 0 65536 0 -900 dbus-daemon
Nov 9 01:02:27 TNAS1 kernel: [ 741] 0 741 11506 676 224 452 0 106496 0 -250 systemd-journal
Nov 9 01:02:27 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:27 TNAS1 kernel: [ 1630] 0 1630 8368224 7493458 7493066 328 64 60821504 0 -1000 asyncio_loop
Nov 9 01:02:27 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:27 TNAS1 kernel: [ 1681] 0 1681 485024 35365 35252 113 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:27 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:27 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:27 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:27 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:27 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:27 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:27 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:27 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:27 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:27 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:27 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:27 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:27 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:27 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:27 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:02:27 TNAS1 kernel: [ 3691] 0 3691 12416 332 224 108 0 94208 0 0 systemd-logind
Nov 9 01:02:27 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:27 TNAS1 kernel: [ 3708] 0 3708 523983 4212 4140 72 0 327680 0 -999 containerd
Nov 9 01:02:27 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:27 TNAS1 kernel: [ 3809] 122 3809 3089 181 121 60 0 49152 0 0 usbhid-ups
Nov 9 01:02:27 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:27 TNAS1 kernel: [ 3962] 122 3962 1695 121 96 25 0 57344 0 0 upsd
Nov 9 01:02:27 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:27 TNAS1 kernel: [ 4009] 122 4009 2127 173 96 77 0 53248 0 0 upsmon
Nov 9 01:02:27 TNAS1 kernel: [ 4132] 0 4132 4136 330 224 106 0 69632 0 0 systemd-machine
Nov 9 01:02:27 TNAS1 kernel: [ 5386] 105 5386 1781 152 96 56 0 49152 0 0 avahi-daemon
Nov 9 01:02:27 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:27 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:27 TNAS1 kernel: [ 6672] 0 6672 13921 5640 5472 168 0 139264 0 -1000 websockify
Nov 9 01:02:27 TNAS1 kernel: [ 6757] 999 6757 116708 29407 29218 125 64 475136 0 -900 netdata
Nov 9 01:02:27 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:27 TNAS1 kernel: [ 7021] 999 7021 22248 5214 4846 368 0 147456 0 -900 python.d.plugin
Nov 9 01:02:27 TNAS1 kernel: [2479756] 999 2479756 1113 260 160 100 0 49152 0 -900 bash
Nov 9 01:02:27 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:02:27 TNAS1 kernel: [2773557] 0 2773557 206757 12334 12274 60 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:27 TNAS1 kernel: [2773561] 0 2773561 207385 13075 12971 104 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:27 TNAS1 kernel: [2776151] 0 2776151 207108 12958 12886 72 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:27 TNAS1 kernel: [2777444] 0 2777444 206501 12381 12278 103 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:27 TNAS1 kernel: [2777446] 0 2777446 207155 12675 12561 114 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:27 TNAS1 kernel: [2782383] 0 2782383 203549 556 512 44 0 212992 0 0 libvirtd
Nov 9 01:02:27 TNAS1 kernel: [2782440] 999 2782440 1577 195 96 99 0 45056 0 -900 awk
Nov 9 01:02:27 TNAS1 kernel: [2782441] 0 2782441 3115 790 448 342 0 61440 0 0 truenas_audit_h
Nov 9 01:02:27 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=libvirtd.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tnaudit.service,task=truenas_audit_h,pid=2782441,uid=0
Nov 9 01:02:27 TNAS1 kernel: Out of memory: Killed process 2782441 (truenas_audit_h) total-vm:12460kB, anon-rss:1792kB, file-rss:1368kB, shmem-rss:0kB, UID:0 pgtables:60kB oom_score_adj:0
Nov 9 01:02:27 TNAS1 kernel: libvirtd invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:02:27 TNAS1 kernel: CPU: 7 UID: 0 PID: 2782383 Comm: libvirtd Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:27 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:27 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:27 TNAS1 kernel: Call Trace:
Nov 9 01:02:27 TNAS1 kernel:
Nov 9 01:02:27 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:27 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:27 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:27 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:27 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:27 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:27 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:27 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:02:27 TNAS1 kernel: do_fault+0x240/0x4d0
Nov 9 01:02:27 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:27 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:02:27 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:02:27 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:02:27 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:02:27 TNAS1 kernel: RIP: 0033:0x7fbd25522262
Nov 9 01:02:27 TNAS1 kernel: Code: 0f 1f 84 00 00 00 00 00 3d 53 e5 74 64 75 0e 48 89 da 44 89 e6 4c 89 ef e8 cb f2 ff ff 49 39 9d c0 02 00 00 0f 84 ae 01 00 00 <8b> 43 c8 48 83 eb 38 83 f8 04 75 d2 48 8b 53 10 49 8b 45 00 4c 8b
Nov 9 01:02:27 TNAS1 kernel: RSP: 002b:00007ffd3b5b7510 EFLAGS: 00010283
Nov 9 01:02:27 TNAS1 kernel: RAX: 0000000000000046 RBX: 00007fbd1834e270 RCX: 000000000000000a
Nov 9 01:02:27 TNAS1 kernel: RDX: 00007fbd1834e040 RSI: 00007fbd18378cd8 RDI: 000000006fffffff
Nov 9 01:02:27 TNAS1 kernel: RBP: 00007ffd3b5b7660 R08: 0000000000000006 R09: 0000000070000025
Nov 9 01:02:27 TNAS1 kernel: R10: 0000000000000035 R11: 000000006ffffdff R12: 0000000000000005
Nov 9 01:02:27 TNAS1 kernel: R13: 0000565147d5d020 R14: 00007ffd3b5b7700 R15: 00007fbd18535ba8
Nov 9 01:02:27 TNAS1 kernel:
Nov 9 01:02:27 TNAS1 kernel: Mem-Info:
Nov 9 01:02:27 TNAS1 kernel: active_anon:65305 inactive_anon:7676961 isolated_anon:0
active_file:422 inactive_file:44 isolated_file:0
unevictable:1000 dirty:180 writeback:291
slab_reclaimable:57924 slab_unreclaimable:80893
mapped:649 shmem:92802 pagetables:16363
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:28971 free_pcp:14 free_cma:272
Nov 9 01:02:27 TNAS1 kernel: Node 0 active_anon:261220kB inactive_anon:30707844kB active_file:1688kB inactive_file:176kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:2596kB dirty:720kB writeback:1164kB shmem:371208kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11744kB pagetables:65452kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:27 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:27 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:27 TNAS1 kernel: Node 0 DMA32 free:7092kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480416kB active_file:20kB inactive_file:8kB unevictable:0kB writepending:8kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:27 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:27 TNAS1 kernel: Node 0 Normal free:108764kB boost:78016kB min:138408kB low:167668kB high:196928kB reserved_highatomic:4096KB active_anon:261216kB inactive_anon:27216192kB active_file:1376kB inactive_file:224kB unevictable:4000kB writepending:1876kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:56kB local_pcp:0kB free_cma:1088kB
Nov 9 01:02:27 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:27 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:27 TNAS1 kernel: Node 0 DMA32: 208*4kB (UME) 82*8kB (UME) 70*16kB (UME) 44*32kB (UME) 34*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7344kB
Nov 9 01:02:27 TNAS1 kernel: Node 0 Normal: 2871*4kB (UMEC) 652*8kB (UMEC) 1618*16kB (UMEC) 2079*32kB (UMEC) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 109180kB
Nov 9 01:02:27 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:27 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:27 TNAS1 kernel: 93277 total pagecache pages
Nov 9 01:02:27 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:27 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:27 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:27 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:27 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:27 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:27 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:27 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:27 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:27 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:27 TNAS1 kernel: [ 728] 104 728 2091 310 256 54 0 65536 0 -900 dbus-daemon
Nov 9 01:02:27 TNAS1 kernel: [ 741] 0 741 11506 676 224 452 0 106496 0 -250 systemd-journal
Nov 9 01:02:27 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:27 TNAS1 kernel: [ 1630] 0 1630 8368224 7493362 7493066 232 64 60821504 0 -1000 asyncio_loop
Nov 9 01:02:27 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:27 TNAS1 kernel: [ 1681] 0 1681 485024 35365 35252 113 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:27 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:27 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:27 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:27 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:27 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:27 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:27 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:27 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:27 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:27 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:27 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:27 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:27 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:27 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:27 TNAS1 kernel: [ 3663] 0 3663 3227 626 531 95 0 73728 0 0 smartd
Nov 9 01:02:27 TNAS1 kernel: [ 3691] 0 3691 12416 332 224 108 0 94208 0 0 systemd-logind
Nov 9 01:02:27 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:27 TNAS1 kernel: [ 3708] 0 3708 523983 4212 4140 72 0 327680 0 -999 containerd
Nov 9 01:02:27 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:27 TNAS1 kernel: [ 3809] 122 3809 3089 181 121 60 0 49152 0 0 usbhid-ups
Nov 9 01:02:27 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:27 TNAS1 kernel: [ 3962] 122 3962 1695 121 96 25 0 57344 0 0 upsd
Nov 9 01:02:27 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:27 TNAS1 kernel: [ 4009] 122 4009 2127 173 96 77 0 53248 0 0 upsmon
Nov 9 01:02:27 TNAS1 kernel: [ 4132] 0 4132 4136 330 224 106 0 69632 0 0 systemd-machine
Nov 9 01:02:27 TNAS1 kernel: [ 5386] 105 5386 1781 120 96 24 0 49152 0 0 avahi-daemon
Nov 9 01:02:27 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:27 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:27 TNAS1 kernel: [ 6672] 0 6672 13921 5640 5472 168 0 139264 0 -1000 websockify
Nov 9 01:02:27 TNAS1 kernel: [ 6757] 999 6757 116708 29375 29218 93 64 475136 0 -900 netdata
Nov 9 01:02:27 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:27 TNAS1 kernel: [ 7021] 999 7021 22248 5086 4846 240 0 147456 0 -900 python.d.plugin
Nov 9 01:02:27 TNAS1 kernel: [2479756] 999 2479756 1113 260 160 100 0 49152 0 -900 bash
Nov 9 01:02:27 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:02:27 TNAS1 kernel: [2773557] 0 2773557 206757 12334 12274 60 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:27 TNAS1 kernel: [2773561] 0 2773561 207385 13075 12971 104 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:27 TNAS1 kernel: [2776151] 0 2776151 207108 12958 12886 72 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:27 TNAS1 kernel: [2777444] 0 2777444 206501 12381 12278 103 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:27 TNAS1 kernel: [2777446] 0 2777446 207155 12675 12561 114 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:27 TNAS1 kernel: [2782383] 0 2782383 203549 556 512 44 0 212992 0 0 libvirtd
Nov 9 01:02:27 TNAS1 kernel: [2782440] 999 2782440 1577 195 96 99 0 45056 0 -900 awk
Nov 9 01:02:27 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=libvirtd.service,mems_allowed=0,global_oom,task_memcg=/system.slice/smartmontools.service,task=smartd,pid=3663,uid=0
Nov 9 01:02:27 TNAS1 kernel: Out of memory: Killed process 3663 (smartd) total-vm:12908kB, anon-rss:2124kB, file-rss:380kB, shmem-rss:0kB, UID:0 pgtables:72kB oom_score_adj:0
Nov 9 01:02:27 TNAS1 systemd-journald[741]: Data hash table of /var/log/journal/83c882df39b9450b907f2293dd7876fa/system.journal has a fill level at 75.0 (8533 of 11377 items, 6553600 file size, 768 bytes per hash table item), suggesting rotation.
Nov 9 01:02:27 TNAS1 systemd-journald[741]: /var/log/journal/83c882df39b9450b907f2293dd7876fa/system.journal: Journal header limits reached or header out-of-date, rotating.
Nov 9 01:02:27 TNAS1 systemd[1]: smartmontools.service: A process of this unit has been killed by the OOM killer.
Nov 9 01:02:27 TNAS1 middlewared[1630]: libvirt: XML-RPC error : Cannot write data: Broken pipe
Nov 9 01:02:27 TNAS1 systemd[1]: smartmontools.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:27 TNAS1 systemd[1]: smartmontools.service: Failed with result 'oom-kill'.
Nov 9 01:02:27 TNAS1 systemd[1]: smartmontools.service: Consumed 1.003s CPU time.
Nov 9 01:02:27 TNAS1 systemd[1]: syslog-ng.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:27 TNAS1 systemd[1]: syslog-ng.service: Failed with result 'signal'.
Nov 9 01:02:27 TNAS1 systemd[1]: tnaudit.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:27 TNAS1 systemd[1]: tnaudit.service: Failed with result 'signal'.
Nov 9 01:02:27 TNAS1 systemd[1]: Starting tnaudit.service - TrueNAS Auditd Handler...
Nov 9 01:02:27 TNAS1 systemd[1]: syslog-ng.service: Scheduled restart job, restart counter is at 3.
Nov 9 01:02:27 TNAS1 systemd[1]: Stopped syslog-ng.service - System Logger Daemon.
Nov 9 01:02:32 TNAS1 systemd[1]: Starting syslog-ng.service - System Logger Daemon...
Nov 9 01:02:32 TNAS1 systemd[1]: getty@tty1.service: Scheduled restart job, restart counter is at 4.
Nov 9 01:02:32 TNAS1 systemd[1]: Started libvirtd.service - Virtualization daemon.
Nov 9 01:02:32 TNAS1 systemd[1]: Stopped getty@tty1.service - Getty on tty1.
Nov 9 01:02:33 TNAS1 kernel: (ndler.py) invoked oom-killer: gfp_mask=0x402dc2(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM|__GFP_NOWARN|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:02:33 TNAS1 kernel: CPU: 5 UID: 0 PID: 2782445 Comm: (ndler.py) Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:33 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:33 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:33 TNAS1 kernel: Call Trace:
Nov 9 01:02:33 TNAS1 kernel:
Nov 9 01:02:33 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:33 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:33 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:33 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:33 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:33 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:33 TNAS1 kernel: alloc_pages_bulk_noprof+0x3df/0x720
Nov 9 01:02:33 TNAS1 kernel: __vmalloc_node_range_noprof+0x24f/0x850
Nov 9 01:02:33 TNAS1 kernel: ? bpf_prog_alloc_no_stats+0x3d/0x240
Nov 9 01:02:33 TNAS1 kernel: ? dmu_read_impl+0x165/0x1e0 [zfs]
Nov 9 01:02:33 TNAS1 kernel: ? __pfx_seccomp_check_filter+0x10/0x10
Nov 9 01:02:33 TNAS1 kernel: __vmalloc_node_noprof+0x4e/0x70
Nov 9 01:02:33 TNAS1 kernel: ? bpf_prog_alloc_no_stats+0x3d/0x240
Nov 9 01:02:33 TNAS1 kernel: bpf_prog_alloc_no_stats+0x3d/0x240
Nov 9 01:02:33 TNAS1 kernel: ? __pfx_seccomp_check_filter+0x10/0x10
Nov 9 01:02:33 TNAS1 kernel: bpf_prog_alloc+0x14/0xb0
Nov 9 01:02:33 TNAS1 kernel: bpf_prog_create_from_user+0x53/0x120
Nov 9 01:02:33 TNAS1 kernel: do_seccomp+0x12f/0xa90
Nov 9 01:02:33 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:02:33 TNAS1 kernel: ? ptep_set_access_flags+0x32/0x40
Nov 9 01:02:33 TNAS1 kernel: ? wp_page_reuse+0x8d/0xa0
Nov 9 01:02:33 TNAS1 kernel: ? do_wp_page+0x8c4/0xc90
Nov 9 01:02:33 TNAS1 kernel: ? __pte_offset_map+0x1b/0x180
Nov 9 01:02:33 TNAS1 kernel: ? __handle_mm_fault+0xb17/0xfd0
Nov 9 01:02:33 TNAS1 kernel: ? __count_memcg_events+0x53/0xf0
Nov 9 01:02:33 TNAS1 kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 9 01:02:33 TNAS1 kernel: ? handle_mm_fault+0xae/0x2d0
Nov 9 01:02:33 TNAS1 kernel: ? do_user_addr_fault+0x379/0x670
Nov 9 01:02:33 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:02:33 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:02:33 TNAS1 kernel: RIP: 0033:0x7f15adcb7799
Nov 9 01:02:33 TNAS1 kernel: Code: 08 89 e8 5b 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 37 06 0d 00 f7 d8 64 89 01 48
Nov 9 01:02:33 TNAS1 kernel: RSP: 002b:00007fffc1aae028 EFLAGS: 00000246 ORIG_RAX: 000000000000013d
Nov 9 01:02:33 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f15adcb7799
Nov 9 01:02:33 TNAS1 kernel: RDX: 0000564cb260cfa0 RSI: 0000000000000000 RDI: 0000000000000001
Nov 9 01:02:33 TNAS1 kernel: RBP: 0000564cb26d4120 R08: 0000000000000001 R09: 0000000700000002
Nov 9 01:02:33 TNAS1 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
Nov 9 01:02:33 TNAS1 kernel: R13: 0000000700000001 R14: 0000000000000001 R15: 0000000000000000
Nov 9 01:02:33 TNAS1 kernel:
Nov 9 01:02:33 TNAS1 kernel: Mem-Info:
Nov 9 01:02:33 TNAS1 kernel: active_anon:65306 inactive_anon:7677858 isolated_anon:0
active_file:916 inactive_file:594 isolated_file:0
unevictable:1000 dirty:26 writeback:55
slab_reclaimable:57892 slab_unreclaimable:80818
mapped:1817 shmem:92804 pagetables:16445
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:26982 free_pcp:0 free_cma:107
Nov 9 01:02:33 TNAS1 kernel: Node 0 active_anon:261224kB inactive_anon:30711432kB active_file:3664kB inactive_file:2376kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:7268kB dirty:104kB writeback:220kB shmem:371216kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11904kB pagetables:65780kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:33 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:33 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:33 TNAS1 kernel: Node 0 DMA32 free:6880kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480656kB active_file:104kB inactive_file:100kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:33 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:33 TNAS1 kernel: Node 0 Normal free:101020kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261220kB inactive_anon:27219248kB active_file:3132kB inactive_file:1796kB unevictable:4000kB writepending:324kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:428kB
Nov 9 01:02:33 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:33 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:33 TNAS1 kernel: Node 0 DMA32: 125*4kB (UME) 80*8kB (UME) 70*16kB (UME) 44*32kB (UME) 34*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6996kB
Nov 9 01:02:33 TNAS1 kernel: Node 0 Normal: 1190*4kB (UMEC) 542*8kB (UME) 1583*16kB (UE) 2042*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 99768kB
Nov 9 01:02:33 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:33 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:33 TNAS1 kernel: 94447 total pagecache pages
Nov 9 01:02:33 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:33 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:33 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:33 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:33 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:33 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:33 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:33 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:33 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:33 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:33 TNAS1 kernel: [ 728] 104 728 2091 461 256 205 0 65536 0 -900 dbus-daemon
Nov 9 01:02:33 TNAS1 kernel: [ 741] 0 741 11506 570 224 346 0 98304 0 -250 systemd-journal
Nov 9 01:02:33 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:33 TNAS1 kernel: [ 1630] 0 1630 8368992 7494452 7493930 458 64 60825600 0 -1000 asyncio_loop
Nov 9 01:02:33 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:33 TNAS1 kernel: [ 1681] 0 1681 485024 35429 35252 177 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:33 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:33 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:33 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:33 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:33 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:33 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:33 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:33 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:33 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:33 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:33 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:33 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:33 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:33 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:33 TNAS1 kernel: [ 3691] 0 3691 12416 601 224 377 0 94208 0 0 systemd-logind
Nov 9 01:02:33 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:33 TNAS1 kernel: [ 3708] 0 3708 523983 4186 4140 46 0 327680 0 -999 containerd
Nov 9 01:02:33 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:33 TNAS1 kernel: [ 3809] 122 3809 3089 292 121 171 0 49152 0 0 usbhid-ups
Nov 9 01:02:33 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:33 TNAS1 kernel: [ 3962] 122 3962 1695 200 96 104 0 57344 0 0 upsd
Nov 9 01:02:33 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:33 TNAS1 kernel: [ 4009] 122 4009 2127 146 96 50 0 53248 0 0 upsmon
Nov 9 01:02:33 TNAS1 kernel: [ 4132] 0 4132 4136 450 224 226 0 69632 0 0 systemd-machine
Nov 9 01:02:33 TNAS1 kernel: [ 5386] 105 5386 1781 219 96 123 0 49152 0 0 avahi-daemon
Nov 9 01:02:33 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:33 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:33 TNAS1 kernel: [ 6672] 0 6672 13921 5639 5472 167 0 139264 0 -1000 websockify
Nov 9 01:02:33 TNAS1 kernel: [ 6757] 999 6757 116708 29417 29218 135 64 475136 0 -900 netdata
Nov 9 01:02:33 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:33 TNAS1 kernel: [ 7021] 999 7021 22248 5303 4846 457 0 147456 0 -900 python.d.plugin
Nov 9 01:02:33 TNAS1 kernel: [2479756] 999 2479756 1113 327 160 167 0 49152 0 -900 bash
Nov 9 01:02:33 TNAS1 kernel: [2709452] 0 2709452 94214 62 32 30 0 86016 0 0 nscd
Nov 9 01:02:33 TNAS1 kernel: [2773557] 0 2773557 206757 12341 12274 67 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:33 TNAS1 kernel: [2773561] 0 2773561 207385 13048 12971 77 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:33 TNAS1 kernel: [2776151] 0 2776151 207108 13003 12886 117 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:33 TNAS1 kernel: [2777444] 0 2777444 206501 12407 12278 129 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:33 TNAS1 kernel: [2777446] 0 2777446 207155 12650 12561 89 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:33 TNAS1 kernel: [2782383] 0 2782383 351207 1032 608 424 0 282624 0 0 libvirtd
Nov 9 01:02:33 TNAS1 kernel: [2782445] 0 2782445 42617 1856 1470 386 0 94208 0 0 (ndler.py)
Nov 9 01:02:33 TNAS1 kernel: [2782448] 0 2782448 3872 323 96 227 0 61440 0 0 syslog-ng
Nov 9 01:02:33 TNAS1 kernel: [2782459] 999 2782459 1113 424 204 220 0 49152 0 -900 bash
Nov 9 01:02:33 TNAS1 kernel: [2782460] 999 2782460 623 195 0 195 0 40960 0 -900 timeout
Nov 9 01:02:33 TNAS1 kernel: [2782461] 999 2782461 1930 477 64 413 0 57344 0 -900 upsc
Nov 9 01:02:33 TNAS1 kernel: [2782462] 0 2782462 42617 1618 1470 148 0 94208 0 0 (cli)
Nov 9 01:02:33 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=tnaudit.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tnaudit.service,task=(ndler.py),pid=2782445,uid=0
Nov 9 01:02:33 TNAS1 kernel: Out of memory: Killed process 2782445 ((ndler.py)) total-vm:170468kB, anon-rss:5880kB, file-rss:1544kB, shmem-rss:0kB, UID:0 pgtables:92kB oom_score_adj:0
Nov 9 01:02:33 TNAS1 systemd[1]: Started getty@tty1.service - Getty on tty1.
Nov 9 01:02:33 TNAS1 systemd[1]: tnaudit.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:33 TNAS1 systemd[1]: tnaudit.service: Failed with result 'signal'.
Nov 9 01:02:34 TNAS1 systemd[1]: Failed to start tnaudit.service - TrueNAS Auditd Handler.
Nov 9 01:02:34 TNAS1 systemd[1]: Starting tnaudit.service - TrueNAS Auditd Handler...
Nov 9 01:02:37 TNAS1 kernel: bash invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=1, oom_score_adj=-900
Nov 9 01:02:37 TNAS1 kernel: CPU: 1 UID: 999 PID: 2782472 Comm: bash Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:37 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:37 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:37 TNAS1 kernel: Call Trace:
Nov 9 01:02:37 TNAS1 kernel:
Nov 9 01:02:37 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:37 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:37 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:37 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:37 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:37 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:37 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:37 TNAS1 kernel: get_free_pages_noprof+0x11/0x40
Nov 9 01:02:37 TNAS1 kernel: pgd_alloc+0x1f/0x1d0
Nov 9 01:02:37 TNAS1 kernel: mm_init+0x1b3/0x3c0
Nov 9 01:02:37 TNAS1 kernel: alloc_bprm+0xb1/0x3a0
Nov 9 01:02:37 TNAS1 kernel: do_execveat_common.isra.0+0x92/0x1e0
Nov 9 01:02:37 TNAS1 kernel: __x64_sys_execve+0x36/0x40
Nov 9 01:02:37 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:02:37 TNAS1 kernel: ? do_fault+0x281/0x4d0
Nov 9 01:02:37 TNAS1 kernel: ? __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:37 TNAS1 kernel: ? __count_memcg_events+0x53/0xf0
Nov 9 01:02:37 TNAS1 kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 9 01:02:37 TNAS1 kernel: ? handle_mm_fault+0xae/0x2d0
Nov 9 01:02:37 TNAS1 kernel: ? do_user_addr_fault+0x379/0x670
Nov 9 01:02:37 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:02:37 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:02:37 TNAS1 kernel: RIP: 0033:0x7fe37389ca97
Nov 9 01:02:37 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7fe37389ca6d.
Nov 9 01:02:37 TNAS1 kernel: RSP: 002b:00007ffe09353a38 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
Nov 9 01:02:37 TNAS1 kernel: RAX: ffffffffffffffda RBX: 000055cbcc0ee4c0 RCX: 00007fe37389ca97
Nov 9 01:02:37 TNAS1 kernel: RDX: 000055cbcc0944d0 RSI: 000055cbcc096600 RDI: 000055cbcc0ee4c0
Nov 9 01:02:37 TNAS1 kernel: RBP: 0000000000000000 R08: 0000000000000007 R09: 000055cbcc10e6a0
Nov 9 01:02:37 TNAS1 kernel: R10: 6c3630914f812537 R11: 0000000000000246 R12: 00000000ffffffff
Nov 9 01:02:37 TNAS1 kernel: R13: 000055cbcc096600 R14: 000055cbcc0944d0 R15: 000055cbcc0ec960
Nov 9 01:02:37 TNAS1 kernel:
Nov 9 01:02:37 TNAS1 kernel: Mem-Info:
Nov 9 01:02:37 TNAS1 kernel: active_anon:65304 inactive_anon:7678486 isolated_anon:0
active_file:50 inactive_file:16 isolated_file:0
unevictable:1000 dirty:0 writeback:78
slab_reclaimable:57889 slab_unreclaimable:80813
mapped:340 shmem:92804 pagetables:16410
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27536 free_pcp:0 free_cma:99
Nov 9 01:02:37 TNAS1 kernel: Node 0 active_anon:261216kB inactive_anon:30713944kB active_file:492kB inactive_file:64kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:1360kB dirty:0kB writeback:312kB shmem:371216kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11856kB pagetables:65640kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:37 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA32 free:7500kB boost:21136kB min:28292kB low:31760kB high:35228kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480924kB active_file:460kB inactive_file:0kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:37 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:37 TNAS1 kernel: Node 0 Normal free:102616kB boost:129024kB min:189416kB low:218676kB high:247936kB reserved_highatomic:4096KB active_anon:261212kB inactive_anon:27221784kB active_file:36kB inactive_file:224kB unevictable:4000kB writepending:312kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:396kB
Nov 9 01:02:37 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA32: 147*4kB (UME) 81*8kB (UME) 73*16kB (UME) 40*32kB (UME) 34*64kB (UME) 4*128kB (ME) 0*256kB 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6884kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 Normal: 2120*4kB (UMEC) 641*8kB (UMEC) 1572*16kB (UMEC) 2014*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 103208kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:37 TNAS1 kernel: 92973 total pagecache pages
Nov 9 01:02:37 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:37 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:37 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:37 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:37 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:37 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:37 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:37 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:37 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:37 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:37 TNAS1 kernel: [ 728] 104 728 2091 365 256 109 0 65536 0 -900 dbus-daemon
Nov 9 01:02:37 TNAS1 kernel: [ 741] 0 741 11506 415 224 191 0 98304 0 -250 systemd-journal
Nov 9 01:02:37 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:37 TNAS1 kernel: [ 1630] 0 1630 8369760 7494613 7494506 43 64 60833792 0 -1000 asyncio_loop
Nov 9 01:02:37 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:37 TNAS1 kernel: [ 1681] 0 1681 485024 35365 35252 113 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:37 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:37 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:37 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:37 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:37 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:37 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:37 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:37 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:37 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:37 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:37 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:37 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:37 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:37 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:37 TNAS1 kernel: [ 3691] 0 3691 12416 352 224 128 0 94208 0 0 systemd-logind
Nov 9 01:02:37 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:37 TNAS1 kernel: [ 3708] 0 3708 523983 4154 4140 14 0 327680 0 -999 containerd
Nov 9 01:02:37 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:37 TNAS1 kernel: [ 3809] 122 3809 3089 196 121 75 0 49152 0 0 usbhid-ups
Nov 9 01:02:37 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:37 TNAS1 kernel: [ 3962] 122 3962 1695 136 96 40 0 57344 0 0 upsd
Nov 9 01:02:37 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:37 TNAS1 kernel: [ 4009] 122 4009 2127 186 96 90 0 53248 0 0 upsmon
Nov 9 01:02:37 TNAS1 kernel: [ 4132] 0 4132 4136 265 224 41 0 69632 0 0 systemd-machine
Nov 9 01:02:37 TNAS1 kernel: [ 5386] 105 5386 1781 155 96 59 0 49152 0 0 avahi-daemon
Nov 9 01:02:37 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:37 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:37 TNAS1 kernel: [ 6672] 0 6672 13921 5613 5472 141 0 139264 0 -1000 websockify
Nov 9 01:02:37 TNAS1 kernel: [ 6757] 999 6757 116708 29363 29218 81 64 475136 0 -900 netdata
Nov 9 01:02:37 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:37 TNAS1 kernel: [ 7021] 999 7021 22248 4858 4846 12 0 147456 0 -900 python.d.plugin
Nov 9 01:02:37 TNAS1 kernel: [2479756] 999 2479756 1113 202 160 42 0 49152 0 -900 bash
Nov 9 01:02:37 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:02:37 TNAS1 kernel: [2773557] 0 2773557 206757 12341 12274 67 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2773561] 0 2773561 207385 13048 12971 77 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2776151] 0 2776151 207108 13003 12886 117 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2777444] 0 2777444 206501 12407 12278 129 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2777446] 0 2777446 207155 12650 12561 89 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2782383] 0 2782383 351207 758 672 86 0 282624 0 0 libvirtd
Nov 9 01:02:37 TNAS1 kernel: [2782448] 0 2782448 4544 306 160 146 0 77824 0 0 syslog-ng
Nov 9 01:02:37 TNAS1 kernel: [2782462] 0 2782462 42617 1554 1470 84 0 94208 0 0 (cli)
Nov 9 01:02:37 TNAS1 kernel: [2782467] 0 2782467 42617 1553 1470 83 0 94208 0 0 (ndler.py)
Nov 9 01:02:37 TNAS1 kernel: [2782472] 999 2782472 1113 216 205 11 0 49152 0 -900 bash
Nov 9 01:02:37 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/system.slice/system-getty.slice/getty@tty1.service,task=(cli),pid=2782462,uid=0
Nov 9 01:02:37 TNAS1 kernel: Out of memory: Killed process 2782462 ((cli)) total-vm:170468kB, anon-rss:5880kB, file-rss:336kB, shmem-rss:0kB, UID:0 pgtables:92kB oom_score_adj:0
Nov 9 01:02:37 TNAS1 kernel: bash invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=1, oom_score_adj=-900
Nov 9 01:02:37 TNAS1 kernel: CPU: 5 UID: 999 PID: 2782472 Comm: bash Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:37 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:37 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:37 TNAS1 kernel: Call Trace:
Nov 9 01:02:37 TNAS1 kernel:
Nov 9 01:02:37 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:37 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:37 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:37 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:37 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:37 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:37 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:37 TNAS1 kernel: get_free_pages_noprof+0x11/0x40
Nov 9 01:02:37 TNAS1 kernel: pgd_alloc+0x1f/0x1d0
Nov 9 01:02:37 TNAS1 kernel: mm_init+0x1b3/0x3c0
Nov 9 01:02:37 TNAS1 kernel: alloc_bprm+0xb1/0x3a0
Nov 9 01:02:37 TNAS1 kernel: do_execveat_common.isra.0+0x92/0x1e0
Nov 9 01:02:37 TNAS1 kernel: __x64_sys_execve+0x36/0x40
Nov 9 01:02:37 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:02:37 TNAS1 kernel: ? do_fault+0x281/0x4d0
Nov 9 01:02:37 TNAS1 kernel: ? __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:37 TNAS1 kernel: ? __count_memcg_events+0x53/0xf0
Nov 9 01:02:37 TNAS1 kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 9 01:02:37 TNAS1 kernel: ? handle_mm_fault+0xae/0x2d0
Nov 9 01:02:37 TNAS1 kernel: ? do_user_addr_fault+0x379/0x670
Nov 9 01:02:37 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:02:37 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:02:37 TNAS1 kernel: RIP: 0033:0x7fe37389ca97
Nov 9 01:02:37 TNAS1 kernel: Code: Unable to access opcode bytes at 0x7fe37389ca6d.
Nov 9 01:02:37 TNAS1 kernel: RSP: 002b:00007ffe09353a38 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
Nov 9 01:02:37 TNAS1 kernel: RAX: ffffffffffffffda RBX: 000055cbcc0ee4c0 RCX: 00007fe37389ca97
Nov 9 01:02:37 TNAS1 kernel: RDX: 000055cbcc0944d0 RSI: 000055cbcc096600 RDI: 000055cbcc0ee4c0
Nov 9 01:02:37 TNAS1 kernel: RBP: 0000000000000000 R08: 0000000000000007 R09: 000055cbcc10e6a0
Nov 9 01:02:37 TNAS1 kernel: R10: 6c3630914f812537 R11: 0000000000000246 R12: 00000000ffffffff
Nov 9 01:02:37 TNAS1 kernel: R13: 000055cbcc096600 R14: 000055cbcc0944d0 R15: 000055cbcc0ec960
Nov 9 01:02:37 TNAS1 kernel:
Nov 9 01:02:37 TNAS1 kernel: Mem-Info:
Nov 9 01:02:37 TNAS1 kernel: active_anon:65304 inactive_anon:7678100 isolated_anon:0
active_file:687 inactive_file:271 isolated_file:0
unevictable:1000 dirty:0 writeback:73
slab_reclaimable:57889 slab_unreclaimable:80813
mapped:1213 shmem:92804 pagetables:16429
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27152 free_pcp:0 free_cma:78
Nov 9 01:02:37 TNAS1 kernel: Node 0 active_anon:261216kB inactive_anon:30712400kB active_file:2748kB inactive_file:1084kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:4852kB dirty:0kB writeback:292kB shmem:371216kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11872kB pagetables:65716kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:37 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA32 free:6884kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480972kB active_file:0kB inactive_file:12kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:37 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:37 TNAS1 kernel: Node 0 Normal free:101696kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261212kB inactive_anon:27220432kB active_file:1656kB inactive_file:1100kB unevictable:4000kB writepending:292kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:312kB
Nov 9 01:02:37 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA32: 154*4kB (UME) 81*8kB (UME) 73*16kB (UME) 40*32kB (UME) 33*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 6976kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 Normal: 1526*4kB (UMEC) 652*8kB (UMEC) 1579*16kB (UMEC) 2015*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 101064kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:37 TNAS1 kernel: 93919 total pagecache pages
Nov 9 01:02:37 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:37 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:37 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:37 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:37 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:37 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:37 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:37 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:37 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:37 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:37 TNAS1 kernel: [ 728] 104 728 2091 462 256 206 0 65536 0 -900 dbus-daemon
Nov 9 01:02:37 TNAS1 kernel: [ 741] 0 741 11506 415 224 191 0 98304 0 -250 systemd-journal
Nov 9 01:02:37 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:37 TNAS1 kernel: [ 1630] 0 1630 8369760 7494645 7494506 75 64 60833792 0 -1000 asyncio_loop
Nov 9 01:02:37 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:37 TNAS1 kernel: [ 1681] 0 1681 485024 35365 35252 113 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:37 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:37 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:37 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:37 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:37 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:37 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:37 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:37 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:37 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:37 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:37 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:37 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:37 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:37 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:37 TNAS1 kernel: [ 3691] 0 3691 12416 529 224 305 0 94208 0 0 systemd-logind
Nov 9 01:02:37 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:37 TNAS1 kernel: [ 3708] 0 3708 523983 4154 4140 14 0 327680 0 -999 containerd
Nov 9 01:02:37 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:37 TNAS1 kernel: [ 3809] 122 3809 3089 196 121 75 0 49152 0 0 usbhid-ups
Nov 9 01:02:37 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:37 TNAS1 kernel: [ 3962] 122 3962 1695 136 96 40 0 57344 0 0 upsd
Nov 9 01:02:37 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:37 TNAS1 kernel: [ 4009] 122 4009 2127 186 96 90 0 53248 0 0 upsmon
Nov 9 01:02:37 TNAS1 kernel: [ 4132] 0 4132 4136 448 224 224 0 69632 0 0 systemd-machine
Nov 9 01:02:37 TNAS1 kernel: [ 5386] 105 5386 1781 155 96 59 0 49152 0 0 avahi-daemon
Nov 9 01:02:37 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:37 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:37 TNAS1 kernel: [ 6672] 0 6672 13921 5613 5472 141 0 139264 0 -1000 websockify
Nov 9 01:02:37 TNAS1 kernel: [ 6757] 999 6757 116708 29363 29218 81 64 475136 0 -900 netdata
Nov 9 01:02:37 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:37 TNAS1 kernel: [ 7021] 999 7021 22248 4846 4846 0 0 147456 0 -900 python.d.plugin
Nov 9 01:02:37 TNAS1 kernel: [2479756] 999 2479756 1113 202 160 42 0 49152 0 -900 bash
Nov 9 01:02:37 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:02:37 TNAS1 kernel: [2773557] 0 2773557 206757 12341 12274 67 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2773561] 0 2773561 207385 13048 12971 77 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2776151] 0 2776151 207108 13003 12886 117 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2777444] 0 2777444 206501 12407 12278 129 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2777446] 0 2777446 207155 12650 12561 89 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2782383] 0 2782383 351207 1112 672 440 0 282624 0 0 libvirtd
Nov 9 01:02:37 TNAS1 kernel: [2782448] 0 2782448 4009 558 192 366 0 73728 0 0 syslog-ng
Nov 9 01:02:37 TNAS1 kernel: [2782467] 0 2782467 1 0 0 0 0 12288 0 0 truenas_audit_h
Nov 9 01:02:37 TNAS1 kernel: [2782472] 999 2782472 1113 216 205 11 0 49152 0 -900 bash
Nov 9 01:02:37 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/system.slice/libvirtd.service,task=libvirtd,pid=2782383,uid=0
Nov 9 01:02:37 TNAS1 kernel: Out of memory: Killed process 2782383 (libvirtd) total-vm:1404828kB, anon-rss:2688kB, file-rss:1760kB, shmem-rss:0kB, UID:0 pgtables:276kB oom_score_adj:0
Nov 9 01:02:37 TNAS1 systemd[1]: getty@tty1.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:37 TNAS1 systemd[1]: getty@tty1.service: Failed with result 'signal'.
Nov 9 01:02:37 TNAS1 systemd[1]: libvirtd.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:37 TNAS1 systemd[1]: libvirtd.service: Failed with result 'signal'.
Nov 9 01:02:37 TNAS1 systemd[1]: system-getty.slice: A process of this unit has been killed by the OOM killer.
Nov 9 01:02:37 TNAS1 systemd[1]: Started tnaudit.service - TrueNAS Auditd Handler.
Nov 9 01:02:37 TNAS1 kernel: bash invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=-900
Nov 9 01:02:37 TNAS1 kernel: CPU: 7 UID: 999 PID: 2782474 Comm: bash Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:37 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:37 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:37 TNAS1 kernel: Call Trace:
Nov 9 01:02:37 TNAS1 kernel:
Nov 9 01:02:37 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:37 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:37 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:37 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:37 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:37 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:37 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:37 TNAS1 kernel: __pud_alloc+0x34/0x1d0
Nov 9 01:02:37 TNAS1 kernel: copy_page_range+0x2be/0x18f0
Nov 9 01:02:37 TNAS1 kernel: ? __memcg_slab_post_alloc_hook+0x212/0x3a0
Nov 9 01:02:37 TNAS1 kernel: copy_process+0x143d/0x26b0
Nov 9 01:02:37 TNAS1 kernel: kernel_clone+0xc2/0x430
Nov 9 01:02:37 TNAS1 kernel: ? __audit_filter_op+0xaf/0x110
Nov 9 01:02:37 TNAS1 kernel: __do_sys_clone+0x66/0x90
Nov 9 01:02:37 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:02:37 TNAS1 kernel: ? __count_memcg_events+0x53/0xf0
Nov 9 01:02:37 TNAS1 kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 9 01:02:37 TNAS1 kernel: ? handle_mm_fault+0xae/0x2d0
Nov 9 01:02:37 TNAS1 kernel: ? do_user_addr_fault+0x379/0x670
Nov 9 01:02:37 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:02:37 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:02:37 TNAS1 kernel: RIP: 0033:0x7fe37389c313
Nov 9 01:02:37 TNAS1 kernel: Code: 00 00 00 00 00 66 90 64 48 8b 04 25 10 00 00 00 45 31 c0 31 d2 31 f6 bf 11 00 20 01 4c 8d 90 d0 02 00 00 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 89 c2 85 c0 75 2c 64 48 8b 04 25 10 00 00
Nov 9 01:02:37 TNAS1 kernel: RSP: 002b:00007ffe093537f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Nov 9 01:02:37 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe37389c313
Nov 9 01:02:37 TNAS1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Nov 9 01:02:37 TNAS1 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00222a2422206f68
Nov 9 01:02:37 TNAS1 kernel: R10: 00007fe3737c5a10 R11: 0000000000000246 R12: 0000000000000001
Nov 9 01:02:37 TNAS1 kernel: R13: 00007ffe09353a30 R14: 000055cb8d949b08 R15: 0000000000000000
Nov 9 01:02:37 TNAS1 kernel:
Nov 9 01:02:37 TNAS1 kernel: Mem-Info:
Nov 9 01:02:37 TNAS1 kernel: active_anon:65304 inactive_anon:7677410 isolated_anon:0
active_file:504 inactive_file:797 isolated_file:0
unevictable:1000 dirty:73 writeback:73
slab_reclaimable:57889 slab_unreclaimable:80810
mapped:1377 shmem:92804 pagetables:16316
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27270 free_pcp:0 free_cma:252
Nov 9 01:02:37 TNAS1 kernel: Node 0 active_anon:261216kB inactive_anon:30709640kB active_file:2016kB inactive_file:3188kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:5508kB dirty:292kB writeback:292kB shmem:371216kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11584kB pagetables:65264kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:37 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA32 free:6724kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480788kB active_file:0kB inactive_file:4kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:37 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:37 TNAS1 kernel: Node 0 Normal free:102328kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261212kB inactive_anon:27217536kB active_file:1740kB inactive_file:2580kB unevictable:4000kB writepending:796kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:1008kB
Nov 9 01:02:37 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 DMA32: 169*4kB (UME) 85*8kB (UME) 74*16kB (UME) 41*32kB (UME) 33*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7116kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 Normal: 1720*4kB (UMEC) 586*8kB (UMEC) 1583*16kB (UMEC) 2015*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 101376kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:37 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:37 TNAS1 kernel: 94288 total pagecache pages
Nov 9 01:02:37 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:37 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:37 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:37 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:37 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:37 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:37 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:37 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:37 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:37 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:37 TNAS1 kernel: [ 728] 104 728 2091 462 256 206 0 65536 0 -900 dbus-daemon
Nov 9 01:02:37 TNAS1 kernel: [ 741] 0 741 11506 703 224 479 0 98304 0 -250 systemd-journal
Nov 9 01:02:37 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:37 TNAS1 kernel: [ 1630] 0 1630 8369760 7494938 7494506 368 64 60833792 0 -1000 asyncio_loop
Nov 9 01:02:37 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:37 TNAS1 kernel: [ 1681] 0 1681 485024 35365 35252 113 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:37 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:37 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:37 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:37 TNAS1 kernel: [ 3516] 0 3516 21731 160 94 66 0 57344 0 -1000 auditd
Nov 9 01:02:37 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:37 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:37 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:37 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:37 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:37 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:37 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:37 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:37 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:37 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:37 TNAS1 kernel: [ 3691] 0 3691 12416 529 224 305 0 94208 0 0 systemd-logind
Nov 9 01:02:37 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:37 TNAS1 kernel: [ 3708] 0 3708 523983 4186 4140 46 0 327680 0 -999 containerd
Nov 9 01:02:37 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:37 TNAS1 kernel: [ 3809] 122 3809 3089 196 121 75 0 49152 0 0 usbhid-ups
Nov 9 01:02:37 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:37 TNAS1 kernel: [ 3962] 122 3962 1695 136 96 40 0 57344 0 0 upsd
Nov 9 01:02:37 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:37 TNAS1 kernel: [ 4009] 122 4009 2127 186 96 90 0 53248 0 0 upsmon
Nov 9 01:02:37 TNAS1 kernel: [ 4132] 0 4132 4136 448 224 224 0 69632 0 0 systemd-machine
Nov 9 01:02:37 TNAS1 kernel: [ 5386] 105 5386 1781 191 96 95 0 49152 0 0 avahi-daemon
Nov 9 01:02:37 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:37 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:37 TNAS1 kernel: [ 6672] 0 6672 13921 5656 5472 184 0 139264 0 -1000 websockify
Nov 9 01:02:37 TNAS1 kernel: [ 6757] 999 6757 116708 29398 29218 116 64 475136 0 -900 netdata
Nov 9 01:02:37 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:37 TNAS1 kernel: [ 7021] 999 7021 22248 4846 4846 0 0 147456 0 -900 python.d.plugin
Nov 9 01:02:37 TNAS1 kernel: [2479756] 999 2479756 1113 373 160 213 0 49152 0 -900 bash
Nov 9 01:02:37 TNAS1 kernel: [2709452] 0 2709452 94214 32 32 0 0 86016 0 0 nscd
Nov 9 01:02:37 TNAS1 kernel: [2773557] 0 2773557 206757 12341 12274 67 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2773561] 0 2773561 207385 13048 12971 77 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2776151] 0 2776151 207108 13003 12886 117 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2777444] 0 2777444 206501 12407 12278 129 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2777446] 0 2777446 207155 12650 12561 89 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:37 TNAS1 kernel: [2782448] 0 2782448 4001 531 192 339 0 73728 0 0 syslog-ng
Nov 9 01:02:37 TNAS1 kernel: [2782467] 0 2782467 3078 569 192 377 0 57344 0 0 truenas_audit_h
Nov 9 01:02:37 TNAS1 kernel: [2782474] 999 2782474 1113 380 205 175 0 49152 0 -900 bash
Nov 9 01:02:37 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=netdata.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tnaudit.service,task=truenas_audit_h,pid=2782467,uid=0
Nov 9 01:02:37 TNAS1 kernel: Out of memory: Killed process 2782467 (truenas_audit_h) total-vm:12312kB, anon-rss:768kB, file-rss:1508kB, shmem-rss:0kB, UID:0 pgtables:56kB oom_score_adj:0
Nov 9 01:02:37 TNAS1 systemd[1]: tnaudit.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:37 TNAS1 systemd[1]: tnaudit.service: Failed with result 'signal'.
Nov 9 01:02:37 TNAS1 systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 3.
Nov 9 01:02:37 TNAS1 systemd[1]: Stopped libvirtd.service - Virtualization daemon.
Nov 9 01:02:37 TNAS1 systemd[1]: Starting libvirtd.service - Virtualization daemon...
Nov 9 01:02:39 TNAS1 systemd[1]: Starting tnaudit.service - TrueNAS Auditd Handler...
Nov 9 01:02:43 TNAS1 systemd[1]: getty@tty1.service: Scheduled restart job, restart counter is at 5.
Nov 9 01:02:43 TNAS1 kernel: (ndler.py) invoked oom-killer: gfp_mask=0x402dc2(GFP_KERNEL_ACCOUNT|__GFP_HIGHMEM|__GFP_NOWARN|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:02:43 TNAS1 kernel: CPU: 7 UID: 0 PID: 2782487 Comm: (ndler.py) Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:43 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:43 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:43 TNAS1 kernel: Call Trace:
Nov 9 01:02:43 TNAS1 kernel:
Nov 9 01:02:43 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:43 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:43 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:43 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:43 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:43 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:43 TNAS1 kernel: alloc_pages_bulk_noprof+0x3df/0x720
Nov 9 01:02:43 TNAS1 kernel: __vmalloc_node_range_noprof+0x24f/0x850
Nov 9 01:02:43 TNAS1 kernel: ? bpf_prog_alloc_no_stats+0x3d/0x240
Nov 9 01:02:43 TNAS1 kernel: ? __pfx_seccomp_check_filter+0x10/0x10
Nov 9 01:02:43 TNAS1 kernel: __vmalloc_node_noprof+0x4e/0x70
Nov 9 01:02:43 TNAS1 kernel: ? bpf_prog_alloc_no_stats+0x3d/0x240
Nov 9 01:02:43 TNAS1 kernel: bpf_prog_alloc_no_stats+0x3d/0x240
Nov 9 01:02:43 TNAS1 kernel: ? __pfx_seccomp_check_filter+0x10/0x10
Nov 9 01:02:43 TNAS1 kernel: bpf_prog_alloc+0x14/0xb0
Nov 9 01:02:43 TNAS1 kernel: bpf_prog_create_from_user+0x53/0x120
Nov 9 01:02:43 TNAS1 kernel: do_seccomp+0x12f/0xa90
Nov 9 01:02:43 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:02:43 TNAS1 kernel: ? do_user_addr_fault+0x379/0x670
Nov 9 01:02:43 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:02:43 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:02:43 TNAS1 kernel: RIP: 0033:0x7f15adcb7799
Nov 9 01:02:43 TNAS1 kernel: Code: 08 89 e8 5b 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 37 06 0d 00 f7 d8 64 89 01 48
Nov 9 01:02:43 TNAS1 kernel: RSP: 002b:00007fffc1aae028 EFLAGS: 00000246 ORIG_RAX: 000000000000013d
Nov 9 01:02:43 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f15adcb7799
Nov 9 01:02:43 TNAS1 kernel: RDX: 0000564cb2502b60 RSI: 0000000000000000 RDI: 0000000000000001
Nov 9 01:02:43 TNAS1 kernel: RBP: 0000564cb24853f0 R08: 0000000000000001 R09: 0000000700000002
Nov 9 01:02:43 TNAS1 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
Nov 9 01:02:43 TNAS1 kernel: R13: 0000000700000001 R14: 0000000000000001 R15: 000000000000001e
Nov 9 01:02:43 TNAS1 kernel:
Nov 9 01:02:43 TNAS1 kernel: Mem-Info:
Nov 9 01:02:43 TNAS1 kernel: active_anon:65304 inactive_anon:7678596 isolated_anon:0
active_file:101 inactive_file:367 isolated_file:0
unevictable:1000 dirty:0 writeback:102
slab_reclaimable:57889 slab_unreclaimable:80801
mapped:909 shmem:92803 pagetables:16352
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27332 free_pcp:3 free_cma:82
Nov 9 01:02:43 TNAS1 kernel: Node 0 active_anon:261216kB inactive_anon:30714384kB active_file:404kB inactive_file:1468kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:3636kB dirty:0kB writeback:408kB shmem:371212kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11552kB pagetables:65408kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:43 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:43 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:43 TNAS1 kernel: Node 0 DMA32 free:7116kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480752kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:43 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:43 TNAS1 kernel: Node 0 Normal free:102184kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261212kB inactive_anon:27222396kB active_file:64kB inactive_file:2096kB unevictable:4000kB writepending:408kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:328kB
Nov 9 01:02:43 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:43 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:43 TNAS1 kernel: Node 0 DMA32: 169*4kB (UME) 85*8kB (UME) 74*16kB (UME) 41*32kB (UME) 33*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7116kB
Nov 9 01:02:43 TNAS1 kernel: Node 0 Normal: 1975*4kB (UMEC) 625*8kB (UMEC) 1607*16kB (UMEC) 2016*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 103124kB
Nov 9 01:02:43 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:43 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:43 TNAS1 kernel: 93590 total pagecache pages
Nov 9 01:02:43 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:43 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:43 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:43 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:43 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:43 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:43 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:43 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:43 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:43 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:43 TNAS1 kernel: [ 728] 104 728 2091 381 256 125 0 65536 0 -900 dbus-daemon
Nov 9 01:02:43 TNAS1 kernel: [ 741] 0 741 11506 362 224 138 0 98304 0 -250 systemd-journal
Nov 9 01:02:43 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:43 TNAS1 kernel: [ 1630] 0 1630 8370272 7495289 7495018 207 64 60837888 0 -1000 asyncio_loop
Nov 9 01:02:43 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:43 TNAS1 kernel: [ 1681] 0 1681 485024 35521 35252 269 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:43 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:43 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:43 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:43 TNAS1 kernel: [ 3516] 0 3516 21731 162 94 68 0 57344 0 -1000 auditd
Nov 9 01:02:43 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:43 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:43 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:43 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:43 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:43 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:43 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:43 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:43 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:43 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:43 TNAS1 kernel: [ 3691] 0 3691 12416 320 224 96 0 94208 0 0 systemd-logind
Nov 9 01:02:43 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:43 TNAS1 kernel: [ 3708] 0 3708 523983 4154 4140 14 0 327680 0 -999 containerd
Nov 9 01:02:43 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:43 TNAS1 kernel: [ 3809] 122 3809 3089 200 121 79 0 49152 0 0 usbhid-ups
Nov 9 01:02:43 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:43 TNAS1 kernel: [ 3962] 122 3962 1695 256 96 160 0 57344 0 0 upsd
Nov 9 01:02:43 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:43 TNAS1 kernel: [ 4009] 122 4009 2127 186 96 90 0 53248 0 0 upsmon
Nov 9 01:02:43 TNAS1 kernel: [ 4132] 0 4132 4136 306 224 82 0 69632 0 0 systemd-machine
Nov 9 01:02:43 TNAS1 kernel: [ 5386] 105 5386 1781 213 96 117 0 49152 0 0 avahi-daemon
Nov 9 01:02:43 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:43 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:43 TNAS1 kernel: [ 6672] 0 6672 13921 5544 5472 72 0 139264 0 -1000 websockify
Nov 9 01:02:43 TNAS1 kernel: [ 6757] 999 6757 116708 29399 29218 117 64 475136 0 -900 netdata
Nov 9 01:02:43 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:43 TNAS1 kernel: [ 7021] 999 7021 22248 5159 4846 313 0 147456 0 -900 python.d.plugin
Nov 9 01:02:43 TNAS1 kernel: [2479756] 999 2479756 1113 312 160 152 0 49152 0 -900 bash
Nov 9 01:02:43 TNAS1 kernel: [2709452] 0 2709452 94214 63 32 31 0 86016 0 0 nscd
Nov 9 01:02:43 TNAS1 kernel: [2773557] 0 2773557 206757 12356 12274 82 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:43 TNAS1 kernel: [2773561] 0 2773561 207385 13050 12971 79 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:43 TNAS1 kernel: [2776151] 0 2776151 207108 12956 12886 70 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:43 TNAS1 kernel: [2777444] 0 2777444 206501 12365 12278 87 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:43 TNAS1 kernel: [2777446] 0 2777446 207155 12683 12561 122 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:43 TNAS1 kernel: [2782448] 0 2782448 5248 618 416 202 0 81920 0 0 syslog-ng
Nov 9 01:02:43 TNAS1 kernel: [2782480] 0 2782480 16875 336 160 176 0 102400 0 0 libvirtd
Nov 9 01:02:43 TNAS1 kernel: [2782487] 0 2782487 42617 1686 1470 216 0 94208 0 0 (ndler.py)
Nov 9 01:02:43 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=tnaudit.service,mems_allowed=0,global_oom,task_memcg=/system.slice/tnaudit.service,task=(ndler.py),pid=2782487,uid=0
Nov 9 01:02:43 TNAS1 kernel: Out of memory: Killed process 2782487 ((ndler.py)) total-vm:170468kB, anon-rss:5880kB, file-rss:864kB, shmem-rss:0kB, UID:0 pgtables:92kB oom_score_adj:0
Nov 9 01:02:43 TNAS1 systemd[1]: tnaudit.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:43 TNAS1 systemd[1]: tnaudit.service: Failed with result 'signal'.
Nov 9 01:02:43 TNAS1 systemd[1]: Failed to start tnaudit.service - TrueNAS Auditd Handler.
Nov 9 01:02:43 TNAS1 systemd[1]: Stopped getty@tty1.service - Getty on tty1.
Nov 9 01:02:43 TNAS1 kernel: syslog-ng invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:02:43 TNAS1 kernel: CPU: 7 UID: 0 PID: 2782448 Comm: syslog-ng Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:43 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:43 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:43 TNAS1 kernel: Call Trace:
Nov 9 01:02:43 TNAS1 kernel:
Nov 9 01:02:43 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:43 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:43 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:43 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:43 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:43 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:43 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:43 TNAS1 kernel: ? set_ptes.constprop.0+0x41/0x90
Nov 9 01:02:43 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:02:43 TNAS1 kernel: do_fault+0x240/0x4d0
Nov 9 01:02:43 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:43 TNAS1 kernel: ? audit_reset_context+0x232/0x300
Nov 9 01:02:43 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:02:43 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:02:43 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:02:43 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:02:43 TNAS1 kernel: RIP: 0033:0x7f8905f2ffa3
Nov 9 01:02:43 TNAS1 kernel: Code: e8 92 73 01 00 e9 74 fd ff ff 48 85 c0 0f 84 f0 08 00 00 48 8b 15 cd 42 04 00 48 89 94 24 f0 00 00 00 48 8b 8c 24 f0 00 00 00 <48> 39 08 0f 85 c5 08 00 00 4c 8b 3c 24 31 f6 4c 89 ff e8 d6 3c ff
Nov 9 01:02:43 TNAS1 kernel: RSP: 002b:00007ffc1e6a87a0 EFLAGS: 00010206
Nov 9 01:02:43 TNAS1 kernel: RAX: 00007f8902981000 RBX: 000055c94ce0f800 RCX: 48524848534b504c
Nov 9 01:02:43 TNAS1 kernel: RDX: 48524848534b504c RSI: 000055c94ce7c190 RDI: 000055c94ce7c0a0
Nov 9 01:02:43 TNAS1 kernel: RBP: 000055c94ce7bea0 R08: 00007ffc1e6a87d0 R09: 00007f8905f2f252
Nov 9 01:02:43 TNAS1 kernel: R10: 0000000000000001 R11: 0000000000000060 R12: 0000000000000001
Nov 9 01:02:43 TNAS1 kernel: R13: 0000000000000000 R14: 0000000000000016 R15: 000055c94ce8cd80
Nov 9 01:02:43 TNAS1 kernel:
Nov 9 01:02:43 TNAS1 kernel: Mem-Info:
Nov 9 01:02:43 TNAS1 kernel: active_anon:65304 inactive_anon:7678304 isolated_anon:0
active_file:702 inactive_file:581 isolated_file:0
unevictable:1000 dirty:73 writeback:102
slab_reclaimable:57889 slab_unreclaimable:80801
mapped:1658 shmem:92803 pagetables:16352
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:26769 free_pcp:0 free_cma:83
Nov 9 01:02:43 TNAS1 kernel: Node 0 active_anon:261216kB inactive_anon:30713216kB active_file:2808kB inactive_file:2324kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:6632kB dirty:292kB writeback:408kB shmem:371212kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11552kB pagetables:65408kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:43 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:43 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:43 TNAS1 kernel: Node 0 DMA32 free:7116kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480752kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:43 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:43 TNAS1 kernel: Node 0 Normal free:99932kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261212kB inactive_anon:27221388kB active_file:2748kB inactive_file:2168kB unevictable:4000kB writepending:408kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:804kB local_pcp:0kB free_cma:332kB
Nov 9 01:02:43 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:43 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:43 TNAS1 kernel: Node 0 DMA32: 178*4kB (UME) 85*8kB (UME) 74*16kB (UME) 41*32kB (UME) 33*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7152kB
Nov 9 01:02:43 TNAS1 kernel: Node 0 Normal: 1237*4kB (UMEC) 599*8kB (UMEC) 1604*16kB (UEC) 2016*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 99916kB
Nov 9 01:02:43 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:43 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:43 TNAS1 kernel: 94266 total pagecache pages
Nov 9 01:02:43 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:43 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:43 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:43 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:43 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:43 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:43 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:43 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:43 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:43 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:43 TNAS1 kernel: [ 728] 104 728 2091 415 256 159 0 65536 0 -900 dbus-daemon
Nov 9 01:02:43 TNAS1 kernel: [ 741] 0 741 11506 526 224 302 0 98304 0 -250 systemd-journal
Nov 9 01:02:43 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:43 TNAS1 kernel: [ 1630] 0 1630 8370272 7495590 7495114 412 64 60837888 0 -1000 asyncio_loop
Nov 9 01:02:43 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:43 TNAS1 kernel: [ 1681] 0 1681 485024 35425 35252 173 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:43 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:43 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:43 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:43 TNAS1 kernel: [ 3516] 0 3516 21731 162 94 68 0 57344 0 -1000 auditd
Nov 9 01:02:43 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:43 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:43 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:43 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:43 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:43 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:43 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:43 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:43 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:43 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:43 TNAS1 kernel: [ 3691] 0 3691 12416 447 224 223 0 94208 0 0 systemd-logind
Nov 9 01:02:43 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:43 TNAS1 kernel: [ 3708] 0 3708 523983 4186 4140 46 0 327680 0 -999 containerd
Nov 9 01:02:43 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:43 TNAS1 kernel: [ 3809] 122 3809 3089 232 121 111 0 49152 0 0 usbhid-ups
Nov 9 01:02:43 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:43 TNAS1 kernel: [ 3962] 122 3962 1695 289 96 193 0 57344 0 0 upsd
Nov 9 01:02:43 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:43 TNAS1 kernel: [ 4009] 122 4009 2127 186 96 90 0 53248 0 0 upsmon
Nov 9 01:02:43 TNAS1 kernel: [ 4132] 0 4132 4136 479 224 255 0 69632 0 0 systemd-machine
Nov 9 01:02:43 TNAS1 kernel: [ 5386] 105 5386 1781 213 96 117 0 49152 0 0 avahi-daemon
Nov 9 01:02:43 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:43 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:43 TNAS1 kernel: [ 6672] 0 6672 13921 5643 5472 171 0 139264 0 -1000 websockify
Nov 9 01:02:43 TNAS1 kernel: [ 6757] 999 6757 116708 29463 29218 181 64 475136 0 -900 netdata
Nov 9 01:02:43 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:43 TNAS1 kernel: [ 7021] 999 7021 22248 5197 4846 351 0 147456 0 -900 python.d.plugin
Nov 9 01:02:43 TNAS1 kernel: [2479756] 999 2479756 1113 412 160 252 0 49152 0 -900 bash
Nov 9 01:02:43 TNAS1 kernel: [2709452] 0 2709452 94214 63 32 31 0 86016 0 0 nscd
Nov 9 01:02:43 TNAS1 kernel: [2773557] 0 2773557 206757 12356 12274 82 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:43 TNAS1 kernel: [2773561] 0 2773561 207385 13050 12971 79 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:43 TNAS1 kernel: [2776151] 0 2776151 207108 12956 12886 70 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:43 TNAS1 kernel: [2777444] 0 2777444 206501 12365 12278 87 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:43 TNAS1 kernel: [2777446] 0 2777446 207155 12683 12561 122 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:43 TNAS1 kernel: [2782448] 0 2782448 15566 825 416 409 0 114688 0 0 syslog-ng
Nov 9 01:02:43 TNAS1 kernel: [2782480] 0 2782480 16875 497 256 241 0 102400 0 0 libvirtd
Nov 9 01:02:43 TNAS1 kernel: [2782491] 0 2782491 42617 1620 1470 150 0 94208 0 0 (cli)
Nov 9 01:02:43 TNAS1 kernel: [2782492] 999 2782492 1113 395 205 190 0 49152 0 -900 bash
Nov 9 01:02:43 TNAS1 kernel: [2782493] 999 2782493 1113 205 205 0 0 49152 0 -900 bash
Nov 9 01:02:43 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=syslog-ng.service,mems_allowed=0,global_oom,task_memcg=/init.scope,task=(cli),pid=2782491,uid=0
Nov 9 01:02:43 TNAS1 kernel: Out of memory: Killed process 2782491 ((cli)) total-vm:170468kB, anon-rss:5880kB, file-rss:600kB, shmem-rss:0kB, UID:0 pgtables:92kB oom_score_adj:0
Nov 9 01:02:44 TNAS1 systemd[1]: Started getty@tty1.service - Getty on tty1.
Nov 9 01:02:45 TNAS1 kernel: syslog-ng invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:02:45 TNAS1 kernel: CPU: 7 UID: 0 PID: 2782448 Comm: syslog-ng Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:45 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:45 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:45 TNAS1 kernel: Call Trace:
Nov 9 01:02:45 TNAS1 kernel:
Nov 9 01:02:45 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:45 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:45 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:45 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:45 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:45 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:45 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:45 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:02:45 TNAS1 kernel: do_fault+0x240/0x4d0
Nov 9 01:02:45 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:45 TNAS1 kernel: ? sched_clock_cpu+0xf/0x190
Nov 9 01:02:45 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:02:45 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:02:45 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:02:45 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:02:45 TNAS1 kernel: RIP: 0033:0x7f8905f1ec4e
Nov 9 01:02:45 TNAS1 kernel: Code: 18 fa ff ff 85 c0 79 c4 e9 f1 fe ff ff e8 da 3a fc ff 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec 08 48 85 ff 74 57 48 85 f6 74 72 <48> 8b 4e 08 b8 b6 ff ff ff 48 83 f9 0f 77 0b 48 83 c4 08 c3 66 0f
Nov 9 01:02:45 TNAS1 kernel: RSP: 002b:00007ffc1e6a8990 EFLAGS: 00010202
Nov 9 01:02:45 TNAS1 kernel: RAX: 0000000000000001 RBX: 000055c94ce023f0 RCX: 000000000008cc98
Nov 9 01:02:45 TNAS1 kernel: RDX: 0000000000000003 RSI: 00007f890341bc98 RDI: 000055c94ce023f0
Nov 9 01:02:45 TNAS1 kernel: RBP: 000000000008cc98 R08: 0000000000000010 R09: 000055c94ce02428
Nov 9 01:02:45 TNAS1 kernel: R10: 000000000000001c R11: 0000000000000000 R12: 00007ffc1e6a89f0
Nov 9 01:02:45 TNAS1 kernel: R13: 00007ffc1e6a89a0 R14: 0000000000000003 R15: 00007f890341bc98
Nov 9 01:02:45 TNAS1 kernel:
Nov 9 01:02:45 TNAS1 kernel: Mem-Info:
Nov 9 01:02:45 TNAS1 kernel: active_anon:65304 inactive_anon:7678501 isolated_anon:0
active_file:616 inactive_file:567 isolated_file:0
unevictable:1000 dirty:117 writeback:116
slab_reclaimable:57889 slab_unreclaimable:80797
mapped:1184 shmem:92802 pagetables:16358
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27283 free_pcp:0 free_cma:75
Nov 9 01:02:45 TNAS1 kernel: Node 0 active_anon:261216kB inactive_anon:30714004kB active_file:2464kB inactive_file:2268kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:4736kB dirty:468kB writeback:464kB shmem:371208kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11536kB pagetables:65432kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:45 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:45 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:45 TNAS1 kernel: Node 0 DMA32 free:7152kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480716kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:45 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:45 TNAS1 kernel: Node 0 Normal free:101952kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261212kB inactive_anon:27221760kB active_file:2044kB inactive_file:1204kB unevictable:4000kB writepending:932kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:300kB
Nov 9 01:02:45 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:45 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:45 TNAS1 kernel: Node 0 DMA32: 178*4kB (UME) 85*8kB (UME) 74*16kB (UME) 41*32kB (UME) 33*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7152kB
Nov 9 01:02:45 TNAS1 kernel: Node 0 Normal: 1607*4kB (UMEC) 590*8kB (UMEC) 1604*16kB (UEC) 2016*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 101324kB
Nov 9 01:02:45 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:45 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:45 TNAS1 kernel: 93828 total pagecache pages
Nov 9 01:02:45 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:45 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:45 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:45 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:45 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:45 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:45 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:45 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:45 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:45 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:45 TNAS1 kernel: [ 728] 104 728 2091 383 256 127 0 65536 0 -900 dbus-daemon
Nov 9 01:02:45 TNAS1 kernel: [ 741] 0 741 11506 596 224 372 0 98304 0 -250 systemd-journal
Nov 9 01:02:45 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:45 TNAS1 kernel: [ 1630] 0 1630 8370272 7495366 7495114 188 64 60837888 0 -1000 asyncio_loop
Nov 9 01:02:45 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:45 TNAS1 kernel: [ 1681] 0 1681 485024 35393 35252 141 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:45 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:45 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:45 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:45 TNAS1 kernel: [ 3516] 0 3516 21731 162 94 68 0 57344 0 -1000 auditd
Nov 9 01:02:45 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:45 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:45 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:45 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:45 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:45 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:45 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:45 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:45 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:45 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:45 TNAS1 kernel: [ 3691] 0 3691 12416 417 224 193 0 94208 0 0 systemd-logind
Nov 9 01:02:45 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:45 TNAS1 kernel: [ 3708] 0 3708 523983 4186 4140 46 0 327680 0 -999 containerd
Nov 9 01:02:45 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:45 TNAS1 kernel: [ 3809] 122 3809 3089 200 121 79 0 49152 0 0 usbhid-ups
Nov 9 01:02:45 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:45 TNAS1 kernel: [ 3962] 122 3962 1695 193 96 97 0 57344 0 0 upsd
Nov 9 01:02:45 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:45 TNAS1 kernel: [ 4009] 122 4009 2127 186 96 90 0 53248 0 0 upsmon
Nov 9 01:02:45 TNAS1 kernel: [ 4132] 0 4132 4136 422 224 198 0 69632 0 0 systemd-machine
Nov 9 01:02:45 TNAS1 kernel: [ 5386] 105 5386 1781 181 96 85 0 49152 0 0 avahi-daemon
Nov 9 01:02:45 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:45 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:45 TNAS1 kernel: [ 6672] 0 6672 13921 5579 5472 107 0 139264 0 -1000 websockify
Nov 9 01:02:45 TNAS1 kernel: [ 6757] 999 6757 116708 29463 29218 181 64 475136 0 -900 netdata
Nov 9 01:02:45 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:45 TNAS1 kernel: [ 7021] 999 7021 22248 5101 4846 255 0 147456 0 -900 python.d.plugin
Nov 9 01:02:45 TNAS1 kernel: [2479756] 999 2479756 1113 380 160 220 0 49152 0 -900 bash
Nov 9 01:02:45 TNAS1 kernel: [2709452] 0 2709452 94214 63 32 31 0 86016 0 0 nscd
Nov 9 01:02:45 TNAS1 kernel: [2773557] 0 2773557 206757 12356 12274 82 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:45 TNAS1 kernel: [2773561] 0 2773561 207385 13050 12971 79 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:45 TNAS1 kernel: [2776151] 0 2776151 207108 12956 12886 70 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:45 TNAS1 kernel: [2777444] 0 2777444 206501 12365 12278 87 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:45 TNAS1 kernel: [2777446] 0 2777446 207155 12683 12561 122 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:45 TNAS1 kernel: [2782448] 0 2782448 40629 710 416 294 0 217088 0 0 syslog-ng
Nov 9 01:02:45 TNAS1 kernel: [2782480] 0 2782480 16875 543 256 287 0 102400 0 0 libvirtd
Nov 9 01:02:45 TNAS1 kernel: [2782494] 999 2782494 1113 367 172 195 0 49152 0 -900 bash
Nov 9 01:02:45 TNAS1 kernel: [2782495] 999 2782495 1113 387 205 182 0 49152 0 -900 bash
Nov 9 01:02:45 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=syslog-ng.service,mems_allowed=0,global_oom,task_memcg=/system.slice/syslog-ng.service,task=syslog-ng,pid=2782448,uid=0
Nov 9 01:02:45 TNAS1 kernel: Out of memory: Killed process 2782448 (syslog-ng) total-vm:162516kB, anon-rss:1664kB, file-rss:1176kB, shmem-rss:0kB, UID:0 pgtables:212kB oom_score_adj:0
Nov 9 01:02:45 TNAS1 kernel: syslog-ng: page allocation failure: order:0, mode:0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=syslog-ng.service,mems_allowed=0
Nov 9 01:02:45 TNAS1 kernel: CPU: 3 UID: 0 PID: 2782448 Comm: syslog-ng Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:45 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:45 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:45 TNAS1 kernel: Call Trace:
Nov 9 01:02:45 TNAS1 kernel:
Nov 9 01:02:45 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:45 TNAS1 kernel: warn_alloc+0x164/0x1e0
Nov 9 01:02:45 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x688/0xdf0
Nov 9 01:02:45 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:45 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:45 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:02:45 TNAS1 kernel: do_fault+0x240/0x4d0
Nov 9 01:02:45 TNAS1 kernel: __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:45 TNAS1 kernel: ? sched_clock_cpu+0xf/0x190
Nov 9 01:02:45 TNAS1 kernel: handle_mm_fault+0x180/0x2d0
Nov 9 01:02:45 TNAS1 kernel: do_user_addr_fault+0x20c/0x670
Nov 9 01:02:45 TNAS1 kernel: exc_page_fault+0x76/0x190
Nov 9 01:02:45 TNAS1 kernel: asm_exc_page_fault+0x26/0x30
Nov 9 01:02:45 TNAS1 kernel: RIP: 0033:0x7f8905f1ec4e
Nov 9 01:02:45 TNAS1 kernel: Code: 18 fa ff ff 85 c0 79 c4 e9 f1 fe ff ff e8 da 3a fc ff 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec 08 48 85 ff 74 57 48 85 f6 74 72 <48> 8b 4e 08 b8 b6 ff ff ff 48 83 f9 0f 77 0b 48 83 c4 08 c3 66 0f
Nov 9 01:02:45 TNAS1 kernel: RSP: 002b:00007ffc1e6a8990 EFLAGS: 00010202
Nov 9 01:02:45 TNAS1 kernel: RAX: 0000000000000001 RBX: 000055c94ce023f0 RCX: 000000000008cc98
Nov 9 01:02:45 TNAS1 kernel: RDX: 0000000000000003 RSI: 00007f890341bc98 RDI: 000055c94ce023f0
Nov 9 01:02:45 TNAS1 kernel: RBP: 000000000008cc98 R08: 0000000000000010 R09: 000055c94ce02428
Nov 9 01:02:45 TNAS1 kernel: R10: 000000000000001c R11: 0000000000000000 R12: 00007ffc1e6a89f0
Nov 9 01:02:45 TNAS1 kernel: R13: 00007ffc1e6a89a0 R14: 0000000000000003 R15: 00007f890341bc98
Nov 9 01:02:45 TNAS1 kernel:
Nov 9 01:02:45 TNAS1 kernel: Mem-Info:
Nov 9 01:02:45 TNAS1 kernel: active_anon:65304 inactive_anon:7678501 isolated_anon:0
active_file:607 inactive_file:522 isolated_file:0
unevictable:1000 dirty:117 writeback:116
slab_reclaimable:57889 slab_unreclaimable:80797
mapped:1222 shmem:92802 pagetables:16358
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27283 free_pcp:0 free_cma:75
Nov 9 01:02:45 TNAS1 kernel: Node 0 active_anon:261216kB inactive_anon:30714004kB active_file:2428kB inactive_file:2088kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:4888kB dirty:468kB writeback:464kB shmem:371208kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11536kB pagetables:65432kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:45 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:45 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:45 TNAS1 kernel: Node 0 DMA32 free:7152kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480716kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:45 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:45 TNAS1 kernel: Node 0 Normal free:101952kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261212kB inactive_anon:27221760kB active_file:2008kB inactive_file:1024kB unevictable:4000kB writepending:932kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:300kB
Nov 9 01:02:45 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:45 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:45 TNAS1 kernel: Node 0 DMA32: 178*4kB (UME) 85*8kB (UME) 74*16kB (UME) 41*32kB (UME) 33*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7152kB
Nov 9 01:02:45 TNAS1 kernel: Node 0 Normal: 1635*4kB (UMEC) 592*8kB (UMEC) 1604*16kB (UEC) 2016*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 101452kB
Nov 9 01:02:45 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:45 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:45 TNAS1 kernel: 93866 total pagecache pages
Nov 9 01:02:45 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:45 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:45 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:45 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:45 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:45 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:45 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:45 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:45 TNAS1 kernel: systemd invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Nov 9 01:02:45 TNAS1 kernel: CPU: 6 UID: 0 PID: 1 Comm: systemd Tainted: P OE 6.12.15-production+truenas #1
Nov 9 01:02:45 TNAS1 kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Nov 9 01:02:45 TNAS1 kernel: Hardware name: Supermicro X10SLH-F/X10SLM+-F/X10SLH-F/X10SLM+-F, BIOS 3.4 01/21/2021
Nov 9 01:02:45 TNAS1 kernel: Call Trace:
Nov 9 01:02:45 TNAS1 kernel:
Nov 9 01:02:45 TNAS1 kernel: dump_stack_lvl+0x64/0x80
Nov 9 01:02:45 TNAS1 kernel: dump_header+0x43/0x160
Nov 9 01:02:45 TNAS1 kernel: oom_kill_process+0xfa/0x200
Nov 9 01:02:46 TNAS1 kernel: out_of_memory+0x257/0x520
Nov 9 01:02:46 TNAS1 kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Nov 9 01:02:46 TNAS1 kernel: __alloc_pages_noprof+0x30e/0x330
Nov 9 01:02:46 TNAS1 kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Nov 9 01:02:46 TNAS1 kernel: pte_alloc_one+0x1d/0x60
Nov 9 01:02:46 TNAS1 kernel: __pte_alloc+0x2a/0xb0
Nov 9 01:02:46 TNAS1 kernel: ? set_ptes.constprop.0+0x41/0x90
Nov 9 01:02:46 TNAS1 kernel: copy_page_range+0x10e1/0x18f0
Nov 9 01:02:46 TNAS1 kernel: copy_process+0x143d/0x26b0
Nov 9 01:02:46 TNAS1 kernel: kernel_clone+0xc2/0x430
Nov 9 01:02:46 TNAS1 kernel: __do_sys_clone+0x66/0x90
Nov 9 01:02:46 TNAS1 kernel: do_syscall_64+0x82/0x190
Nov 9 01:02:46 TNAS1 kernel: ? do_fault+0x281/0x4d0
Nov 9 01:02:46 TNAS1 kernel: ? __handle_mm_fault+0x7b8/0xfd0
Nov 9 01:02:46 TNAS1 kernel: ? do_syscall_64+0x8e/0x190
Nov 9 01:02:46 TNAS1 kernel: ? __count_memcg_events+0x53/0xf0
Nov 9 01:02:46 TNAS1 kernel: ? count_memcg_events.constprop.0+0x1a/0x30
Nov 9 01:02:46 TNAS1 kernel: ? handle_mm_fault+0xae/0x2d0
Nov 9 01:02:46 TNAS1 kernel: ? do_user_addr_fault+0x17f/0x670
Nov 9 01:02:46 TNAS1 kernel: ? exc_page_fault+0x76/0x190
Nov 9 01:02:46 TNAS1 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 9 01:02:46 TNAS1 kernel: RIP: 0033:0x7f15adc8a313
Nov 9 01:02:46 TNAS1 kernel: Code: 00 00 00 00 00 66 90 64 48 8b 04 25 10 00 00 00 45 31 c0 31 d2 31 f6 bf 11 00 20 01 4c 8d 90 d0 02 00 00 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 89 c2 85 c0 75 2c 64 48 8b 04 25 10 00 00
Nov 9 01:02:46 TNAS1 kernel: RSP: 002b:00007fffc1aae388 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Nov 9 01:02:46 TNAS1 kernel: RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f15adc8a313
Nov 9 01:02:46 TNAS1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Nov 9 01:02:46 TNAS1 kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 00007f15ae21e7a0
Nov 9 01:02:46 TNAS1 kernel: R10: 00007f15ad148c10 R11: 0000000000000246 R12: 0000000000000000
Nov 9 01:02:46 TNAS1 kernel: R13: 00007fffc1aae6c0 R14: 00007fffc1aae390 R15: 0000000000000000
Nov 9 01:02:46 TNAS1 kernel:
Nov 9 01:02:46 TNAS1 kernel: Mem-Info:
Nov 9 01:02:46 TNAS1 kernel: active_anon:65303 inactive_anon:7678057 isolated_anon:0
active_file:428 inactive_file:661 isolated_file:0
unevictable:1000 dirty:108 writeback:125
slab_reclaimable:57889 slab_unreclaimable:80796
mapped:1480 shmem:92802 pagetables:16333
sec_pagetables:265 bounce:0
kernel_misc_reclaimable:0
free:27677 free_pcp:0 free_cma:205
Nov 9 01:02:46 TNAS1 kernel: Node 0 active_anon:261212kB inactive_anon:30712228kB active_file:1712kB inactive_file:2644kB unevictable:4000kB isolated(anon):0kB isolated(file):0kB mapped:5920kB dirty:432kB writeback:500kB shmem:371208kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:786432kB writeback_tmp:0kB kernel_stack:11552kB pagetables:65332kB sec_pagetables:1060kB all_unreclaimable? no
Nov 9 01:02:46 TNAS1 kernel: Node 0 DMA free:28kB boost:0kB min:28kB low:40kB high:52kB reserved_highatomic:0KB active_anon:0kB inactive_anon:11236kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:46 TNAS1 kernel: lowmem_reserve[]: 0 3387 31964 0 0
Nov 9 01:02:46 TNAS1 kernel: Node 0 DMA32 free:7152kB boost:0kB min:7156kB low:10624kB high:14092kB reserved_highatomic:0KB active_anon:4kB inactive_anon:3480620kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3611128kB managed:3545592kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 9 01:02:46 TNAS1 kernel: lowmem_reserve[]: 0 0 28576 0 0
Nov 9 01:02:46 TNAS1 kernel: Node 0 Normal free:103528kB boost:178368kB min:238760kB low:268020kB high:297280kB reserved_highatomic:4096KB active_anon:261208kB inactive_anon:27220372kB active_file:544kB inactive_file:2688kB unevictable:4000kB writepending:932kB present:29884416kB managed:29269672kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:820kB
Nov 9 01:02:46 TNAS1 kernel: lowmem_reserve[]: 0 0 0 0 0
Nov 9 01:02:46 TNAS1 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 28kB
Nov 9 01:02:46 TNAS1 kernel: Node 0 DMA32: 198*4kB (UME) 85*8kB (UME) 73*16kB (UME) 40*32kB (UME) 34*64kB (UME) 3*128kB (ME) 1*256kB (M) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 7248kB
Nov 9 01:02:46 TNAS1 kernel: Node 0 Normal: 1605*4kB (UMEC) 638*8kB (UMEC) 1611*16kB (UMEC) 2016*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 101812kB
Nov 9 01:02:46 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 9 01:02:46 TNAS1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 9 01:02:46 TNAS1 kernel: 94055 total pagecache pages
Nov 9 01:02:46 TNAS1 kernel: 0 pages in swap cache
Nov 9 01:02:46 TNAS1 kernel: Free swap = 0kB
Nov 9 01:02:46 TNAS1 kernel: Total swap = 0kB
Nov 9 01:02:46 TNAS1 kernel: 8377883 pages RAM
Nov 9 01:02:46 TNAS1 kernel: 0 pages HighMem/MovableOnly
Nov 9 01:02:46 TNAS1 kernel: 170227 pages reserved
Nov 9 01:02:46 TNAS1 kernel: 65536 pages cma reserved
Nov 9 01:02:46 TNAS1 kernel: 0 pages hwpoisoned
Nov 9 01:02:46 TNAS1 kernel: Tasks state (memory values in pages):
Nov 9 01:02:46 TNAS1 kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Nov 9 01:02:46 TNAS1 kernel: [ 728] 104 728 2091 351 256 95 0 65536 0 -900 dbus-daemon
Nov 9 01:02:46 TNAS1 kernel: [ 741] 0 741 11506 667 224 443 0 98304 0 -250 systemd-journal
Nov 9 01:02:46 TNAS1 kernel: [ 763] 0 763 6768 559 512 47 0 77824 0 -1000 systemd-udevd
Nov 9 01:02:46 TNAS1 kernel: [ 1630] 0 1630 8370272 7495308 7495114 130 64 60837888 0 -1000 asyncio_loop
Nov 9 01:02:46 TNAS1 kernel: [ 1648] 0 1648 4194 1618 1568 50 0 69632 0 -1000 python3
Nov 9 01:02:46 TNAS1 kernel: [ 1681] 0 1681 485024 35361 35252 109 0 651264 0 -1000 middlewared (ze
Nov 9 01:02:46 TNAS1 kernel: [ 1692] 0 1692 9218 4450 4343 107 0 114688 0 -1000 python3
Nov 9 01:02:46 TNAS1 kernel: [ 3513] 0 3513 711 207 66 141 0 45056 0 0 rpc.idmapd
Nov 9 01:02:46 TNAS1 kernel: [ 3515] 0 3515 1302 218 97 121 0 49152 0 0 nfsdcld
Nov 9 01:02:46 TNAS1 kernel: [ 3516] 0 3516 21731 162 94 68 0 57344 0 -1000 auditd
Nov 9 01:02:46 TNAS1 kernel: [ 3593] 106 3593 1970 177 96 81 0 57344 0 0 rpcbind
Nov 9 01:02:46 TNAS1 kernel: [ 3594] 107 3594 1134 163 73 90 0 49152 0 0 rpc.statd
Nov 9 01:02:46 TNAS1 kernel: [ 3596] 0 3596 1258 259 99 160 0 57344 0 0 rpc.mountd
Nov 9 01:02:46 TNAS1 kernel: [ 3597] 0 3597 1258 238 131 107 0 57344 0 0 rpc.mountd
Nov 9 01:02:46 TNAS1 kernel: [ 3598] 0 3598 1258 194 99 95 0 57344 0 0 rpc.mountd
Nov 9 01:02:46 TNAS1 kernel: [ 3599] 0 3599 5605 368 320 48 0 86016 0 0 certmonger
Nov 9 01:02:46 TNAS1 kernel: [ 3618] 0 3618 28122 194 96 98 0 73728 0 0 gssproxy
Nov 9 01:02:46 TNAS1 kernel: [ 3629] 0 3629 38187 144 64 80 0 65536 0 -1000 lxcfs
Nov 9 01:02:46 TNAS1 kernel: [ 3635] 131 3635 4715 192 138 54 0 61440 0 0 chronyd
Nov 9 01:02:46 TNAS1 kernel: [ 3648] 131 3648 2686 250 118 132 0 61440 0 0 chronyd
Nov 9 01:02:46 TNAS1 kernel: [ 3691] 0 3691 12416 459 224 235 0 94208 0 0 systemd-logind
Nov 9 01:02:46 TNAS1 kernel: [ 3695] 0 3695 60178 320 256 64 0 86016 0 0 zed
Nov 9 01:02:46 TNAS1 kernel: [ 3708] 0 3708 523983 4218 4140 78 0 327680 0 -999 containerd
Nov 9 01:02:46 TNAS1 kernel: [ 3783] 0 3783 3857 411 288 123 0 73728 0 -1000 sshd
Nov 9 01:02:46 TNAS1 kernel: [ 3809] 122 3809 3089 200 121 79 0 49152 0 0 usbhid-ups
Nov 9 01:02:46 TNAS1 kernel: [ 3891] 0 3891 1005 100 32 68 0 45056 0 0 cron
Nov 9 01:02:46 TNAS1 kernel: [ 3962] 122 3962 1695 193 96 97 0 57344 0 0 upsd
Nov 9 01:02:46 TNAS1 kernel: [ 3964] 0 3964 1223 150 32 118 0 45056 0 0 upsmon
Nov 9 01:02:46 TNAS1 kernel: [ 4009] 122 4009 2127 186 96 90 0 53248 0 0 upsmon
Nov 9 01:02:46 TNAS1 kernel: [ 4132] 0 4132 4136 394 224 170 0 69632 0 0 systemd-machine
Nov 9 01:02:46 TNAS1 kernel: [ 5386] 105 5386 1781 149 96 53 0 49152 0 0 avahi-daemon
Nov 9 01:02:46 TNAS1 kernel: [ 5450] 105 5450 1722 181 73 108 0 49152 0 0 avahi-daemon
Nov 9 01:02:46 TNAS1 kernel: [ 6054] 0 6054 16985 514 448 66 0 114688 0 -900 virtlogd
Nov 9 01:02:46 TNAS1 kernel: [ 6672] 0 6672 13921 5646 5472 174 0 139264 0 -1000 websockify
Nov 9 01:02:46 TNAS1 kernel: [ 6757] 999 6757 116708 29367 29218 85 64 475136 0 -900 netdata
Nov 9 01:02:46 TNAS1 kernel: [ 6759] 999 6759 11686 294 192 102 0 69632 0 -900 netdata
Nov 9 01:02:46 TNAS1 kernel: [ 7021] 999 7021 22248 5170 4846 324 0 147456 0 -900 python.d.plugin
Nov 9 01:02:46 TNAS1 kernel: [2479756] 999 2479756 1113 220 160 60 0 49152 0 -900 bash
Nov 9 01:02:46 TNAS1 kernel: [2709452] 0 2709452 94214 63 32 31 0 86016 0 0 nscd
Nov 9 01:02:46 TNAS1 kernel: [2773557] 0 2773557 206757 12356 12274 82 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:46 TNAS1 kernel: [2773561] 0 2773561 207385 13050 12971 79 0 315392 0 -1000 middlewared (wo
Nov 9 01:02:46 TNAS1 kernel: [2776151] 0 2776151 207108 12956 12886 70 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:46 TNAS1 kernel: [2777444] 0 2777444 206501 12365 12278 87 0 299008 0 -1000 middlewared (wo
Nov 9 01:02:46 TNAS1 kernel: [2777446] 0 2777446 207155 12683 12561 122 0 303104 0 -1000 middlewared (wo
Nov 9 01:02:46 TNAS1 kernel: [2782480] 0 2782480 16875 715 256 459 0 102400 0 0 libvirtd
Nov 9 01:02:46 TNAS1 kernel: [2782494] 999 2782494 1113 271 172 99 0 49152 0 -900 bash
Nov 9 01:02:46 TNAS1 kernel: [2782495] 999 2782495 623 131 0 131 0 45056 0 -900 timeout
Nov 9 01:02:46 TNAS1 kernel: [2782496] 999 2782496 1191 170 32 138 0 49152 0 -900 upsc
Nov 9 01:02:46 TNAS1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/libvirtd.service,task=libvirtd,pid=2782480,uid=0
Nov 9 01:02:46 TNAS1 kernel: Out of memory: Killed process 2782480 (libvirtd) total-vm:67500kB, anon-rss:1024kB, file-rss:1836kB, shmem-rss:0kB, UID:0 pgtables:100kB oom_score_adj:0
Nov 9 01:02:46 TNAS1 systemd[1]: Starting tnaudit.service - TrueNAS Auditd Handler...
Nov 9 01:02:46 TNAS1 systemd[1]: init.scope: A process of this unit has been killed by the OOM killer.
Nov 9 01:02:46 TNAS1 systemd[1]: syslog-ng.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:46 TNAS1 systemd[1]: syslog-ng.service: Failed with result 'signal'.
Nov 9 01:02:46 TNAS1 systemd[1]: Failed to start syslog-ng.service - System Logger Daemon.
Nov 9 01:02:46 TNAS1 systemd[1]: libvirtd.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:46 TNAS1 systemd[1]: libvirtd.service: Failed with result 'signal'.
Nov 9 01:02:46 TNAS1 systemd[1]: Failed to start libvirtd.service - Virtualization daemon.
Nov 9 01:02:46 TNAS1 systemd[1]: getty@tty1.service: Main process exited, code=killed, status=9/KILL
Nov 9 01:02:46 TNAS1 systemd[1]: getty@tty1.service: Failed with result 'signal'.
Nov 9 01:02:46 TNAS1 systemd[1]: Started tnaudit.service - TrueNAS Auditd Handler.
Nov 9 01:02:47 TNAS1 systemd[1]: syslog-ng.service: Scheduled restart job, restart counter is at 4.
Nov 9 01:02:47 TNAS1 systemd[1]: libvirtd.service: Scheduled restart job, restart counter is at 4.
Nov 9 01:02:47 TNAS1 systemd[1]: Stopped libvirtd.service - Virtualization daemon.
Nov 9 01:02:47 TNAS1 systemd[1]: Starting libvirtd.service - Virtualization daemon...
Nov 9 01:02:47 TNAS1 systemd[1]: Stopped syslog-ng.service - System Logger Daemon.
Nov 9 01:02:47 TNAS1 systemd[1]: Starting syslog-ng.service - System Logger Daemon...
Nov 9 01:02:47 TNAS1 systemd[1]: Started syslog-ng.service - System Logger Daemon.
Nov 9 01:02:47 TNAS1 systemd[1]: Started libvirtd.service - Virtualization daemon.
Nov 9 01:02:50 TNAS1 middlewared[1630]: libvirt: XML-RPC error : internal error: client socket is closed
Nov 9 01:02:50 TNAS1 middlewared[1630]: libvirt: QEMU Driver error : Domain not found: no domain with matching name '2_NixVM'
Nov 9 01:02:50 TNAS1 middlewared[1630]: libvirt: XML-RPC error : internal error: client socket is closed
Nov 9 01:02:51 TNAS1 systemd[1]: getty@tty1.service: Scheduled restart job, restart counter is at 6.
Nov 9 01:02:51 TNAS1 systemd[1]: Stopped getty@tty1.service - Getty on tty1.
Nov 9 01:02:51 TNAS1 systemd[1]: Started getty@tty1.service - Getty on tty1.
Nov 9 01:02:57 TNAS1 middlewared[1630]: libvirt: XML-RPC error : internal error: client socket is closed
Nov 9 01:02:57 TNAS1 middlewared[1630]: libvirt: XML-RPC error : internal error: client socket is closed
Nov 9 01:03:14 TNAS1 systemd[1]: Received SIGRTMIN+21 from PID 2782643 (cli).
Nov 9 01:04:54 TNAS1 nscd[2785154]: 2785154 monitoring file `/etc/hosts` (1)
Nov 9 01:04:54 TNAS1 nscd[2785154]: 2785154 monitoring directory `/etc` (2)
Nov 9 01:04:54 TNAS1 nscd[2785154]: 2785154 monitoring file `/etc/resolv.conf` (3)
Nov 9 01:04:54 TNAS1 nscd[2785154]: 2785154 monitoring directory `/etc` (2)
Nov 9 01:04:54 TNAS1 nscd[2785154]: 2785154 monitoring file `/etc/nsswitch.conf` (4)
Nov 9 01:04:54 TNAS1 nscd[2785154]: 2785154 monitoring directory `/etc` (2)
Nov 9 01:07:54 TNAS1 certmonger[2788864]: 2025-11-09 01:07:54 [2788864] Running enrollment/cadata helper "/usr/lib/certmonger/ipa-submit".
Nov 9 01:07:54 TNAS1 certmonger[2788865]: 2025-11-09 01:07:54 [2788865] Running enrollment/cadata helper "/usr/lib/certmonger/dogtag-ipa-renew-agent-submit".