Hey there, I have 64 GB of RAM in my NAS and have an Incus Instance running on it with 32 GB RAM assigned. Aside from that I have three Docker containers running which only use around 3 GB of RAM combined.
Still, my Instance got OOM killed this morning for whatever reason:
[1117055.604412] runc:[2:INIT] invoked oom-killer: gfp_mask=0x42cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP), order=2, oom_score_adj=0
[1117055.604424] CPU: 6 UID: 0 PID: 4031130 Comm: runc:[2:INIT] Tainted: P OE 6.12.15-production+truenas #1
[1117055.604427] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[1117055.604428] Hardware name: ASRock H610M-HDV/H610M-HDV, BIOS 11.01 03/29/2023
[1117055.604429] Call Trace:
[1117055.604431] <TASK>
[1117055.604433] dump_stack_lvl+0x64/0x80
[1117055.604437] dump_header+0x43/0x160
[1117055.604440] oom_kill_process+0xfa/0x200
[1117055.604441] out_of_memory+0x257/0x520
[1117055.604443] __alloc_pages_slowpath.constprop.0+0x696/0xdf0
[1117055.604447] __alloc_pages_noprof+0x30e/0x330
[1117055.604450] ___kmalloc_large_node+0x80/0x110
[1117055.604452] __kmalloc_large_node_noprof+0x1d/0xa0
[1117055.604453] ? __pfx_seccomp_check_filter+0x10/0x10
[1117055.604455] __kmalloc_node_track_caller_noprof+0x331/0x3f0
[1117055.604457] ? bpf_prepare_filter+0x35c/0x510
[1117055.604460] ? __pfx_seccomp_check_filter+0x10/0x10
[1117055.604461] ? kmemdup_noprof+0x20/0x50
[1117055.604463] kmemdup_noprof+0x20/0x50
[1117055.604466] bpf_prepare_filter+0x35c/0x510
[1117055.604468] ? __pfx_seccomp_check_filter+0x10/0x10
[1117055.604469] bpf_prog_create_from_user+0xc1/0x120
[1117055.604471] do_seccomp+0x12f/0xa90
[1117055.604473] do_syscall_64+0x82/0x190
[1117055.604476] ? handle_mm_fault+0xae/0x2d0
[1117055.604478] ? do_user_addr_fault+0x17f/0x670
[1117055.604481] ? exc_page_fault+0x76/0x190
[1117055.604483] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[1117055.604486] RIP: 0033:0x55ea3fa4d92e
[1117055.604488] Code: 48 83 ec 38 e8 13 00 00 00 48 83 c4 38 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc cc 49 89 f2 48 89 fa 48 89 ce 48 89 df 0f 05 <48> 3d 01 f0 ff ff 76 15 48 f7 d8 48 89 c1 48 c7 c0 ff ff ff ff 48
[1117055.604490] RSP: 002b:000000c00011b738 EFLAGS: 00000202 ORIG_RAX: 000000000000013d
[1117055.604492] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 000055ea3fa4d92e
[1117055.604493] RDX: 000000c00011b840 RSI: 0000000000000004 RDI: 0000000000000001
[1117055.604493] RBP: 000000c00011b778 R08: 0000000000000000 R09: 0000000000000000
[1117055.604494] R10: 0000000000000000 R11: 0000000000000202 R12: 000000c00011b880
[1117055.604495] R13: 0000000000000000 R14: 000000c0000061c0 R15: 00000000000001b8
[1117055.604497] </TASK>
[1117055.604498] Mem-Info:
[1117055.604536] active_anon:8395789 inactive_anon:1183327 isolated_anon:0
active_file:0 inactive_file:7 isolated_file:18
unevictable:2630 dirty:0 writeback:5
slab_reclaimable:313867 slab_unreclaimable:1664407
mapped:8414688 shmem:8419097 pagetables:22313
sec_pagetables:16838 bounce:0
kernel_misc_reclaimable:0
free:576601 free_pcp:4897 free_cma:46
[1117055.604545] Node 0 active_anon:33583156kB inactive_anon:4733308kB active_file:0kB inactive_file:28kB unevictable:10520kB isolated(anon):0kB isolated(file):72kB mapped:33658752kB dirty:0kB writeback:20kB shmem:33676388kB shmem_thp:8192kB shmem_pmdmapped:0kB anon_thp:1902592kB writeback_tmp:0kB kernel_stack:16384kB pagetables:89252kB sec_pagetables:67352kB all_unreclaimable? no
[1117055.604553] Node 0 DMA free:1444kB boost:0kB min:16kB low:28kB high:40kB reserved_highatomic:0KB active_anon:2100kB inactive_anon:3624kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[1117055.604561] lowmem_reserve[]: 0 475 63060 0 0
[1117055.604565] Node 0 DMA32 free:114880kB boost:0kB min:508kB low:992kB high:1476kB reserved_highatomic:0KB active_anon:92596kB inactive_anon:343948kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:628600kB managed:563056kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[1117055.604572] lowmem_reserve[]: 0 0 62584 0 0
[1117055.604576] Node 0 Normal free:2190080kB boost:292836kB min:359892kB low:423976kB high:488060kB reserved_highatomic:8192KB active_anon:33488460kB inactive_anon:4385736kB active_file:0kB inactive_file:1560kB unevictable:10520kB writepending:20kB present:65265664kB managed:64093844kB mlocked:0kB bounce:0kB free_pcp:19660kB local_pcp:0kB free_cma:184kB
[1117055.604584] lowmem_reserve[]: 0 0 0 0 0
[1117055.604587] Node 0 DMA: 9*4kB (M) 4*8kB (M) 4*16kB (M) 3*32kB (M) 1*64kB (M) 1*128kB (M) 0*256kB 0*512kB 1*1024kB (U) 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 1444kB
[1117055.604601] Node 0 DMA32: 197*4kB (ME) 67*8kB (ME) 41*16kB (ME) 78*32kB (UME) 66*64kB (UME) 69*128kB (UE) 66*256kB (UE) 61*512kB (UE) 48*1024kB (UE) 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 114812kB
[1117055.604618] Node 0 Normal: 307659*4kB (UMEC) 119733*8kB (UMEC) 20*16kB (C) 19*32kB (C) 3*64kB (C) 1*128kB (C) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 2189748kB
[1117055.604632] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[1117055.604634] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[1117055.604637] 8419495 total pagecache pages
[1117055.604639] 0 pages in swap cache
[1117055.604640] Free swap = 0kB
[1117055.604641] Total swap = 0kB
[1117055.604642] 16477564 pages RAM
[1117055.604643] 0 pages HighMem/MovableOnly
[1117055.604644] 309499 pages reserved
[1117055.604646] 65536 pages cma reserved
[1117055.604647] 0 pages hwpoisoned
[1117055.604648] Tasks state (memory values in pages):
[1117055.604648] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[1117055.604669] [ 809] 104 809 2166 565 344 221 0 65536 0 -900 dbus-daemon
[1117055.604672] [ 821] 0 821 11541 285 224 61 0 94208 0 -250 systemd-journal
[1117055.604675] [ 836] 0 836 6767 608 480 128 0 77824 0 -1000 systemd-udevd
[1117055.604679] [ 1700] 0 1700 953601 257196 257133 63 0 2605056 0 -1000 asyncio_loop
[1117055.604681] [ 1715] 0 1715 4194 1599 1472 127 0 65536 0 -1000 python3
[1117055.604683] [ 1758] 0 1758 9211 4519 4378 141 0 114688 0 -1000 python3
[1117055.604690] [ 3098] 0 3098 21730 160 98 62 0 61440 0 -1000 auditd
[1117055.604692] [ 3122] 0 3122 27525 4131 4049 82 0 122880 0 0 truenas_audit_h
[1117055.604694] [ 3365] 0 3365 5604 468 352 116 0 86016 0 0 certmonger
[1117055.604696] [ 3374] 0 3374 11738 196 96 100 0 69632 0 0 gssproxy
[1117055.604698] [ 3387] 0 3387 38188 104 0 104 0 61440 0 -1000 lxcfs
[1117055.604700] [ 3401] 0 3401 3010 517 384 133 0 65536 0 0 smartd
[1117055.604702] [ 3407] 0 3407 269034 2145 2077 68 0 438272 0 0 syslog-ng
[1117055.604704] [ 3413] 0 3413 6269 320 224 96 0 69632 0 0 systemd-logind
[1117055.604707] [ 3417] 0 3417 60178 342 256 86 0 86016 0 0 zed
[1117055.604709] [ 3419] 0 3419 579722 5459 5428 31 0 376832 0 -999 containerd
[1117055.604710] [ 3433] 0 3433 1005 127 32 95 0 45056 0 0 cron
[1117055.604712] [ 3458] 131 3458 4715 167 138 29 0 61440 0 0 chronyd
[1117055.604714] [ 3463] 131 3463 2686 221 116 105 0 61440 0 0 chronyd
[1117055.604715] [ 3483] 0 3483 91720 13495 13421 74 0 221184 0 0 cli
[1117055.604718] [ 3638] 0 3638 7048 444 366 78 0 65536 0 0 nginx
[1117055.604720] [ 3643] 33 3643 7324 739 623 116 0 65536 0 0 nginx
[1117055.604725] [ 3713] 0 3713 696 83 0 83 0 40960 0 0 iscsi-scstd
[1117055.604730] [ 4643] 999 4643 124372 36835 36711 92 32 528384 0 -900 netdata
[1117055.604732] [ 4648] 999 4648 11686 342 192 150 0 73728 0 -900 netdata
[1117055.604733] [ 5615] 999 5615 22275 4872 4800 72 0 139264 0 -900 python.d.plugin
[1117055.604735] [ 5716] 105 5716 1788 186 96 90 0 61440 0 0 avahi-daemon
[1117055.604737] [ 5721] 105 5721 1722 162 73 89 0 57344 0 0 avahi-daemon
[1117055.604738] [ 5785] 1 5785 8086 3978 3872 106 0 102400 0 0 wsdd.py
[1117055.604740] [ 5807] 0 5807 17600 663 544 119 0 122880 0 0 nmbd
[1117055.604741] [ 5952] 0 5952 18964 794 640 58 96 126976 0 0 winbindd
[1117055.604743] [ 5983] 0 5983 18978 785 638 112 35 126976 0 0 wb[TRUENAS]
[1117055.604745] [ 5988] 0 5988 20141 1492 640 127 725 147456 0 0 smbd
[1117055.604746] [ 6024] 0 6024 19608 682 620 62 0 135168 0 0 smbd-notifyd
[1117055.604748] [ 6025] 0 6025 19612 725 620 105 0 139264 0 0 smbd-cleanupd
[1117055.604750] [ 7363] 0 7363 718756 9095 8992 103 0 536576 0 -500 dockerd
[1117055.604751] [ 7806] 0 7806 399401 690 625 65 0 143360 0 -500 docker-proxy
[1117055.604753] [ 7812] 0 7812 436267 652 626 26 0 155648 0 -500 docker-proxy
[1117055.604754] [ 7833] 0 7833 309627 1111 1099 12 0 135168 0 -998 containerd-shim
[1117055.604756] [ 7854] 568 7854 5107763 115604 88111 64 27429 1929216 0 0 jellyfin
[1117055.604758] [ 7926] 0 7926 309515 1570 1562 8 0 122880 0 -998 containerd-shim
[1117055.604760] [ 7952] 0 7952 287 64 64 0 0 36864 0 0 cinit
[1117055.604761] [ 8892] 1000 8892 22588 20184 20118 0 66 286720 0 0 Xvnc
[1117055.604763] [ 8935] 1000 8935 3270 2460 2454 6 0 61440 0 0 openbox
[1117055.604765] [ 8966] 1000 8966 367 288 288 0 0 40960 0 0 xcompmgr
[1117055.604767] [ 8972] 1000 8972 2089 1453 1453 0 0 57344 0 0 nginx
[1117055.604769] [ 8978] 1000 8978 2203 1565 1565 0 0 57344 0 0 nginx
[1117055.604770] [ 8979] 1000 8979 2203 1565 1565 0 0 57344 0 0 nginx
[1117055.604772] [ 8982] 1000 8982 409 58 0 58 0 32768 0 0 startapp.sh
[1117055.604774] [ 9345] 1000 9345 4580485 549286 549197 52 37 4837376 0 0 java
[1117055.604775] [ 15010] 0 15010 1688639 12014 11840 174 0 458752 0 0 incusd
[1117055.604777] [ 15442] 0 15442 19577 789 638 81 70 131072 0 0 wb-idmap
[1117055.604779] [ 33428] 65534 33428 9295972 8421702 36127 135 8385440 68509696 0 0 qemu-system-x86
[1117055.604781] [ 36106] 0 36106 454700 657 626 31 0 159744 0 -500 docker-proxy
[1117055.604783] [ 36113] 0 36113 417834 627 625 2 0 147456 0 -500 docker-proxy
[1117055.604785] [ 36141] 0 36141 309627 1289 1156 133 0 143360 0 -998 containerd-shim
[1117055.604786] [ 36162] 0 36162 582 0 0 0 0 40960 0 0 dumb-init
[1117055.604788] [ 36204] 0 36204 2967946 25488 25340 148 0 3330048 0 0 node
[1117055.604790] [ 370985] 0 370985 4161 285 224 61 0 77824 0 0 systemd-machine
[1117055.604792] [3492474] 0 3492474 20348 851 630 86 135 151552 0 0 smbd-scavenger
[1117055.604803] [3971685] 999 3971685 1113 242 128 114 0 53248 0 -900 bash
[1117055.604805] [3999344] 0 3999344 207078 13133 13021 112 0 307200 0 -1000 middlewared (wo
[1117055.604807] [4000200] 0 4000200 207053 13034 12932 102 0 303104 0 -1000 middlewared (wo
[1117055.604808] [4019967] 0 4019967 207330 13262 13185 77 0 307200 0 -1000 middlewared (wo
[1117055.604811] [4023522] 0 4023522 207053 12981 12906 75 0 311296 0 -1000 middlewared (wo
[1117055.604812] [4024440] 0 4024440 207057 12805 12686 119 0 307200 0 -1000 middlewared (wo
[1117055.604814] [4025124] 0 4025124 23531 1571 886 68 617 172032 0 0 smbd[fe80::1329
[1117055.604820] [4028816] 0 4028816 12294 160 32 128 0 69632 0 0 nscd
[1117055.604826] [4031110] 0 4031110 456756 767 672 95 0 188416 0 -998 runc
[1117055.604827] [4031111] 0 4031111 438259 709 672 37 0 180224 0 -998 runc
[1117055.604829] [4031130] 0 4031130 383096 922 855 67 0 151552 0 0 runc:[2:INIT]
[1117055.604831] [4031135] 568 4031135 401393 766 610 156 0 151552 0 0 runc:[2:INIT]
[1117055.604832] [4031143] 1000 4031143 403 100 0 100 0 49152 0 0 sleep
[1117055.604834] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=a9a85db74f92ad22fcbc8109cbef04382dc7e4cd34b4a406941a7ff4e4f5ae80,mems_allowed=0,global_oom,task_memcg=/system.slice/incus.service,task=qemu-system-x86,pid=33428,uid=65534
[1117055.604871] Out of memory: Killed process 33428 (qemu-system-x86) total-vm:37183888kB, anon-rss:144508kB, file-rss:540kB, shmem-rss:33541760kB, UID:65534 pgtables:66904kB oom_score_adj:0
[1117057.637464] oom_reaper: reaped process 33428 (qemu-system-x86), now anon-rss:0kB, file-rss:412kB, shmem-rss:33541760kB
Any idea what caused this? I never had this VM get killed over the last two years on previous versions.