Hi,
After finding the following, I thought it might be related with Stash-AppJun 25 17:03:02 nas kernel: ffmpeg invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
But it crashed even after stopping stash - the following seems to be related to Paperless-NGXJun 25 17:49:26 nas kernel: [celeryd: celer invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
System specs:
TrueNAS 25.04.1@Supermicro AOC-SLG3-2M2)
Any idea why this is happening and where to look more in detail?
Thanks
EDIT: Seems to be not related to Apps - so moving to General
Did you set limits per app? Most people aren’t having it (otherwise you’d have 20 posts on the forums). Check your htop and other processes, there’s gotta be more info or clues
Most of the apps are from the catalogue - I just use the default settings for RAM…which is 4GB for the apps I use.
Is the following more helpful?
Jun 25 17:03:02 nas kernel: ffmpeg invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Jun 25 17:03:02 nas kernel: CPU: 13 UID: 0 PID: 171622 Comm: ffmpeg Tainted: P OE 6.12.15-production+truenas #1
Jun 25 17:03:02 nas kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Jun 25 17:03:02 nas kernel: Hardware name: Supermicro Super Server/X11SDV-8C+-TLN2F, BIOS 1.2 11/14/2019
Jun 25 17:03:02 nas kernel: Call Trace:
Jun 25 17:03:02 nas kernel: <TASK>
Jun 25 17:03:02 nas kernel: dump_stack_lvl+0x64/0x80
Jun 25 17:03:02 nas kernel: dump_header+0x43/0x160
Jun 25 17:03:02 nas kernel: oom_kill_process+0xfa/0x200
Jun 25 17:03:02 nas kernel: out_of_memory+0x257/0x520
Jun 25 17:03:02 nas kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Jun 25 17:03:02 nas kernel: __alloc_pages_noprof+0x30e/0x330
Jun 25 17:03:02 nas kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Jun 25 17:03:02 nas kernel: folio_alloc_mpol_noprof+0x14/0x30
Jun 25 17:03:02 nas kernel: vma_alloc_folio_noprof+0x6b/0xc0
Jun 25 17:03:02 nas kernel: do_anonymous_page+0x2bd/0x7d0
Jun 25 17:03:02 nas kernel: ? __pte_offset_map+0x1b/0x180
Jun 25 17:03:02 nas kernel: __handle_mm_fault+0xb6d/0xfd0
Jun 25 17:03:02 nas kernel: ? sched_clock_cpu+0xf/0x190
Jun 25 17:03:02 nas kernel: handle_mm_fault+0x180/0x2d0
Jun 25 17:03:02 nas kernel: do_user_addr_fault+0x20c/0x670
Jun 25 17:03:02 nas kernel: exc_page_fault+0x76/0x190
Jun 25 17:03:02 nas kernel: asm_exc_page_fault+0x26/0x30
Jun 25 17:03:02 nas kernel: RIP: 0033:0x2b69ab9
Jun 25 17:03:02 nas kernel: Code: c6 c1 f8 04 44 01 e8 45 89 c5 83 f8 7e 41 0f 4f c2 85 c0 41 0f 4e c1 41 29 c5 41 39 c5 44 0f 4f e8 c1 f8 06 45 01 ed 44 09 e8 <88> 04 17 48 8d 42 01 49 39 d3 75 bb 83 c6 01 48 81 c7 00 04 00 00
Jun 25 17:03:02 nas kernel: RSP: 002b:00007ffcb3ccee58 EFLAGS: 00010206
Jun 25 17:03:02 nas kernel: RAX: 0000000000000077 RBX: 000000000554d7c0 RCX: 000000000492abc0
Jun 25 17:03:02 nas kernel: RDX: 0000000000000040 RSI: 000000000000001a RDI: 0000000005553fc0
Jun 25 17:03:02 nas kernel: RBP: 000000000492b3c0 R08: 000000000000007f R09: 0000000000000001
Jun 25 17:03:02 nas kernel: R10: 000000000000007e R11: 00000000000001cb R12: 00000000055677c0
Jun 25 17:03:02 nas kernel: R13: 0000000000000076 R14: 000000004271ad90 R15: 000000004271ad90
Jun 25 17:03:02 nas kernel: </TASK>
Jun 25 17:03:02 nas kernel: Mem-Info:
Jun 25 17:03:03 nas kernel: active_anon:3023 inactive_anon:30817727 isolated_anon:0
active_file:863 inactive_file:818 isolated_file:21
unevictable:768 dirty:49 writeback:0
slab_reclaimable:66819 slab_unreclaimable:789837
mapped:8060 shmem:8819 pagetables:64912
sec_pagetables:1044 bounce:0
kernel_misc_reclaimable:0
free:118882 free_pcp:33 free_cma:0
Jun 25 17:03:03 nas kernel: Node 0 active_anon:12092kB inactive_anon:123270932kB active_file:3344kB inactive_file:3168kB unevictable:3072kB isolated(anon):0kB isolated(file):136kB mapped:31636kB dirty:196kB writeback:0kB shmem:35276kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:110854144kB writeback_tmp:0kB kernel_stack:20512kB pagetables:259648kB sec_pagetables:4176kB all_unreclaimable? no
Jun 25 17:03:03 nas kernel: Node 0 DMA free:872kB boost:2048kB min:2052kB low:2064kB high:2076kB reserved_highatomic:0KB active_anon:0kB inactive_anon:6296kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 25 17:03:03 nas kernel: lowmem_reserve[]: 0 1550 128385 0 0
Jun 25 17:03:03 nas kernel: Node 0 DMA32 free:2108kB boost:5976kB min:6792kB low:8376kB high:9960kB reserved_highatomic:0KB active_anon:0kB inactive_anon:1645280kB active_file:16kB inactive_file:0kB unevictable:0kB writepending:0kB present:1732032kB managed:1664180kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 25 17:03:03 nas kernel: lowmem_reserve[]: 0 0 126834 0 0
Jun 25 17:03:03 nas kernel: Node 0 Normal free:478676kB boost:419840kB min:486596kB low:616472kB high:746348kB reserved_highatomic:10240KB active_anon:12092kB inactive_anon:121620084kB active_file:1032kB inactive_file:1556kB unevictable:3072kB writepending:196kB present:132120576kB managed:129885652kB mlocked:0kB bounce:0kB free_pcp:20kB local_pcp:0kB free_cma:0kB
Jun 25 17:03:03 nas kernel: lowmem_reserve[]: 0 0 0 0 0
Jun 25 17:03:03 nas kernel: Node 0 DMA: 0*4kB 12*8kB (U) 16*16kB (U) 16*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 864kB
Jun 25 17:03:03 nas kernel: Node 0 DMA32: 14*4kB (UM) 11*8kB (UM) 15*16kB (UM) 20*32kB (UM) 6*64kB (U) 5*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 2048kB
Jun 25 17:03:03 nas kernel: Node 0 Normal: 1242*4kB (UME) 604*8kB (UME) 6001*16kB (UME) 11764*32kB (UME) 2*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 482392kB
Jun 25 17:03:03 nas kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 25 17:03:03 nas kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 25 17:03:03 nas kernel: 9016 total pagecache pages
Jun 25 17:03:03 nas kernel: 0 pages in swap cache
Jun 25 17:03:03 nas kernel: Free swap = 0kB
Jun 25 17:03:03 nas kernel: Total swap = 0kB
Jun 25 17:03:03 nas kernel: 33467151 pages RAM
Jun 25 17:03:03 nas kernel: 0 pages HighMem/MovableOnly
Jun 25 17:03:03 nas kernel: 575853 pages reserved
Jun 25 17:03:03 nas kernel: 65536 pages cma reserved
Jun 25 17:03:03 nas kernel: 0 pages hwpoisoned
Jun 25 17:03:03 nas kernel: Tasks state (memory values in pages):
Jun 25 17:03:03 nas kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Jun 25 17:03:03 nas kernel: [ 833] 104 833 2082 327 160 167 0 57344 0 -900 dbus-daemon
Jun 25 17:03:03 nas kernel: [ 848] 0 848 15164 565 287 277 1 98304 0 -250 systemd-journal
Jun 25 17:03:03 nas kernel: [ 878] 0 878 7392 1282 1043 239 0 81920 0 -1000 systemd-udevd
Jun 25 17:03:03 nas kernel: [ 1842] 0 1842 924939 119978 119682 221 75 1544192 0 -1000 asyncio_loop
Jun 25 17:03:03 nas kernel: [ 1857] 0 1857 4196 2043 1824 219 0 73728 0 -1000 python3
Jun 25 17:03:03 nas kernel: [ 1901] 0 1901 470156 37070 36779 291 0 712704 0 -1000 middlewared (ze
Jun 25 17:03:03 nas kernel: [ 1914] 0 1914 9211 4465 4187 278 0 110592 0 -1000 python3
Jun 25 17:03:03 nas kernel: [ 3863] 0 3863 21732 328 124 204 0 53248 0 -1000 auditd
Jun 25 17:03:03 nas kernel: [ 3900] 0 3900 27525 4401 4147 254 0 114688 0 0 truenas_audit_h
Jun 25 17:03:03 nas kernel: [ 3903] 0 3903 5604 493 288 205 0 86016 0 0 certmonger
Jun 25 17:03:03 nas kernel: [ 3919] 0 3919 38191 233 64 169 0 65536 0 -1000 lxcfs
Jun 25 17:03:03 nas kernel: [ 3920] 0 3920 11738 294 96 198 0 69632 0 0 gssproxy
Jun 25 17:03:03 nas kernel: [ 3932] 0 3932 110598 202 32 170 0 106496 0 0 nscd
Jun 25 17:03:03 nas kernel: [ 3951] 0 3951 3078 686 480 206 0 73728 0 0 smartd
Jun 25 17:03:03 nas kernel: [ 3972] 0 3972 309076 2875 2587 288 0 630784 0 0 syslog-ng
Jun 25 17:03:03 nas kernel: [ 3988] 0 3988 4223 420 224 196 0 69632 0 0 systemd-logind
Jun 25 17:03:03 nas kernel: [ 4003] 0 4003 43794 430 256 174 0 77824 0 0 zed
Jun 25 17:03:03 nas kernel: [ 4011] 0 4011 691216 6523 6338 185 0 446464 0 -999 containerd
Jun 25 17:03:03 nas kernel: [ 4053] 0 4053 3910 546 320 226 0 77824 0 -1000 sshd
Jun 25 17:03:03 nas kernel: [ 4108] 0 4108 1005 260 64 196 0 45056 0 0 cron
Jun 25 17:03:03 nas kernel: [ 4161] 122 4161 3089 180 108 72 0 49152 0 0 usbhid-ups
Jun 25 17:03:03 nas kernel: [ 4208] 131 4208 4715 313 138 175 0 61440 0 0 chronyd
Jun 25 17:03:03 nas kernel: [ 4229] 131 4229 2686 301 116 185 0 61440 0 0 chronyd
Jun 25 17:03:03 nas kernel: [ 4335] 0 4335 91753 13588 13439 149 0 221184 0 0 cli
Jun 25 17:03:03 nas kernel: [ 4336] 122 4336 1695 318 96 222 0 49152 0 0 upsd
Jun 25 17:03:03 nas kernel: [ 4337] 0 4337 1223 224 64 160 0 45056 0 0 upsmon
Jun 25 17:03:03 nas kernel: [ 4339] 0 4339 36124 13386 13153 233 0 192512 0 0 cli
Jun 25 17:03:03 nas kernel: [ 4340] 122 4340 2127 281 97 184 0 53248 0 0 upsmon
Jun 25 17:03:03 nas kernel: [ 4346] 0 4346 7048 486 368 118 0 61440 0 0 nginx
Jun 25 17:03:03 nas kernel: [ 4347] 33 4347 7343 753 560 193 0 69632 0 0 nginx
Jun 25 17:03:03 nas kernel: [ 4928] 999 4928 137830 52303 52080 159 64 651264 0 -900 netdata
Jun 25 17:03:03 nas kernel: [ 5141] 999 5141 11686 383 192 191 0 69632 0 -900 netdata
Jun 25 17:03:03 nas kernel: [ 5490] 105 5490 1920 267 96 171 0 57344 0 0 avahi-daemon
Jun 25 17:03:03 nas kernel: [ 5547] 105 5547 1722 221 73 148 0 57344 0 0 avahi-daemon
Jun 25 17:03:03 nas kernel: [ 6083] 999 6083 1113 257 128 129 0 57344 0 -900 bash
Jun 25 17:03:03 nas kernel: [ 6202] 999 6202 22247 5011 4864 147 0 139264 0 -900 python.d.plugin
Jun 25 17:03:03 nas kernel: [ 6414] 1 6414 8070 4137 3872 265 0 106496 0 0 wsdd.py
Jun 25 17:03:03 nas kernel: [ 8232] 0 8232 18932 1019 640 283 96 135168 0 0 winbindd
Jun 25 17:03:03 nas kernel: [ 8235] 0 8235 2267720 12106 12089 17 0 1306624 0 -500 dockerd
Jun 25 17:03:03 nas kernel: [ 8237] 0 8237 18978 986 639 309 38 135168 0 0 wb[NAS]
Jun 25 17:03:03 nas kernel: [ 8239] 0 8239 20141 1609 640 215 754 143360 0 0 smbd
Jun 25 17:03:03 nas kernel: [ 8254] 0 8254 19608 765 623 142 0 126976 0 0 smbd-notifyd
Jun 25 17:03:03 nas kernel: [ 8255] 0 8255 19612 803 623 180 0 131072 0 0 smbd-cleanupd
Jun 25 17:03:03 nas kernel: [ 11120] 0 11120 436267 1267 1154 113 0 155648 0 -500 docker-proxy
Jun 25 17:03:03 nas kernel: [ 11140] 0 11140 436267 1330 1143 187 0 159744 0 -500 docker-proxy
Jun 25 17:03:03 nas kernel: [ 12072] 0 12072 309563 1254 1091 163 0 118784 0 -998 containerd-shim
Jun 25 17:03:03 nas kernel: [ 12131] 0 12131 309563 1090 915 175 0 118784 0 -998 containerd-shim
Jun 25 17:03:03 nas kernel: [ 12205] 0 12205 408 32 0 32 0 45056 0 0 entry.sh
Jun 25 17:03:03 nas kernel: [ 12206] 999 12206 33923 1498 1274 224 0 122880 0 0 redis-server
Jun 25 17:03:03 nas kernel: [ 12924] 0 12924 309563 1130 962 168 0 122880 0 -998 containerd-shim
Jun 25 17:03:03 nas kernel: [ 12996] 0 12996 309563 1182 1034 148 0 118784 0 -998 containerd-shim
Jun 25 17:03:03 nas kernel: [ 13053] 999 13053 54913 3144 352 136 2656 163840 0 0 postgres
Jun 25 17:03:03 nas kernel: [ 13097] 0 13097 360260 15857 15761 96 0 393216 0 0 crowdsec
Jun 25 17:03:03 nas kernel: [ 14611] 0 14611 403 66 0 66 0 45056 0 0 sleep
Jun 25 17:03:03 nas kernel: [ 15009] 0 15009 309563 1193 1014 179 0 126976 0 -998 containerd-shim
Jun 25 17:03:03 nas kernel: [ 15240] 0 15240 328630 12206 12046 160 0 311296 0 0 caddy
Jun 25 17:03:03 nas kernel: [ 15249] 999 15249 54943 987 559 29 399 155648 0 0 postgres
Jun 25 17:03:03 nas kernel: [ 15252] 999 15252 54947 945 559 129 257 143360 0 0 postgres
Jun 25 17:03:03 nas kernel: [ 15384] 999 15384 54913 1754 559 138 1057 139264 0 0 postgres
Jun 25 17:03:03 nas kernel: [ 15385] 999 15385 55309 955 623 94 238 155648 0 0 postgres
Jun 25 17:03:03 nas kernel: [ 15392] 999 15392 55303 867 623 109 135 143360 0 0 postgres
Jun 25 17:03:03 nas kernel: [ 15463] 0 15463 309563 1268 1090 178 0 118784 0 -998 containerd-shim
Jun 25 17:03:03 nas kernel: [ 15681] 0 15681 947106 23279 23122 157 0 688128 0 0 stash
Jun 25 17:03:03 nas kernel: [ 16846] 0 16846 309563 1284 1124 160 0 122880 0 -998 containerd-shim
Jun 25 17:03:03 nas kernel: [ 16898] 0 16898 670241 7895 7695 200 0 421888 0 0 authelia
Jun 25 17:03:03 nas kernel: [ 18296] 999 18296 55698 2145 879 187 1079 163840 0 0 postgres
Jun 25 17:03:03 nas kernel: [ 18388] 999 18388 55667 1943 879 73 991 163840 0 0 postgres
Jun 25 17:03:03 nas kernel: [ 20148] 65534 20148 314440 2491 2395 96 0 167936 0 0 notification-em
Jun 25 17:03:03 nas kernel: [ 59708] 1001 59708 24388 2288 921 267 1100 176128 0 0 smbd[192.168.1.
Jun 25 17:03:03 nas kernel: [ 59710] 0 59710 19478 859 639 156 64 135168 0 0 wb-idmap
Jun 25 17:03:03 nas kernel: [ 59716] 1001 59716 5326 1173 928 245 0 81920 0 100 systemd
Jun 25 17:03:03 nas kernel: [ 59717] 1001 59717 43170 2010 1795 215 0 98304 0 100 (sd-pam)
Jun 25 17:03:03 nas kernel: [ 99320] 0 99320 4558 809 572 237 0 77824 0 0 sshd
Jun 25 17:03:03 nas kernel: [ 99389] 0 99389 5329 1196 960 236 0 81920 0 100 systemd
Jun 25 17:03:03 nas kernel: [ 99390] 0 99390 43163 2013 1797 216 0 98304 0 100 (sd-pam)
Jun 25 17:03:03 nas kernel: [ 99427] 0 99427 1486 373 192 181 0 57344 0 0 zsh
Jun 25 17:03:03 nas kernel: [ 99923] 1001 99923 29721 8656 5771 199 2686 221184 0 0 smbd[192.168.1.
Jun 25 17:03:03 nas kernel: [ 109201] 0 109201 434 99 32 67 0 40960 0 0 sh
Jun 25 17:03:03 nas kernel: [ 122618] 0 122618 644 161 0 161 0 49152 0 -1000 sh
Jun 25 17:03:03 nas kernel: [ 122620] 0 122620 644 78 0 78 0 49152 0 -1000 sh
Jun 25 17:03:03 nas kernel: [ 122622] 0 122622 644 78 0 78 0 49152 0 -1000 sh
Jun 25 17:03:03 nas kernel: [ 122623] 0 122623 644 78 0 78 0 49152 0 -1000 sh
Jun 25 17:03:03 nas kernel: [ 122625] 0 122625 644 157 0 157 0 49152 0 -1000 sh
Jun 25 17:03:03 nas kernel: [ 122626] 0 122626 644 71 0 71 0 49152 0 -1000 sh
Jun 25 17:03:03 nas kernel: [ 122627] 0 122627 644 72 0 72 0 49152 0 -1000 sh
Jun 25 17:03:03 nas kernel: [ 122628] 0 122628 644 72 0 72 0 49152 0 -1000 sh
Jun 25 17:03:03 nas kernel: [ 122629] 0 122629 644 138 0 138 0 45056 0 -1000 sh
Jun 25 17:03:03 nas kernel: [ 122630] 0 122630 536873819 30286196 30286043 153 0 246779904 0 -1000 mbuffer
Jun 25 17:03:03 nas kernel: [ 122631] 0 122631 3772 1377 1141 236 0 77824 0 -1000 ssh
Jun 25 17:03:03 nas kernel: [ 122632] 0 122632 644 117 0 117 0 45056 0 -1000 sh
Jun 25 17:03:03 nas kernel: [ 122633] 0 122633 21526 579 320 259 0 73728 0 -1000 zfs
Jun 25 17:03:03 nas kernel: [ 164252] 0 164252 224959 12559 12279 280 0 315392 0 -1000 middlewared (wo
Jun 25 17:03:03 nas kernel: [ 164711] 0 164711 225116 12457 12233 224 0 307200 0 -1000 middlewared (wo
Jun 25 17:03:03 nas kernel: [ 165045] 0 165045 225115 12376 12178 198 0 311296 0 -1000 middlewared (wo
Jun 25 17:03:03 nas kernel: [ 170119] 0 170119 206678 12212 12081 131 0 299008 0 -1000 middlewared (wo
Jun 25 17:03:03 nas kernel: [ 170531] 0 170531 206679 12426 12182 244 0 299008 0 -1000 middlewared (wo
Jun 25 17:03:03 nas kernel: [ 171050] 0 171050 440978 30483 30217 266 0 634880 0 0 ffmpeg
Jun 25 17:03:03 nas kernel: [ 171338] 0 171338 424695 22440 22335 105 0 573440 0 0 ffmpeg
Jun 25 17:03:03 nas kernel: [ 171622] 0 171622 398931 16961 16689 272 0 483328 0 0 ffmpeg
Jun 25 17:03:03 nas kernel: [ 171781] 0 171781 424445 19258 19057 201 0 565248 0 0 ffmpeg
Jun 25 17:03:03 nas kernel: [ 171820] 0 171820 3061 444 256 188 0 65536 0 -1000 zfs
Jun 25 17:03:03 nas kernel: [ 171885] 0 171885 27018 1820 1568 252 0 172032 0 0 ffmpeg
Jun 25 17:03:03 nas kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=fb2b5e46bd75eb5c5ade5debe4f8a3923d7e13faebce1eed1cd49b901e084804,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/user@0.service/init.scope,task=(sd-pam),pid=99390,uid=0
Jun 25 17:03:03 nas kernel: systemd-journal invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-250
Jun 25 17:03:03 nas kernel: CPU: 6 UID: 0 PID: 848 Comm: systemd-journal Tainted: P OE 6.12.15-production+truenas #1
Jun 25 17:03:03 nas kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Jun 25 17:03:03 nas kernel: Hardware name: Supermicro Super Server/X11SDV-8C+-TLN2F, BIOS 1.2 11/14/2019
You really need to provide all the details for your system and the apps you are running. We can only go off the information provided within the forum posts. Were there any recent changes like app updates? What version of the apps? What does your memory usage look like over time?
1 Like
Sorry…
TrueNAS 25.04.1
The following apps are configured but stopped now
authelia (Catalog)
authelia-old (YAML)
caddy (YAML)
calibre (YAML)
change-detection
changedetection-old (YAML)
dokuwiki (YAML)
fireflyiii (YAML)
home-assistant (YAML)
joplin (Catalog)
lidarr (Catalog)
metube (Catalog)
ntfy (Catalog)
nzbget (Catalog)
overseerr (Catalog)
paperless-ngx (Catalog)
pgadmin (Catalog)
playwright (Catalog)
plex (Catalog)
prowlarr (Catalog)
radarr (Catalog)
readarr (Catalog)
scrutiny (Catalog)
smtp (Custom)
smtp-old (YAML)
sonarr (Catalog)
stash (Catalog)
stash-old (YAML)
tautulli (Catalog)
threadfin (YAML)
uptime-kuma (Catalog)
vaultwarden (Catalog)
watchtower (YAML)
whisparr (YAML)
As mentioned before - to find the app that is causing the issued, I stopped all the apps and just started i.e. Paperless.
The latest update was yesterday at 04:00am which was Home Assistant - via Watchtower.
Any more details I missed here?
Thank you for your help!!!
Actually it now crashed even with all apps stopped!
PK1048
June 25, 2025, 6:24pm
7
You list 34 apps.
You need to follow this TrueNAS Hardware Guide | TrueNAS Documentation Hub as a MINIMUM RAM for just the TrueNAS OS and base functionality. Then add enough RAM for the Apps load (another 136 GiB).
Now, you may say, “Why, all of the Apps do not need all of their memory at once.”, which may be true, until they do. At which point the Out of Memory (oom) will start killing things off, hopefully not important things. Once the oom-killer has been called it is safest to reboot to ensure that all critical processes are running and fully funded with memory.
Thanks for the hint.
PK1048
June 25, 2025, 6:29pm
9
What is your system configuration?
What do the systems logs show at the time it crashed? The oom-killer runs to prevent the system from crashing.
This is the log from the latest crash - without any apps running.
I am not an expert - but could it be that “mbuffer” process has something to do with it?
Jun 25 18:07:19 nas kernel: vethbb7fd90 (unregistering): left allmulticast mode
Jun 25 18:07:19 nas kernel: vethbb7fd90 (unregistering): left promiscuous mode
Jun 25 18:07:19 nas kernel: br-219ae9da4429: port 4(vethbb7fd90) entered disabled state
Jun 25 18:07:19 nas kernel: br-219ae9da4429: port 2(vethae82663) entered disabled state
Jun 25 18:07:19 nas kernel: vethae82663 (unregistering): left allmulticast mode
Jun 25 18:07:19 nas kernel: vethae82663 (unregistering): left promiscuous mode
Jun 25 18:07:19 nas kernel: br-219ae9da4429: port 2(vethae82663) entered disabled state
Jun 25 18:07:19 nas kernel: br-219ae9da4429: port 3(vetha28d957) entered disabled state
Jun 25 18:07:19 nas kernel: vetha28d957 (unregistering): left allmulticast mode
Jun 25 18:07:19 nas kernel: vetha28d957 (unregistering): left promiscuous mode
Jun 25 18:07:19 nas kernel: br-219ae9da4429: port 3(vetha28d957) entered disabled state
Jun 25 18:07:19 nas kernel: br-219ae9da4429: port 1(vethab72376) entered disabled state
Jun 25 18:07:19 nas kernel: vethab72376 (unregistering): left allmulticast mode
Jun 25 18:07:19 nas kernel: vethab72376 (unregistering): left promiscuous mode
Jun 25 18:07:19 nas kernel: br-219ae9da4429: port 1(vethab72376) entered disabled state
Jun 25 18:10:07 nas kernel: loop0: detected capacity change from 0 to 3241176
Jun 25 18:21:34 nas kernel: loop0: detected capacity change from 0 to 272
Jun 25 18:21:34 nas kernel: loop1: detected capacity change from 0 to 272
Jun 25 18:21:34 nas kernel: loop2: detected capacity change from 0 to 272
Jun 25 18:21:35 nas kernel: loop3: detected capacity change from 0 to 272
Jun 25 18:27:38 nas kernel: loop1: detected capacity change from 0 to 3241176
Jun 25 18:39:09 nas kernel: loop1: detected capacity change from 0 to 3241176
Jun 25 19:42:08 nas kernel: loop1: detected capacity change from 0 to 3241176
Jun 25 19:58:40 nas kernel: loop1: detected capacity change from 0 to 3241176
Jun 25 20:06:02 nas kernel: smbd[192.168.1. invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Jun 25 20:06:02 nas kernel: CPU: 6 UID: 1001 PID: 82282 Comm: smbd[192.168.1. Tainted: P OE 6.12.15-production+truenas #1
Jun 25 20:06:03 nas kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Jun 25 20:06:03 nas kernel: Hardware name: Supermicro Super Server/X11SDV-8C+-TLN2F, BIOS 1.2 11/14/2019
Jun 25 20:06:03 nas kernel: Call Trace:
Jun 25 20:06:03 nas kernel: <TASK>
Jun 25 20:06:03 nas kernel: dump_stack_lvl+0x64/0x80
Jun 25 20:06:03 nas kernel: dump_header+0x43/0x160
Jun 25 20:06:03 nas kernel: oom_kill_process+0xfa/0x200
Jun 25 20:06:03 nas kernel: out_of_memory+0x257/0x520
Jun 25 20:06:03 nas kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Jun 25 20:06:03 nas kernel: __alloc_pages_noprof+0x30e/0x330
Jun 25 20:06:03 nas kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Jun 25 20:06:03 nas kernel: folio_alloc_noprof+0x5b/0xb0
Jun 25 20:06:03 nas kernel: __filemap_get_folio+0x1aa/0x2e0
Jun 25 20:06:03 nas kernel: filemap_fault+0x647/0xd40
Jun 25 20:06:03 nas kernel: __do_fault+0x30/0x170
Jun 25 20:06:03 nas kernel: do_fault+0x2a8/0x4d0
Jun 25 20:06:03 nas kernel: __handle_mm_fault+0x7b8/0xfd0
Jun 25 20:06:03 nas kernel: handle_mm_fault+0x180/0x2d0
Jun 25 20:06:03 nas kernel: do_user_addr_fault+0x20c/0x670
Jun 25 20:06:03 nas kernel: exc_page_fault+0x76/0x190
Jun 25 20:06:03 nas kernel: asm_exc_page_fault+0x26/0x30
Jun 25 20:06:03 nas kernel: RIP: 0033:0x7f4d3dfcb150
Jun 25 20:06:03 nas kernel: Code: Unable to access opcode bytes at 0x7f4d3dfcb126.
Jun 25 20:06:03 nas kernel: RSP: 002b:00007ffce7bbd5b8 EFLAGS: 00010246
Jun 25 20:06:03 nas kernel: RAX: 0000000000000000 RBX: 000055d334c92980 RCX: 0000000000000000
Jun 25 20:06:03 nas kernel: RDX: 000055d334c94380 RSI: 00000000000009ac RDI: 00007ffce7bbd5c0
Jun 25 20:06:03 nas kernel: RBP: 000055d334c94390 R08: 0000000000000000 R09: 0000000000000000
Jun 25 20:06:03 nas kernel: R10: 00000000000003c2 R11: 0000000000000202 R12: 00007f4d3e97b546
Jun 25 20:06:03 nas kernel: R13: 000055d334c94260 R14: 0000000000008070 R15: 000055d334e22a50
Jun 25 20:06:03 nas kernel: </TASK>
Jun 25 20:06:03 nas kernel: Mem-Info:
Jun 25 20:06:03 nas kernel: active_anon:2889 inactive_anon:31367187 isolated_anon:0
active_file:0 inactive_file:2 isolated_file:0
unevictable:768 dirty:0 writeback:10
slab_reclaimable:53560 slab_unreclaimable:148333
mapped:4416 shmem:6386 pagetables:65628
sec_pagetables:1044 bounce:0
kernel_misc_reclaimable:0
free:137294 free_pcp:8 free_cma:0
Jun 25 20:06:03 nas kernel: Node 0 active_anon:11556kB inactive_anon:125468748kB active_file:0kB inactive_file:0kB unevictable:3072kB isolated(anon):0kB isolated(file):0kB mapped:17652kB dirty:0kB writeback:40kB shmem:25544kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:106158080kB writeback_tmp:0kB kernel_stack:13408kB pagetables:262512kB sec_pagetables:4176kB all_unreclaimable? no
Jun 25 20:06:03 nas kernel: Node 0 DMA free:664kB boost:2048kB min:2052kB low:2064kB high:2076kB reserved_highatomic:0KB active_anon:0kB inactive_anon:6504kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 25 20:06:03 nas kernel: lowmem_reserve[]: 0 1550 128385 0 0
Jun 25 20:06:03 nas kernel: Node 0 DMA32 free:1952kB boost:5976kB min:6792kB low:8376kB high:9960kB reserved_highatomic:0KB active_anon:0kB inactive_anon:1645932kB active_file:4kB inactive_file:8kB unevictable:0kB writepending:0kB present:1732032kB managed:1664180kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 25 20:06:03 nas kernel: lowmem_reserve[]: 0 0 126834 0 0
Jun 25 20:06:03 nas kernel: Node 0 Normal free:546568kB boost:489760kB min:556516kB low:686392kB high:816268kB reserved_highatomic:0KB active_anon:11556kB inactive_anon:123816372kB active_file:440kB inactive_file:0kB unevictable:3072kB writepending:40kB present:132120576kB managed:129885660kB mlocked:0kB bounce:0kB free_pcp:32kB local_pcp:0kB free_cma:0kB
Jun 25 20:06:03 nas kernel: lowmem_reserve[]: 0 0 0 0 0
Jun 25 20:06:03 nas kernel: Node 0 DMA: 0*4kB 1*8kB (U) 9*16kB (U) 16*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 664kB
Jun 25 20:06:03 nas kernel: Node 0 DMA32: 6*4kB (U) 14*8kB (UM) 12*16kB (UM) 8*32kB (UM) 6*64kB (U) 5*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 1864kB
Jun 25 20:06:03 nas kernel: Node 0 Normal: 63*4kB (UME) 54*8kB (UME) 12074*16kB (UME) 10983*32kB (UME) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 545388kB
Jun 25 20:06:03 nas kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 25 20:06:03 nas kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 25 20:06:03 nas kernel: 6588 total pagecache pages
Jun 25 20:06:03 nas kernel: 0 pages in swap cache
Jun 25 20:06:03 nas kernel: Free swap = 0kB
Jun 25 20:06:03 nas kernel: Total swap = 0kB
Jun 25 20:06:03 nas kernel: 33467151 pages RAM
Jun 25 20:06:03 nas kernel: 0 pages HighMem/MovableOnly
Jun 25 20:06:03 nas kernel: 575851 pages reserved
Jun 25 20:06:03 nas kernel: 65536 pages cma reserved
Jun 25 20:06:03 nas kernel: 0 pages hwpoisoned
Jun 25 20:06:03 nas kernel: Tasks state (memory values in pages):
Jun 25 20:06:03 nas kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Jun 25 20:06:03 nas kernel: [ 836] 104 836 2088 307 160 147 0 61440 0 -900 dbus-daemon
Jun 25 20:06:03 nas kernel: [ 849] 0 849 11514 405 275 130 0 86016 0 -250 systemd-journal
Jun 25 20:06:03 nas kernel: [ 880] 0 880 6701 599 439 160 0 69632 0 -1000 systemd-udevd
Jun 25 20:06:03 nas kernel: [ 1843] 0 1843 866978 125405 125069 231 105 1556480 0 -1000 asyncio_loop
Jun 25 20:06:03 nas kernel: [ 1863] 0 1863 4196 1619 1472 147 0 73728 0 -1000 python3
Jun 25 20:06:03 nas kernel: [ 1902] 0 1902 922579 38475 38296 179 0 823296 0 -1000 middlewared (ze
Jun 25 20:06:03 nas kernel: [ 1911] 0 1911 9211 4519 4316 203 0 110592 0 -1000 python3
Jun 25 20:06:03 nas kernel: [ 3910] 0 3910 21735 237 66 171 0 57344 0 -1000 auditd
Jun 25 20:06:03 nas kernel: [ 3938] 0 3938 27525 4306 4128 178 0 122880 0 0 truenas_audit_h
Jun 25 20:06:03 nas kernel: [ 3941] 0 3941 5604 556 288 268 0 86016 0 0 certmonger
Jun 25 20:06:03 nas kernel: [ 3956] 0 3956 38191 179 32 147 0 61440 0 -1000 lxcfs
Jun 25 20:06:03 nas kernel: [ 3957] 0 3957 11738 257 96 161 0 69632 0 0 gssproxy
Jun 25 20:06:03 nas kernel: [ 3988] 0 3988 3078 694 512 182 0 65536 0 0 smartd
Jun 25 20:06:03 nas kernel: [ 4012] 0 4012 361315 1804 1597 207 0 663552 0 0 syslog-ng
Jun 25 20:06:03 nas kernel: [ 4026] 0 4026 4220 449 224 225 0 81920 0 0 systemd-logind
Jun 25 20:06:03 nas kernel: [ 4040] 0 4040 60178 520 288 232 0 98304 0 0 zed
Jun 25 20:06:03 nas kernel: [ 4049] 0 4049 691024 5282 5110 172 0 425984 0 -999 containerd
Jun 25 20:06:03 nas kernel: [ 4077] 0 4077 3910 475 256 219 0 65536 0 -1000 sshd
Jun 25 20:06:03 nas kernel: [ 4140] 0 4140 1005 184 32 152 0 49152 0 0 cron
Jun 25 20:06:03 nas kernel: [ 4226] 122 4226 3089 205 94 111 0 57344 0 0 usbhid-ups
Jun 25 20:06:03 nas kernel: [ 4292] 0 4292 91746 13505 13263 242 0 225280 0 0 cli
Jun 25 20:06:03 nas kernel: [ 4293] 122 4293 1695 245 64 181 0 53248 0 0 upsd
Jun 25 20:06:03 nas kernel: [ 4295] 0 4295 1223 200 32 168 0 53248 0 0 upsmon
Jun 25 20:06:03 nas kernel: [ 4296] 122 4296 2127 264 96 168 0 61440 0 0 upsmon
Jun 25 20:06:03 nas kernel: [ 4326] 131 4326 4715 338 138 200 0 57344 0 0 chronyd
Jun 25 20:06:03 nas kernel: [ 4334] 131 4334 2686 264 116 148 0 57344 0 0 chronyd
Jun 25 20:06:03 nas kernel: [ 4376] 0 4376 36116 13471 13249 222 0 196608 0 0 cli
Jun 25 20:06:03 nas kernel: [ 4388] 0 4388 7048 424 366 58 0 61440 0 0 nginx
Jun 25 20:06:03 nas kernel: [ 4389] 33 4389 7343 757 558 199 0 77824 0 0 nginx
Jun 25 20:06:03 nas kernel: [ 5288] 999 5288 132189 46658 46520 106 32 598016 0 -900 netdata
Jun 25 20:06:03 nas kernel: [ 5314] 999 5314 11686 438 192 246 0 77824 0 -900 netdata
Jun 25 20:06:03 nas kernel: [ 5603] 105 5603 1788 269 64 205 0 57344 0 0 avahi-daemon
Jun 25 20:06:03 nas kernel: [ 5656] 105 5656 1722 178 73 105 0 57344 0 0 avahi-daemon
Jun 25 20:06:03 nas kernel: [ 6315] 1 6315 8070 4209 3968 241 0 106496 0 0 wsdd.py
Jun 25 20:06:03 nas kernel: [ 6360] 999 6360 1113 231 128 103 0 45056 0 -900 bash
Jun 25 20:06:03 nas kernel: [ 6361] 999 6361 22247 4866 4760 106 0 139264 0 -900 python.d.plugin
Jun 25 20:06:03 nas kernel: [ 8306] 0 8306 18932 967 640 231 96 131072 0 0 winbindd
Jun 25 20:06:03 nas kernel: [ 8313] 0 8313 18978 871 639 196 36 131072 0 0 wb[NAS]
Jun 25 20:06:03 nas kernel: [ 8315] 0 8315 20141 1631 640 270 721 143360 0 0 smbd
Jun 25 20:06:03 nas kernel: [ 8329] 0 8329 19608 783 623 160 0 126976 0 0 smbd-notifyd
Jun 25 20:06:03 nas kernel: [ 8331] 0 8331 19612 758 623 135 0 131072 0 0 smbd-cleanupd
Jun 25 20:06:03 nas kernel: [ 8365] 0 8365 903614 8121 7937 184 0 614400 0 -500 dockerd
Jun 25 20:06:03 nas kernel: [ 11900] 0 11900 4557 811 563 248 0 77824 0 0 sshd
Jun 25 20:06:03 nas kernel: [ 11911] 0 11911 5051 1019 736 283 0 77824 0 100 systemd
Jun 25 20:06:03 nas kernel: [ 11912] 0 11912 42864 1719 1580 139 0 98304 0 100 (sd-pam)
Jun 25 20:06:03 nas kernel: [ 11952] 0 11952 1486 358 192 166 0 53248 0 0 zsh
Jun 25 20:06:03 nas kernel: [ 63369] 0 63369 644 123 0 123 0 49152 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 63371] 0 63371 644 69 0 69 0 49152 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 63373] 0 63373 644 69 0 69 0 49152 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 63374] 0 63374 644 69 0 69 0 49152 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 63376] 0 63376 644 148 0 148 0 49152 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 63377] 0 63377 4363 2004 1763 241 0 81920 0 -1000 ssh
Jun 25 20:06:03 nas kernel: [ 63378] 0 63378 644 47 0 47 0 49152 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 63380] 0 63380 644 47 0 47 0 49152 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 63382] 0 63382 536873819 8481827 8481697 130 0 72167424 0 -1000 mbuffer
Jun 25 20:06:03 nas kernel: [ 82282] 1001 82282 31078 9863 5892 198 3773 237568 0 0 smbd[192.168.1.
Jun 25 20:06:03 nas kernel: [ 82283] 0 82283 19478 854 639 151 64 131072 0 0 wb-idmap
Jun 25 20:06:03 nas kernel: [ 82285] 1001 82285 5039 916 704 212 0 81920 0 100 systemd
Jun 25 20:06:03 nas kernel: [ 82287] 1001 82287 42864 1757 1583 174 0 98304 0 100 (sd-pam)
Jun 25 20:06:03 nas kernel: [ 140452] 0 140452 4105 521 224 297 0 77824 0 0 systemd-machine
Jun 25 20:06:03 nas kernel: [ 187513] 0 187513 206912 12847 12568 279 0 307200 0 -1000 middlewared (wo
Jun 25 20:06:03 nas kernel: [ 187514] 0 187514 208476 14392 14162 230 0 327680 0 -1000 middlewared (wo
Jun 25 20:06:03 nas kernel: [ 187523] 0 187523 754862 17723 17499 224 0 491520 0 -1000 middlewared (wo
Jun 25 20:06:03 nas kernel: [ 189175] 0 189175 226611 14126 13865 261 0 323584 0 -1000 middlewared (wo
Jun 25 20:06:03 nas kernel: [ 190740] 0 190740 28678 195 32 163 0 86016 0 0 nscd
Jun 25 20:06:03 nas kernel: [ 197450] 0 197450 206860 12900 12691 209 0 299008 0 -1000 middlewared (wo
Jun 25 20:06:03 nas kernel: [ 202970] 0 202970 644 101 0 101 0 45056 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 202972] 0 202972 644 44 0 44 0 45056 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 202973] 0 202973 644 44 0 44 0 45056 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 202974] 0 202974 644 44 0 44 0 45056 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 202975] 0 202975 644 113 0 113 0 40960 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 202977] 0 202977 3921 1513 1289 224 0 65536 0 -1000 ssh
Jun 25 20:06:03 nas kernel: [ 202979] 0 202979 644 45 0 45 0 40960 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 202980] 0 202980 644 45 0 45 0 40960 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 202981] 0 202981 644 45 0 45 0 40960 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 202982] 0 202982 644 132 0 132 0 45056 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 202983] 0 202983 536873819 22519197 22519054 143 0 184586240 0 -1000 mbuffer
Jun 25 20:06:03 nas kernel: [ 202984] 0 202984 644 82 0 82 0 45056 0 -1000 sh
Jun 25 20:06:03 nas kernel: [ 202985] 0 202985 21526 483 320 163 0 81920 0 -1000 zfs
Jun 25 20:06:03 nas kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1001.slice/user@1001.service/init.scope,task=(sd-pam),pid=82287,uid=1001
Jun 25 20:06:03 nas kernel: cli invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Jun 25 20:06:03 nas kernel: CPU: 13 UID: 0 PID: 4292 Comm: cli Tainted: P OE 6.12.15-production+truenas #1
Jun 25 20:06:03 nas kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Jun 25 20:06:03 nas kernel: Hardware name: Supermicro Super Server/X11SDV-8C+-TLN2F, BIOS 1.2 11/14/2019
Jun 25 20:06:03 nas kernel: Call Trace:
Jun 25 20:06:03 nas kernel: <TASK>
Jun 25 20:06:03 nas kernel: dump_stack_lvl+0x64/0x80
Jun 25 20:06:03 nas kernel: dump_header+0x43/0x160
Jun 25 20:06:03 nas kernel: oom_kill_process+0xfa/0x200
Jun 25 20:06:03 nas kernel: out_of_memory+0x257/0x520
Jun 25 20:06:03 nas kernel: __alloc_pages_slowpath.constprop.0+0x696/0xdf0
Jun 25 20:06:03 nas kernel: __alloc_pages_noprof+0x30e/0x330
Jun 25 20:06:03 nas kernel: alloc_pages_mpol_noprof+0x8f/0x1f0
Jun 25 20:06:03 nas kernel: folio_alloc_noprof+0x5b/0xb0
Jun 25 20:06:03 nas kernel: __filemap_get_folio+0x1aa/0x2e0
Jun 25 20:06:03 nas kernel: filemap_fault+0x647/0xd40
Jun 25 20:06:03 nas kernel: __do_fault+0x30/0x170
Jun 25 20:06:03 nas kernel: do_fault+0x2a8/0x4d0
Jun 25 20:06:03 nas kernel: __handle_mm_fault+0x7b8/0xfd0
Jun 25 20:06:04 nas kernel: handle_mm_fault+0x180/0x2d0
Jun 25 20:06:04 nas kernel: do_user_addr_fault+0x20c/0x670
Jun 25 20:06:04 nas kernel: exc_page_fault+0x76/0x190
Jun 25 20:06:04 nas kernel: asm_exc_page_fault+0x26/0x30
Jun 25 20:06:04 nas kernel: RIP: 0033:0x7f7c4b2a0ea6
Jun 25 20:06:04 nas kernel: Code: Unable to access opcode bytes at 0x7f7c4b2a0e7c.
Jun 25 20:06:04 nas kernel: RSP: 002b:00007ffd87d5a880 EFLAGS: 00010293
Jun 25 20:06:04 nas kernel: RAX: 0000000000000000 RBX: 0000000000a76338 RCX: 00007f7c4b2a0ea6
Jun 25 20:06:04 nas kernel: RDX: 0000000000000002 RSI: 00007f7c43910750 RDI: 0000000000000003
Jun 25 20:06:04 nas kernel: RBP: 00007f7c4afd9698 R08: 0000000000000000 R09: 0000000000000000
Jun 25 20:06:04 nas kernel: R10: 00000000000001f4 R11: 0000000000000293 R12: 00007f7c4b197240
Jun 25 20:06:04 nas kernel: R13: 0000000000000002 R14: 00007f7c49214af0 R15: 00007f7c43910750
Jun 25 20:06:04 nas kernel: </TASK>
Jun 25 20:06:04 nas kernel: Mem-Info:
Jun 25 20:06:04 nas kernel: active_anon:2889 inactive_anon:31369629 isolated_anon:0
active_file:411 inactive_file:114 isolated_file:0
unevictable:768 dirty:0 writeback:10
slab_reclaimable:53334 slab_unreclaimable:146868
mapped:4692 shmem:6386 pagetables:65639
sec_pagetables:1044 bounce:0
kernel_misc_reclaimable:0
free:136902 free_pcp:28 free_cma:0
Jun 25 20:06:04 nas kernel: Node 0 active_anon:11556kB inactive_anon:125481180kB active_file:1436kB inactive_file:0kB unevictable:3072kB isolated(anon):0kB isolated(file):0kB mapped:19212kB dirty:0kB writeback:40kB shmem:25544kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:106158080kB writeback_tmp:0kB kernel_stack:13392kB pagetables:262556kB sec_pagetables:4176kB all_unreclaimable? no
Jun 25 20:06:04 nas kernel: Node 0 DMA free:620kB boost:0kB min:4kB low:16kB high:28kB reserved_highatomic:0KB active_anon:0kB inactive_anon:6520kB active_file:12kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 25 20:06:04 nas kernel: lowmem_reserve[]: 0 1550 128385 0 0
Jun 25 20:06:04 nas kernel: Node 0 DMA32 free:1912kB boost:5976kB min:6792kB low:8376kB high:9960kB reserved_highatomic:0KB active_anon:0kB inactive_anon:1645932kB active_file:0kB inactive_file:12kB unevictable:0kB writepending:0kB present:1732032kB managed:1664180kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jun 25 20:06:04 nas kernel: lowmem_reserve[]: 0 0 126834 0 0
Jun 25 20:06:04 nas kernel: Node 0 Normal free:541004kB boost:489760kB min:556516kB low:686392kB high:816268kB reserved_highatomic:0KB active_anon:11556kB inactive_anon:123828468kB active_file:1008kB inactive_file:0kB unevictable:3072kB writepending:40kB present:132120576kB managed:129885660kB mlocked:0kB bounce:0kB free_pcp:92kB local_pcp:0kB free_cma:0kB
Jun 25 20:06:04 nas kernel: lowmem_reserve[]: 0 0 0 0 0
Jun 25 20:06:04 nas kernel: Node 0 DMA: 2*4kB (U) 2*8kB (U) 6*16kB (U) 16*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 632kB
Jun 25 20:06:04 nas kernel: Node 0 DMA32: 11*4kB (UM) 13*8kB (UM) 15*16kB (UM) 11*32kB (UM) 9*64kB (U) 4*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 1828kB
Jun 25 20:06:04 nas kernel: Node 0 Normal: 182*4kB (UME) 12*8kB (UME) 11693*16kB (UME) 10924*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB = 537480kB
Jun 25 20:06:04 nas kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 25 20:06:04 nas kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 25 20:06:04 nas kernel: 6823 total pagecache pages
Jun 25 20:06:04 nas kernel: 0 pages in swap cache
Jun 25 20:06:04 nas kernel: Free swap = 0kB
Jun 25 20:06:04 nas kernel: Total swap = 0kB
Jun 25 20:06:04 nas kernel: 33467151 pages RAM
Jun 25 20:06:04 nas kernel: 0 pages HighMem/MovableOnly
Jun 25 20:06:04 nas kernel: 575851 pages reserved
Jun 25 20:06:04 nas kernel: 65536 pages cma reserved
Jun 25 20:06:04 nas kernel: 0 pages hwpoisoned
Jun 25 20:06:04 nas kernel: Tasks state (memory values in pages):
Jun 25 20:06:04 nas kernel: [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Jun 25 20:06:04 nas kernel: [ 836] 104 836 2088 307 160 147 0 61440 0 -900 dbus-daemon
Jun 25 20:06:04 nas kernel: [ 849] 0 849 11514 509 275 234 0 86016 0 -250 systemd-journal
Jun 25 20:06:04 nas kernel: [ 880] 0 880 6701 599 439 160 0 69632 0 -1000 systemd-udevd
Jun 25 20:06:04 nas kernel: [ 1843] 0 1843 866978 125405 125069 231 105 1556480 0 -1000 asyncio_loop
Jun 25 20:06:04 nas kernel: [ 1863] 0 1863 4196 1619 1472 147 0 73728 0 -1000 python3
Jun 25 20:06:04 nas kernel: [ 1902] 0 1902 922579 38572 38296 276 0 823296 0 -1000 middlewared (ze
Jun 25 20:06:04 nas kernel: [ 1911] 0 1911 9211 4519 4316 203 0 110592 0 -1000 python3
Jun 25 20:06:04 nas kernel: [ 3910] 0 3910 21735 237 66 171 0 57344 0 -1000 auditd
Jun 25 20:06:04 nas kernel: [ 3938] 0 3938 27525 4306 4128 178 0 122880 0 0 truenas_audit_h
Jun 25 20:06:04 nas kernel: [ 3941] 0 3941 5604 556 288 268 0 86016 0 0 certmonger
Jun 25 20:06:04 nas kernel: [ 3956] 0 3956 38191 179 32 147 0 61440 0 -1000 lxcfs
Jun 25 20:06:04 nas kernel: [ 3957] 0 3957 11738 257 96 161 0 69632 0 0 gssproxy
Jun 25 20:06:04 nas kernel: [ 3988] 0 3988 3078 694 512 182 0 65536 0 0 smartd
Jun 25 20:06:04 nas kernel: [ 4012] 0 4012 361315 1804 1597 207 0 663552 0 0 syslog-ng
Jun 25 20:06:04 nas kernel: [ 4026] 0 4026 4220 449 224 225 0 81920 0 0 systemd-logind
Jun 25 20:06:04 nas kernel: [ 4040] 0 4040 60178 520 288 232 0 98304 0 0 zed
Jun 25 20:06:04 nas kernel: [ 4049] 0 4049 691024 5348 5110 238 0 425984 0 -999 containerd
Jun 25 20:06:04 nas kernel: [ 4077] 0 4077 3910 475 256 219 0 65536 0 -1000 sshd
Jun 25 20:06:04 nas kernel: [ 4140] 0 4140 1005 184 32 152 0 49152 0 0 cron
Jun 25 20:06:04 nas kernel: [ 4226] 122 4226 3089 205 94 111 0 57344 0 0 usbhid-ups
Jun 25 20:06:04 nas kernel: [ 4292] 0 4292 91746 13529 13263 266 0 225280 0 0 cli
Jun 25 20:06:04 nas kernel: [ 4293] 122 4293 1695 286 64 222 0 53248 0 0 upsd
Jun 25 20:06:04 nas kernel: [ 4295] 0 4295 1223 200 32 168 0 53248 0 0 upsmon
Jun 25 20:06:04 nas kernel: [ 4296] 122 4296 2127 264 96 168 0 61440 0 0 upsmon
Jun 25 20:06:04 nas kernel: [ 4326] 131 4326 4715 338 138 200 0 57344 0 0 chronyd
Jun 25 20:06:04 nas kernel: [ 4334] 131 4334 2686 264 116 148 0 57344 0 0 chronyd
Jun 25 20:06:04 nas kernel: [ 4376] 0 4376 36116 13441 13249 192 0 196608 0 0 cli
Jun 25 20:06:04 nas kernel: [ 4388] 0 4388 7048 424 366 58 0 61440 0 0 nginx
Jun 25 20:06:04 nas kernel: [ 4389] 33 4389 7343 757 558 199 0 77824 0 0 nginx
Jun 25 20:06:04 nas kernel: [ 5288] 999 5288 132189 46758 46520 206 32 598016 0 -900 netdata
Jun 25 20:06:04 nas kernel: [ 5314] 999 5314 11686 438 192 246 0 77824 0 -900 netdata
Jun 25 20:06:04 nas kernel: [ 5603] 105 5603 1788 269 64 205 0 57344 0 0 avahi-daemon
Jun 25 20:06:04 nas kernel: [ 5656] 105 5656 1722 178 73 105 0 57344 0 0 avahi-daemon
Jun 25 20:06:04 nas kernel: [ 6315] 1 6315 8070 4209 3968 241 0 106496 0 0 wsdd.py
Jun 25 20:06:04 nas kernel: [ 6360] 999 6360 1113 305 128 177 0 45056 0 -900 bash
Jun 25 20:06:04 nas kernel: [ 6361] 999 6361 22247 4963 4760 203 0 139264 0 -900 python.d.plugin
Jun 25 20:06:04 nas kernel: [ 8306] 0 8306 18932 967 640 231 96 131072 0 0 winbindd
Jun 25 20:06:04 nas kernel: [ 8313] 0 8313 18978 871 639 196 36 131072 0 0 wb[NAS]
Jun 25 20:06:04 nas kernel: [ 8315] 0 8315 20141 1631 640 270 721 143360 0 0 smbd
Jun 25 20:06:04 nas kernel: [ 8329] 0 8329 19608 783 623 160 0 126976 0 0 smbd-notifyd
Jun 25 20:06:04 nas kernel: [ 8331] 0 8331 19612 726 623 103 0 131072 0 0 smbd-cleanupd
Jun 25 20:06:04 nas kernel: [ 8365] 0 8365 903614 8121 7937 184 0 614400 0 -500 dockerd
Jun 25 20:06:04 nas kernel: [ 11900] 0 11900 4557 811 563 248 0 77824 0 0 sshd
Jun 25 20:06:04 nas kernel: [ 11911] 0 11911 5051 1019 736 283 0 77824 0 100 systemd
Jun 25 20:06:04 nas kernel: [ 11912] 0 11912 42864 1719 1580 139 0 98304 0 100 (sd-pam)
Jun 25 20:06:04 nas kernel: [ 11952] 0 11952 1486 358 192 166 0 53248 0 0 zsh
Jun 25 20:06:04 nas kernel: [ 63369] 0 63369 644 123 0 123 0 49152 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 63371] 0 63371 644 69 0 69 0 49152 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 63373] 0 63373 644 69 0 69 0 49152 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 63374] 0 63374 644 69 0 69 0 49152 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 63376] 0 63376 644 148 0 148 0 49152 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 63377] 0 63377 4363 1942 1763 179 0 81920 0 -1000 ssh
Jun 25 20:06:04 nas kernel: [ 63378] 0 63378 644 47 0 47 0 49152 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 63380] 0 63380 644 47 0 47 0 49152 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 63382] 0 63382 536873819 8481827 8481697 130 0 72167424 0 -1000 mbuffer
Jun 25 20:06:04 nas kernel: [ 82282] 1001 82282 31078 9871 5892 206 3773 237568 0 0 smbd[192.168.1.
Jun 25 20:06:04 nas kernel: [ 82283] 0 82283 19478 854 639 151 64 131072 0 0 wb-idmap
Jun 25 20:06:04 nas kernel: [ 82285] 1001 82285 5039 909 704 205 0 81920 0 100 systemd
Jun 25 20:06:04 nas kernel: [ 140452] 0 140452 4105 521 224 297 0 77824 0 0 systemd-machine
Jun 25 20:06:04 nas kernel: [ 187513] 0 187513 206912 12847 12568 279 0 307200 0 -1000 middlewared (wo
Jun 25 20:06:04 nas kernel: [ 187514] 0 187514 208476 14392 14162 230 0 327680 0 -1000 middlewared (wo
Jun 25 20:06:04 nas kernel: [ 187523] 0 187523 754862 17723 17499 224 0 491520 0 -1000 middlewared (wo
Jun 25 20:06:04 nas kernel: [ 189175] 0 189175 226611 14126 13865 261 0 323584 0 -1000 middlewared (wo
Jun 25 20:06:04 nas kernel: [ 190740] 0 190740 28678 133 32 101 0 86016 0 0 nscd
Jun 25 20:06:04 nas kernel: [ 197450] 0 197450 206860 12900 12691 209 0 299008 0 -1000 middlewared (wo
Jun 25 20:06:04 nas kernel: [ 202970] 0 202970 644 101 0 101 0 45056 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 202972] 0 202972 644 44 0 44 0 45056 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 202973] 0 202973 644 44 0 44 0 45056 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 202974] 0 202974 644 44 0 44 0 45056 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 202975] 0 202975 644 113 0 113 0 40960 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 202977] 0 202977 3921 1515 1289 226 0 65536 0 -1000 ssh
Jun 25 20:06:04 nas kernel: [ 202979] 0 202979 644 45 0 45 0 40960 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 202980] 0 202980 644 45 0 45 0 40960 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 202981] 0 202981 644 45 0 45 0 40960 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 202982] 0 202982 644 132 0 132 0 45056 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 202983] 0 202983 536873819 22523360 22523182 178 0 184619008 0 -1000 mbuffer
Jun 25 20:06:04 nas kernel: [ 202984] 0 202984 644 82 0 82 0 45056 0 -1000 sh
Jun 25 20:06:04 nas kernel: [ 202985] 0 202985 21526 549 320 229 0 81920 0 -1000 zfs
Jun 25 20:06:04 nas kernel: [ 209408] 999 209408 623 98 0 98 0 45056 0 -900 timeout
Jun 25 20:06:04 nas kernel: [ 209409] 999 209409 1268 211 32 179 0 53248 0 -900 upsc
Jun 25 20:06:04 nas kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=system-getty.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/user@0.service/init.scope,task=(sd-pam),pid=11912,uid=0
Jun 25 20:06:04 nas kernel: upsc invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-900
Jun 25 20:06:04 nas kernel: CPU: 3 UID: 999 PID: 209409 Comm: upsc Tainted: P OE 6.12.15-production+truenas #1
Jun 25 20:06:04 nas kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Jun 25 20:06:04 nas kernel: Hardware name: Supermicro Super Server/X11SDV-8C+-TLN2F, BIOS 1.2 11/14/2019
PK1048
June 25, 2025, 7:14pm
11
Please provide full system configuration information.
Hardware: Motherboard, CPU, RAM, HBAs, NICs, disks, UPS, etc.
As much detail as possible.
But the 4 GiB is the limit, it’s wild to think that every single app would be using the max amount. If each app is blocking the memory listed in the Resources configuration section, the documentation ought to be changed from “Memory limit” to something else.
There’s also the bit where most of these apps weren’t running.
I added everything in the first post
PK1048
June 25, 2025, 7:30pm
14
You missed the second part of my reply where I said
PK1048:
Now, you may say, “Why, all of the Apps do not need all of their memory at once.”, which may be true, until they do.
Are you building a system that you hope is reliable or one that you expect to be reliable?
I did post this prior to seeing the post that it crashed with no Apps running. So there appears to be something else wrong with this system.
And I have asked again for the complete system configuration.
@OptimisticClearing you need to post very verbose details of what you are doing with this system and how you are trying to diagnose.
If you have turned off all the apps and reboot the system, how much memory is used? Are you showing almost everything is free? What tasks are you doing on this server? You mentioned ZFS replication was running? Is anything else running? Is this a home server or enterprise? What are you seeing when looking at TOP or HTOP for the memory?
You have to provide as much as you can as we aren’t at the console or GUI to see what is going on.
You may also want to take a look at your Applications tab in the GUI and see what is reported for memory and cpu stats
Looking at the apps, then yes, yes I would be surprised if every app used over 90% of the memory limit, which is what it would take for the system to go over 120GiB used.
Now, if there’s something else running on the system, like VMs, the the memory pressure increases, lowering what’s available for safe app usage.
The mbuffer numbers in that trace look interesting, but I am not sure how to intepret it, perhaps it’s benign, perhaps not.
More information is definitely going to be required.
PK1048
June 25, 2025, 7:40pm
17
But you are still not telling us the full story …
How is the storage laid out? (output from /sbin/zpool status works for this).
What jobs are running, you mentioned replication, what else, when are they configured to run, how often?
You are asking us to figure out why your system is crashing and not telling us what you are doing with it?
My backup TrueNAS which is an HP MicroProliant with a dual core N36L processor, 16 GiB of ECC RAM and is doing nothing but receiving ZFS replication to a single 4 x 4GB RAIDz1 vdev/pool and running the Pi-Hole App on 2 x 500GB SSDs whose OS is 25.04.1 and is installed on a pair of mirrored 120GB SSD has never crashed due to an out of memory condition.
1 Like
The oom killer output seems to pretty clearly show huge memory consumption by mbuffer. That’s something you’re doing on the system.
I am sorry - as I do not really know what could be helpful it is not easy for me to give all details that I do not no
root@nas[~]# /sbin/zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0B in 00:01:53 with 0 errors on Mon Jun 23 03:46:54 2025
config:
NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
ata-SuperMicro_SSD_SMC0515D94117B2F2367-part2 ONLINE 0 0 0
errors: No known data errors
pool: pool1
state: ONLINE
scan: scrub canceled on Wed Jun 25 18:26:56 2025
config:
NAME STATE READ WRITE CKSUM
pool1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
62751e9f-ada9-401e-bc43-517a990ebcb3 ONLINE 0 0 0
b8a4a657-891a-4247-b430-ab30251faaf1 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
a74b886e-eb80-44bb-b330-998cddfc7a8a ONLINE 0 0 0
ed90bca7-e4dd-442e-a34c-2c45ccfc9e66 ONLINE 0 0 0
errors: No known data errors
pool: pool2
state: ONLINE
scan: scrub repaired 0B in 00:19:25 with 0 errors on Sun Jun 8 00:19:40 2025
config:
NAME STATE READ WRITE CKSUM
pool2 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
0e3ac31e-51fa-4bb8-b60a-708be9c02c65 ONLINE 0 0 0
cdf8a1bf-c79a-4b2b-ace8-7e015163d97e ONLINE 0 0 0
errors: No known data errors
The server is just used as a NAS for home usage.
Some of the datasets are replicated via ZFS send/rcv to rsync.net .
It is now running for more than 1 hour
I kept the replication off.
authelia-old
caddy
home-assistant
plex
stash
I hope that’s all.
Look at your Applications tab in the GUI and see what resources you are using while the apps are running. Is Plex doing anything or is it just idle? Do you have scheduled tasks or maintenance in Plex?
Keep an eye on the memory and what tasks you are performing. It could help to know your pool size and how full it is. Maybe how much data you are backing up?
