I’m a new and novice TrueNAS user, so please excuse my posting if it’s not entirely up to standard.
I recently had a memory-heavy user process (a scheduled restic backup/prune job, unrelated to TrueNAS itself) exhaust available RAM on my system. The root cause turned out to be entirely on the user-script side (confirmed via the restic community — the process needed more memory than was available for its own internal data structures), so this isn’t really about that specific tool.
What I want to ask about is what happened during the OOM event itself. Rather than the kernel OOM-killer targeting just the offending process, it cascaded and killed a wide range of core system services in the same event — smbd, dockerd, winbindd, syncthing, systemd-journal, and systemd-logind among others — which effectively forced a full reboot to recover.
Environment: TrueNAS SCALE, 8GB RAM, Intel i3-1315U (UGREEN DXP4800 Pro)
Relevant journalctl -b -1 -p err excerpt from the incident:
Jul 05 22:52:19 truenas kernel: Out of memory: Killed process 3034 (cli)...
Jul 05 22:52:19 truenas kernel: Out of memory: Killed process 2320721 (syncthing)...
Jul 05 22:52:19 truenas kernel: Out of memory: Killed process 4750 (dockerd)...
Jul 05 22:52:19 truenas kernel: Out of memory: Killed process 4502 (winbindd)...
Jul 05 22:52:20 truenas kernel: Out of memory: Killed process 655 (systemd-journal)...
Jul 05 22:52:20 truenas kernel: Out of memory: Killed process 2620 (systemd-logind)...
[~30 processes killed within the same second or two]
Questions:
-
Is this expected/normal OOM behavior on TrueNAS SCALE, or does the platform have (or intend to have) any protection for core services via
oom_score_adjor cgroup memory limits, so a single runaway user process can’t take down the whole system? -
If not built-in, is there a supported/recommended way for users to set this up themselves — e.g. wrapping cron-scheduled user scripts with a memory cgroup limit (via systemd-run, a slice, or similar) so they get OOM-killed in isolation rather than the kernel picking targets system-wide?
-
Has anyone else experienced a similar cascade, or is this specific to my hardware/RAM constraints (8GB) rather than something others hit more commonly on larger systems?
Not looking to relitigate the original memory issue (that’s been solved) — mainly curious about hardening against this kind of cascade in general, since it’s a rougher failure mode than I’d expect from an appliance-oriented platform.