Middlewared repeatedly crashes with SIGSEGV during active UI use

TrueNAS General
1

I spend quite some time (assisted with AI) to research this bug. Hope this info can contribute to keep improving this project.

Summary to

middlewared crashes intermittently with a segmentation fault (SIGSEGV), most often while I am actively using the web UI. The service is automatically restarted by systemd, but this causes repeated forced logouts and ~15-20 second delays before the UI becomes usable again. This has been occurring since a fresh install and persists after updating BIOS and confirming RAM integrity, so hardware causes have largely been ruled out (see below).

Both captured coredumps crash in the exact same code path: Python’s _Py_dg_dtoa (double-to-string conversion) called from the _json module, apparently while serializing floating-point statistics (likely CPU/sensor/reporting data) for the UI over the websocket API.

Environment

  • TrueNAS version: 25.10.5 (Goldeye) — fresh install, not upgraded from an earlier branch
  • Motherboard: ASRock H610M-HDV/M.2 R2.0
  • BIOS: Updated from v8.02 to v11.01 (latest available) during troubleshooting — issue persisted after update, in the same boot session
  • CPU: Intel Core i5-13500 (6P+8E cores, 20 threads)
  • RAM: 2x 8GB Kingston FURY Beast DDR4 (KF3200C16D4/8GX), running at 2400 MT/s (rated 3200 MT/s), non-ECC — tested with PassMark Memtest86, 2+ hours, 0 errors on both modules
  • Boot/OS drive: Patriot M.2 P310 240GB NVMe — SMART status clean (Critical Warning: 0x00, Available Spare: 100%, Percentage Used: 0%, Media and Data Integrity Errors: 0, no kernel I/O errors logged)
  • Storage pool drives: 3x Western Digital Red (2x mirror + 1x single), all connected directly to motherboard SATA ports (no HBA/expansion card)
  • PSU: be quiet! Pure Power 10M 550W
  • Apps running: Immich (immich-server, postgres/pgvecto, machine-learning, redis containers), immich_power_tools

What’s been ruled out

  • RAM (Memtest86, clean, 2+ hour run)
  • BIOS/microcode outdated (updated to latest, issue persists)
  • NVMe drive failure (SMART clean, no kernel-level I/O errors)
  • ZFS dedup (confirmed off on all datasets — ruling out the known ddt_prefetch_pools() crash pattern)
  • SATA expansion card / PSU rail issues (not applicable — direct motherboard SATA, adequate PSU, only 3 drives on a daisy-chained power cable)

Observed pattern

  • Crashes correlate strongly with having the web UI open and active in a browser tab (not confirmed to be exclusively UI-triggered, but the correlation is consistent)
  • systemctl status middlewared shows a climbing restart counter (observed up to 7 restarts) with very short uptimes between crashes
  • NVMe SMART log shows 114 “Unsafe Shutdowns”, consistent with these repeated forced restarts

Crash 1 (before BIOS update)

Signal: 11 (SIGSEGV)
Command: middlewared
Executable: /usr/bin/python3.11

Stack trace of crashing thread:
#0 __memset_avx2_unaligned_erms (libc.so.6 + 0x15344a)
#1  n/a (python3.11 + 0x1e1a97)
#2  n/a (python3.11 + 0x1e20a0)
#3  _Py_dg_dtoa (python3.11 + 0x1e5c7d)
#4  PyOS_double_to_string (python3.11 + 0x1e0531)
#5  n/a (python3.11 + 0x9f55e)
#6  n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x4e36)
#7  n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x51e2)
#8  n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x51e2)
#9  n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x57a0)
#10 _PyObject_MakeTpCall (python3.11 + 0x811ad)
#11 _PyEval_EvalFrameDefault (python3.11 + 0x265ab)
...
#16 n/a (_asyncio.cpython-311-x86_64-linux-gnu.so + 0x8c89)
#17 n/a (_asyncio.cpython-311-x86_64-linux-gnu.so + 0x987f)
...

Also logged at time of crash:

resource_tracker: There appear to be 12 leaked semaphore objects to clean up at shutdown

Crash 2 (confirmed AFTER BIOS update to v11.01, same boot session — boot at 22:22:55, crash at 23:10:35)

Signal: 11 (SIGSEGV)
Command: middlewared
Executable: /usr/bin/python3.11

Stack trace of crashing thread:
#0  arena_for_chunk (libc.so.6 + 0x98f45)
#1  n/a (python3.11 + 0xdb96f)
#2  _Py_dg_dtoa (python3.11 + 0x1e55ac)
#3  PyOS_double_to_string (python3.11 + 0x1e0531)
#4  n/a (python3.11 + 0x9f55e)
#5  n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x4e36)
#6  n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x51e2)
#7  n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x51e2)
#8  n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x51e2)
#9  n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x51e2)
#10 n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x51e2)
#11 n/a (_json.cpython-311-x86_64-linux-gnu.so + 0x57a0)
#12 _PyObject_MakeTpCall (python3.11 + 0x811ad)
#13 _PyEval_EvalFrameDefault (python3.11 + 0x265ab)
... (recursive PyObject_Call / _PyEval_EvalFrameDefault chain, consistent with
     nested JSON serialization of a large/deeply-nested dict) ...

Also logged at time of crash:

resource_tracker: There appear to be 12 leaked semaphore objects to clean up at shutdown

Hypothesis

Both crashes occur inside CPython’s _Py_dg_dtoa (David Gay’s double-to-ASCII algorithm), invoked from _json while encoding floating-point values — most likely system/hardware statistics (CPU load, temperatures, fan speeds, etc.) being pushed to the UI. The two crashes differ in exact fault location (memset vs. glibc malloc arena internals) but hit the identical function chain, suggesting heap corruption that surfaces at slightly different points depending on allocator state, rather than two unrelated bugs.

One possible trigger I have not been able to fully verify: this is a consumer desktop motherboard (no BMC/IPMI), and hardware sensors are read via the Super I/O chip through lm-sensors. Consumer boards are known to sometimes report malformed/out-of-range sensor values (unlike server-grade hardware with clean BMC-reported sensor data), which could produce an edge-case float value that _Py_dg_dtoa mishandles. I have not yet captured a sensors output showing an obviously malformed reading at the exact moment of a crash, so this is a hypothesis, not a confirmed root cause.

Request

  • Has this crash signature (SIGSEGV in _Py_dg_dtoa via _json serialization) been reported before?
  • Any guidance on what statistics/data specifically was being serialized at the crash point would help narrow this down further.
  • Happy to provide full coredumps (both are still available via coredumpctl) if useful — omitted here for brevity/size.
2

which version of truenas do you use?
I believe 25.10.5 fixes a memory leak issue, maybe you’re affected by that issue
From the release notes of 25.10.5:

Edit: ok, i just saw that you already use 25.10.5

3

Update — new coredump captured, widget removal did not fix it

Captured a fresh coredump since my original post. Same crash signature, this time aborting via SIGABRT/munmap_chunk(): invalid pointer instead of SIGSEGV:

#4  malloc_printerr (libc.so.6 + 0x9486a)
#5  munmap_chunk (libc.so.6 + 0x94a2c)
#6  __GI___libc_free (libc.so.6 + 0x98f98)
#7  n/a (python3.11 + 0xdb96f)
#8  _Py_dg_dtoa (python3.11 + 0x1e55ac)
#9  PyOS_double_to_string (python3.11 + 0x1e0531)
#10 n/a (python3.11 + 0x9f55e)
#11-16 n/a (_json.cpython-311-x86_64-linux-gnu.so, repeated — nested JSON encoding)
#18 _PyObject_MakeTpCall
#19 _PyEval_EvalFrameDefault

Identical dtoa/_json path as before — just a different fault type (SIGABRT vs SIGSEGV), consistent with heap corruption surfacing at slightly different points depending on allocator state.

I removed the CPU/temperature dashboard widgets to test the sensor-value hypothesis. Crashes continued the next day regardless. That points away from a specific widget or sensor reading and toward JSON serialization of floats in general (memory %, storage %, network throughput, etc. all go through the same _json/dtoa code path) — so I’m now leaning toward general heap corruption that happens to surface most often here, rather than a malformed-sensor-value trigger.

4

I don’t experience this with my ASRockRack server board. Since you’re using a consumer gaming-oriented board, have you disabled any automatic overclocking in the BIOS?

5

It occurred to me you’re using an asymmetric CPU with P/E cores. The intel-microcode package bundled with TrueNAS may not be current enough for this CPU. On my 25.10.6 system, I have intel-microcode package version 3.20250512.1~deb12u1. Latest from upstream Debian is 3.20251111.1~deb12u1.