middlewared core-dump on ddt_prefetch_pools with dedup=off - 25.10.3.1

TrueNAS General
, ,
1

TrueNAS Version: 25.10.3.1
Hardware: Intel i3-12100, 32GB RAM, NVMe boot drive

Problem:
middlewared crashes with core-dump multiple times per day (3-5x).
All my pools have dedup=off, confirmed with “zfs get dedup” on each pool.

Pools:

  • Apps (NVMe stripe)
  • Media (7x12TB RAIDZ2)
  • Famille (2x4TB mirror)

The pattern is 100% reproducible: ddt_prefetch_pools() runs on all 3 pools,
then middlewared crashes seconds later. This happens every few hours and
takes Docker down with it, making all my containers unavailable.

Logs:

Jun 01 18:54:21 middlewared: worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 01 18:54:21 middlewared: worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 01 18:54:21 middlewared: worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 01 18:54:39 systemd: middlewared.service: Failed with result ‘core-dump’.

Jun 02 16:45:49 middlewared: worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 02 16:45:49 middlewared: worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 02 16:45:49 middlewared: worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 02 16:46:04 systemd: middlewared.service: Failed with result ‘core-dump’.

Full logs over 2 days (same pattern repeating):

Jun 01 18:54:21 truenas middlewared[2043]: [2026/06/01 18:54:21] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 01 18:54:21 truenas middlewared[2043]: [2026/06/01 18:54:21] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 01 18:54:21 truenas middlewared[2043]: [2026/06/01 18:54:21] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 01 18:54:39 truenas systemd[1]: middlewared.service: Failed with result ‘core-dump’.
Jun 01 18:58:58 truenas middlewared[2063]: [2026/06/01 18:58:58] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 01 18:58:58 truenas middlewared[2063]: [2026/06/01 18:58:58] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 01 18:58:58 truenas middlewared[2063]: [2026/06/01 18:58:58] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 02 08:15:38 truenas systemd[1]: middlewared.service: Failed with result ‘core-dump’.
Jun 02 08:15:49 truenas systemd[1]: middlewared.service: Failed with result ‘core-dump’.
Jun 02 10:33:43 truenas middlewared[1933]: [2026/06/02 10:33:43] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 02 10:33:43 truenas middlewared[1933]: [2026/06/02 10:33:43] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 02 10:33:43 truenas middlewared[1933]: [2026/06/02 10:33:43] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 02 10:38:11 truenas middlewared[1936]: [2026/06/02 10:38:11] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 02 10:38:11 truenas middlewared[1936]: [2026/06/02 10:38:11] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 02 10:38:11 truenas middlewared[1936]: [2026/06/02 10:38:11] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 02 16:13:40 truenas middlewared[1933]: [2026/06/02 16:13:40] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 02 16:13:40 truenas middlewared[1933]: [2026/06/02 16:13:40] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 02 16:13:40 truenas middlewared[1933]: [2026/06/02 16:13:40] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 02 16:27:06 truenas middlewared[1936]: [2026/06/02 16:27:06] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 02 16:27:06 truenas middlewared[1936]: [2026/06/02 16:27:06] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 02 16:27:06 truenas middlewared[1936]: [2026/06/02 16:27:06] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 02 16:45:49 truenas middlewared[1964]: [2026/06/02 16:45:49] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 02 16:45:49 truenas middlewared[1964]: [2026/06/02 16:45:49] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 02 16:45:49 truenas middlewared[1964]: [2026/06/02 16:45:49] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 02 16:46:04 truenas systemd[1]: middlewared.service: Failed with result ‘core-dump’.
Jun 02 17:09:53 truenas middlewared[1933]: [2026/06/02 17:09:53] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 02 17:09:53 truenas middlewared[1933]: [2026/06/02 17:09:53] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 02 17:09:53 truenas middlewared[1933]: [2026/06/02 17:09:53] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 02 17:12:17 truenas middlewared[1941]: [2026/06/02 17:12:17] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 02 17:12:17 truenas middlewared[1941]: [2026/06/02 17:12:17] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 02 17:12:17 truenas middlewared[1941]: [2026/06/02 17:12:17] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool
Jun 02 17:30:08 truenas middlewared[1937]: [2026/06/02 17:30:08] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Apps’ pool
Jun 02 17:30:08 truenas middlewared[1937]: [2026/06/02 17:30:08] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Media’ pool
Jun 02 17:30:08 truenas middlewared[1937]: [2026/06/02 17:30:08] (INFO) worker.ddt_prefetch_pools():244 - Prefetching ddt table of ‘Famille’ pool

Is there a workaround to disable ddt_prefetch on pools without dedup enabled?

2

Did you create your pools with dedup enabled, and then disabled dedup afterwards? You may need to export and reimport your pools to get this to stick.

3

Thank you for your response!

Just to clarify: this is a fresh TrueNAS installation from 3-4 days ago.
I have never enabled deduplication on any of my pools since their creation,
so dedup has always been off.

I tried your suggestion and exported/reimported all pools, but the issue persists.

After more investigation, I found a second type of crash that seems unrelated
to DDT:

Jun 03 03:05:01 middlewared: Main process exited, code=dumped, status=11/SEGV
Jun 03 03:05:01 middlewared: resource_tracker: There appear to be 12 leaked
semaphore objects to clean up at shutdown
Jun 03 03:05:01 middlewared: Failed with result ‘core-dump’

This SEGV crash happens independently of the DDT prefetch, after middlewared
has been running for some time.

So it seems there are actually two separate issues:

  1. DDT prefetch crashing on pools with dedup=off and empty DDT tables
    (confirmed with zdb -D: “All DDTs are empty”)
  2. A segfault with semaphore leaks after some uptime

Both result in middlewared core-dump and Docker becoming unavailable.

Do you have any idea where this could come from? Apart from a fairly large
upload on Immich (which is resource-limited in its compose to avoid
overloading the CPU with machine learning), the NAS is not doing much
besides SMB shares that are always active.

Thank you in advance!

4

You’ve got hardware issues. Either your power supplies aren’t providing enough juice or your RAM is suspect. Please provide a full manifest of your hardware.

5

Here is my full hardware manifest:

  • Motherboard: Asus Prime B760 Plus D4

  • CPU: Intel i3-12100 (65W TDP)

  • RAM: 32GB DDR4

  • PSU: Corsair RM650e Platinum (650W)

  • Boot drive: Emtec 128GB NVMe

  • Apps pool: WD Black SN7100 1TB NVMe

  • PCIe expansion card: 2x PCIe 3.0x1 to 4-SATA

Storage (9 HDDs total):

  • 7x 12TB HDD

  • 2x 4TB HDD

PSU cabling for HDDs:

  • 1 cable with 2 SATA + 2 Molex→SATA adapters: powers 5 HDDs + 1 fan

  • 1 cable with 4 SATA connectors: powers 4 HDDs

Could the PSU be undersized for this configuration, or could the 5 HDDs on a single cable be causing voltage drops that lead to these crashes?

6

It’s your 4-port SATA expansion card. Those are notorious for not playing nice with ZFS. You’ll need to replace it with a real HBA in IT mode.

Also, most consumer power supplies prioritize the 12V rail for CPU/GPU performance, and skimp on the 5V rail for drives, since they only need to support 2 drives at most. Server power supplies prioritize the 5V rail for drive cages. Your sharing pigtails to power multiple drives off a single power supply output is also not helping. Your power supply is probably not able to continuously supply clean power to all the drives.

7

Building a server is nothing like building a run-of-the-mill gaming machine. The key requirement for a server is reliability, not performance. This starts with server-grade components: motherboard with IPMI, ECC DRAM, server-grade power supply, NAS-grade drives, etc.

This means reading specs for your components and using manufacturer recommendations for validated CPUs and DRAM. Throwing together random components isn’t a recipe for success.

8

Thank you for all this information,

I had been using these components for 6 months with only 4x12TB and 2x4TB HDDs without any issues. The problems started when I moved everything to a new case and added 3 more HDDs + 1 NVMe. The HDDs are indeed NAS-grade drives, but the rest (motherboard, RAM, PSU) came from my old gaming PC.

Based on your advice, the first steps would be:

  1. Replace my 2x PCIe SATA expansion cards with a proper LSI SAS SATA 9211-8i IT mode for example ? (found some on eBay)
  2. Find a more suitable power supply for this configuration

Do you have any specific recommendations for PSU brands/models that would be better suitted for a NAS with 9 HDDs ?

Also, would you recommend replacing the motherboard and RAM urgently, or would replacing the SATA expansion card and PSU first already be a significant improvement that would allow changing the rest of the components gradually over time ?

Thank you again for taking the time to help

9

Seasonic is a good power supply for servers. I would start off with the power supply and the HBA to stabilize your setup. The motherboard, RAM, and CPU can come later. Note, the HBA will require quite a lot of cooling, so see if you can repurpose an old AIO CPU liquid cooler or provide some sort of duct to redirect air from an existing case fan.

10

Great, thank you so much!

I will make sure to pay attention to the LSI card cooling and will replace both components within the week. The rest can wait but I’ll keep it in mind.

Thanks again, hoping these changes will fix all the issues I’ve been experiencing these past few days !

I will come back to validate the solution on this topic once the components are replaced, to confirm whether this was the root cause or not.

11

Hello,

Back after replacing the hardware and doing a full reinstall of TrueNAS. I re-imported my pools without any issues, and so far I haven’t experienced a single problem, crash, or error — so it was most likely a hardware issue causing all my troubles, just as you had suggested.

Thank you again for your analysis and your advice. I’ll be more careful about the brands and models I use from now on.

All the best