Beelink ME Mini, Parts of TrueNAS Regularly Become Unresponsive

TrueNAS General
,
1

Hello all! This is my first post so please let me know if there is something I can do better.

  • Update 1
    To be super clear, when the system is NOT under load, some job (such as catalog.sync) apepars hung and new jobs can’t start. I don’t know how to find out more about whatever the job is or what is causing it to hang. The entire system typically doesn’t just become unresponsive unless I try to do a soft reset/shutdown, then the whole system becomes unresponsive. It starts the process and kills most services but never actually shuts down or restarts. I’m forced to hard shutdown the device.

Last summer I purchased a Beelink ME mini and a few NVME drives to try out TrueNAS.

Product: ME mini

Model: Intel(R) N150

Memory: 11 GiB

VDEVs: 1 x RAIDZ2 | 5 wide | Mixed Capacity

Though all five drives are the same model (PNY CS1030 2TB), I ordered the 5th drive later in the summer (now nvme5) and it is technically of a slightly higher capacity (1.86 vs. the others at 1.82) so it is technically “Mixed Capacity”.

I kept having issues - services or apps became somewhat if not completely unresponsive every few days. I’d often find catalog.sync ‘Updating catalog repository’ as a running job at a very low %, often started hours if not days ago. If tried to reboot or shutdown the OS from the GUI, it would stop responding entirely and never reboot/shutdown, at least after waiting for 5-10 minutes.

Sometimes I’d check and see an app needed an update, so I’d update it. The job would occasionally just hang indefinitely. Again, I’d give up and try to reboot/shutdown and again it would simply become unresponsive. It mostly sat idle with a few infrequent attempts to troubleshoot.

A few weeks ago I tried to revisit the issue and decided a fresh install of the latest version might fix things - some forum post somewhere mentioned an issue after upgrading from the version I’d first installed where some outdated scripts kept running and causing issues.

In the last two weeks it happened at least three times. Unfortunately, I’ve never had an issue start while I was actively using it. I would go to use it for the first time in a day or two and find it unresponsive so I don’t know exactly when the problems start. There is no log in the GUI that I can find, so I’ve tried journalctl -p err but I don’t see anything that is obvious to me but I’m not too experienced digging through those kinds of logs or recognizing important entries. I’ve not found anything conclusive searching for the text of these errors but again maybe it’s a failure on my part - perhaps it’s because I am foolishly resisting our technology overlords’ commandment to abandon the old ways and direct all queries to their AI chat bots.

I’ve verified that the OS is installed on nvme4 (the only x2 m.2) as per the manufacturer’s diagram and the BIOS. The time in the BIOS is set to UTC. I forced an extended SMART test on all drives and nvme3 & nvme1 both showed failed segments with 7 in the ‘Seg’ column in the current and a few (but not all) historic tests while nvme2 had one historic test 1300 ‘Power_on_Hours’ ago with failed segments showing 7. I’m getting very little on what ‘failed segment’ even means when I look for information.

Drive Pool Error Hours Read Written
nvme0 zpool 155 1.90 TB 1.28 TB
nvme1 zpool failed segments 7 5,721 1.90 TB 1.28 TB
nvme2 zpool 5,977 1.88 TB 1.28 TB
nvme3 zpool failed segments 7 5,831 1.74 TB 1.27 TB
nvme4 boot-pool 6,163 587 GB 390 GB
nvme5 zpool 156 1.74 TB 1.27 TB

I’m curious how nvme0 shows fewer hours than 1, 2, and 3. Perhaps my initial pool last year only had 3 disks? I honestly don’t remember.

Here is the entirety of journal -p err for the first three boot cycles of running after reinstalling. I believe the issue happened twice. I rebooted through the GUI on the 8th. The next two reboots were due to unresponsiveness.

Jun 08 18:48:42 localhost kernel: proc_thermal_pci 0000:00:04.0: error: proc_thermal_add, will continue
Jun 08 18:48:42 localhost kernel: Error: Driver 'pcspkr' is already registered, aborting...
Jun 08 18:48:43 localhost kernel: usb 3-5.3.4: 2:1: cannot get freq at ep 0x1
Jun 08 18:49:55 truenas systemd[1]: ipa-epn.timer: Refusing to start, unit ipa-epn.service to trigger not loaded.
Jun 08 18:49:55 truenas systemd[1]: Failed to start ipa-epn.timer - Execute IPA Expiring Password Notification (EPN) every day at 1AM.
Jun 08 19:29:34 truenas systemd[1]: Failed unmounting var-log-journal.mount.
Jun 08 19:29:34 truenas systemd[1]: Failed unmounting var-log.mount.
Jun 08 19:29:34 truenas systemd[1]: Failed unmounting var.mount.
Jun 08 19:29:34 truenas kernel: watchdog: watchdog0: watchdog did not stop!
-- Boot bfa874b796ea411c9b1b73ba3edf5ae9 --
Jun 08 19:30:17 truenas kernel: proc_thermal_pci 0000:00:04.0: error: proc_thermal_add, will continue
Jun 08 19:30:17 truenas kernel: Error: Driver 'pcspkr' is already registered, aborting...
Jun 08 19:30:59 truenas systemd[1]: ipa-epn.timer: Refusing to start, unit ipa-epn.service to trigger not loaded.
Jun 08 19:30:59 truenas systemd[1]: Failed to start ipa-epn.timer - Execute IPA Expiring Password Notification (EPN) every day at 1AM.
Jun 13 11:46:22 Mini-NAS kernel: INFO: task txg_sync:1270 blocked for more than 120 seconds.
Jun 13 11:46:22 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 11:46:22 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 13 11:50:24 Mini-NAS kernel: INFO: task IoThread:1090 blocked for more than 120 seconds.
Jun 13 11:50:24 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 11:50:24 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 13 11:52:25 Mini-NAS kernel: INFO: task IoThread:1090 blocked for more than 241 seconds.
Jun 13 11:52:25 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 11:52:25 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 13 11:54:25 Mini-NAS kernel: INFO: task IoThread:1090 blocked for more than 362 seconds.
Jun 13 11:54:25 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 11:54:25 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 13 11:56:26 Mini-NAS kernel: INFO: task IoThread:1090 blocked for more than 483 seconds.
Jun 13 11:56:26 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 11:56:26 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 13 11:58:27 Mini-NAS kernel: INFO: task IoThread:1090 blocked for more than 604 seconds.
Jun 13 11:58:27 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 11:58:27 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 13 12:00:28 Mini-NAS kernel: INFO: task IoThread:1090 blocked for more than 724 seconds.
Jun 13 12:00:28 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 12:00:28 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 13 12:02:29 Mini-NAS kernel: INFO: task IoThread:1090 blocked for more than 845 seconds.
Jun 13 12:02:29 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 12:02:29 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 13 12:04:30 Mini-NAS kernel: INFO: task IoThread:1090 blocked for more than 966 seconds.
Jun 13 12:04:30 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 12:04:30 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 13 12:06:30 Mini-NAS kernel: INFO: task IoThread:1090 blocked for more than 1087 seconds.
Jun 13 12:06:30 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 12:06:30 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
-- Boot ffcb75f53bba42a9bfaeec937eba8aae --
Jun 13 15:10:53 Mini-NAS kernel: proc_thermal_pci 0000:00:04.0: error: proc_thermal_add, will continue
Jun 13 15:10:53 Mini-NAS kernel: Error: Driver 'pcspkr' is already registered, aborting...
Jun 13 15:11:35 Mini-NAS systemd[1]: ipa-epn.timer: Refusing to start, unit ipa-epn.service to trigger not loaded.
Jun 13 15:11:35 Mini-NAS systemd[1]: Failed to start ipa-epn.timer - Execute IPA Expiring Password Notification (EPN) every day at 1AM.
Jun 18 12:22:59 Mini-NAS kernel: INFO: task txg_sync:1247 blocked for more than 120 seconds.
Jun 18 12:22:59 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:22:59 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 18 12:29:02 Mini-NAS kernel: INFO: task python.d.plugin:4158 blocked for more than 120 seconds.
Jun 18 12:29:02 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:29:02 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 18 12:31:03 Mini-NAS kernel: INFO: task python.d.plugin:4158 blocked for more than 241 seconds.
Jun 18 12:31:03 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:31:03 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 18 12:33:04 Mini-NAS kernel: INFO: task python.d.plugin:4158 blocked for more than 362 seconds.
Jun 18 12:33:04 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:33:04 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 18 12:35:04 Mini-NAS kernel: INFO: task python.d.plugin:4158 blocked for more than 483 seconds.
Jun 18 12:35:04 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:35:04 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 18 12:37:05 Mini-NAS kernel: INFO: task python.d.plugin:4158 blocked for more than 604 seconds.
Jun 18 12:37:05 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:37:05 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 18 12:39:06 Mini-NAS kernel: INFO: task python.d.plugin:4158 blocked for more than 724 seconds.
Jun 18 12:39:06 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:39:06 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 18 12:41:07 Mini-NAS kernel: INFO: task python.d.plugin:4158 blocked for more than 845 seconds.
Jun 18 12:41:07 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:41:07 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 18 12:43:08 Mini-NAS kernel: INFO: task python.d.plugin:4158 blocked for more than 966 seconds.
Jun 18 12:43:08 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:43:08 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 18 12:45:09 Mini-NAS kernel: INFO: task python.d.plugin:4158 blocked for more than 1087 seconds.
Jun 18 12:45:09 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:45:09 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
-- Boot 7b8252a5febe4f0298a37309f965d165 --

I’ve found similar log entries, not flagged as errors, repeating with the eid incrementing. This might be significant since the ‘delay’ is often VERY long but I’ve found very few mentions of class=deadman to understand what it means.

Jun 13 12:17:28 Mini-NAS zed[1807026]: eid=321 class=deadman pool='zpool' vdev=29dc30bb-f85a-4a34-a379-b98ec4f345bf size=20480 offset=103349219328 priority=0 err=0 flags=0x300080 delay=405391406ms bookmark=438:28743:0:36
...
Jun 13 14:42:52 Mini-NAS zed[1843557]: eid=463 class=deadman pool='zpool' vdev=29dc30bb-f85a-4a34-a379-b98ec4f345bf size=20480 offset=103349219328 priority=0 err=0 flags=0x300080 delay=405391406ms bookmark=438:28743:0:36

Jun 18 14:27:43 Mini-NAS zed[1777100]: eid=115 class=deadman pool='zpool' vdev=25c9adc9-6f70-422d-a816-d3a2f28676e0 size=4096 offset=580958289920 priority=1 err=0 flags=0x300080 delay=428807657ms bookmark=3209:0:-2:17717
...
Jun 19 14:36:40 Mini-NAS zed[2107205]: eid=1530 class=deadman pool='zpool' vdev=25c9adc9-6f70-422d-a816-d3a2f28676e0 size=4096 offset=580958289920 priority=1 err=0 flags=0x300080 delay=428807657ms bookmark=3209:0:-2:17717

Jun 20 03:10:36 Mini-NAS zed[182956]: eid=111 class=deadman pool='zpool' vdev=f6730fb0-c854-4088-923a-60eddd72c850 size=36864 offset=53482278912 priority=0 err=0 flags=0x300080 delay=42126899ms bookmark=438:26386:0:13
...
Jun 20 12:02:04 Mini-NAS zed[307261]: eid=630 class=deadman pool='zpool' vdev=f6730fb0-c854-4088-923a-60eddd72c850 size=36864 offset=53482278912 priority=0 err=0 flags=0x300080 delay=42126899ms bookmark=438:26386:0:13

This morning I wanted to try something so I enabled containers on the pool.

Jun 21 08:35:01 Mini-NAS systemd[1]: Started lxcfs.service - FUSE filesystem for LXC.
Jun 21 08:35:01 Mini-NAS systemd[1]: Starting incus.service - Incus - Main daemon...
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: Running constructor lxcfs_init to reload liblxcfs
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: mount namespace: 5
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: hierarchies:
Jun 21 08:35:01 Mini-NAS lxcfs[302018]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: Kernel supports pidfds
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: Kernel does not support swap accounting
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: api_extensions:
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - cgroups
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - sys_cpu_online
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - proc_cpuinfo
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - proc_diskstats
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - proc_loadavg
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - proc_meminfo
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - proc_stat
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - proc_swaps
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - proc_uptime
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - proc_slabinfo
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - shared_pidns
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - cpuview_daemon
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - loadavg_daemon
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: - pidfds
Jun 21 08:35:01 Mini-NAS lxcfs[302018]: Ignoring invalid max threads value 4294967295 > max (100000).
Jun 21 08:35:01 Mini-NAS kernel: NET: Registered PF_VSOCK protocol family

The GUI had two jobs running indefinitely and the containers UI was unresponsive until I eventually shut it down 40 minutes later. No subsequent log entries from lxcfs or kernel until after it restarted. This is behavior similar to the initial problems I’d experience before I reinstalled.

Any help would be greatly appreciated.

Updates:

I can’t rule out a power issue but I don’t think it’s likely. The issues don’t occur under load it may be a power issue, though I want to explore other options.

2

You need to watch the power budget of the system. Find the wattage of the SSDs, multiply by 5, that probably exceeds the specs of the Mini. There are half a dozen or so threads about that on the forum, already including numbers.

EDIT: you can use this command to get the power levels of your first drive:

nvme id-ctrl /dev/nvme0n1
2 Likes
3

Thank you for your response! I had been aware of the power issue, though I forgot about it until you mentioned it. I remember doing the math but I can’t find any official ‘typical’ usage power consumption for the drives. I purposely didn’t aim for performance drives because why bother when each m.2 slot is only PCIe Gen3x1?

However, according to the firmware, the CS1030 does have a max of 6.50W at PS0, and the boot drive has a max of 4.50W as PS0.

So yes if all six drives, including the boot drive, were pushed to the literal limit and maxing out Power State 0, they could theoretically pull 37W, and if the N150 is also full title that is another 6W which is awfully close to 45W and I’m sure other components use power as well and would put it well over the capabilities of the little device.

However…

  • I’ve never had a spontaneous reboot, as most people reporting power issues have experienced
  • Besides the boot drive (SP UD90 with a max of 4.50W), the CS1030’s are all Gen3 in x1 slots, which as I understand it would make it literally impossible to run these things anywhere near full tilt
  • I don’t push this thing very hard - I copy media to it occasionally and stream audio or video to a single client, maybe two if my SO actually uses the thing once in a blue moon
  • As I mentioned, I’ve never experienced an issue with the device while utilizing it (while it was under load) - I just come back to it and find it stuck on some task or an app or a service like smb is unresponsive
  • The same issues originally occurred using only 3 of the CS1030 drives, which would have dropped the max draw by 13W to 24W

But, I’m not an expert so maybe I’m confused on what causes a drive to draw more power? I am assuming that there is no way to monitor the actual power draw, correct?

I’m going to keep it in mind for sure, but focus on other possible problems. If I get desperate I’ll start from scratch, yet again, and pull the two newly added drives from the pool - or maybe try just mirroring with two of the CS1030’s - and see if the problems continue.

For now, I’d love any other ideas, and explanation of what I revealed from the log, or any way I can actually see an error that hints at why things are just spontaneously hung or unresponsive.

4

Is 11GiB of RAM correct above? What apps or containers are you using? You can try looking on the manufacturer website of your NVMe and see what diag software they have. You may end up having to install on a Windows machine and test the NVMe one by one while having a degraded pool. I don’t think any of the manufactures have a live Linux version of the tools for diag of their NVMe.

5

I just copied that system info with “11 GiB” from the Support section under General Settings. Technically, it shows as 11.5GB for some reason (update: memtest86 shows 11.7GB!) but allegedly the device should have 12GB. I think the newer version has 16GB though.

Tangentially related, I’m actually going to test the memory - I saw someone recommend that in another post just before your reply. Like I said - I don’t use it heavily so it’s not a big deal if it is down for a while.

I was just exploring today and flipped the “Enable Containers” toggle for the first time. Haven’t done anything with it. Before the reinstall ~2 weeks ago, and immediately after it, I installed a few “Apps” which I assume are just cleanly packaged docker containers. I have audiobookshelf, jellyfin, and navidrome running - this is where occasional streaming comes from. I also have homebox and mealie running to poke around.

I uh… don’t have a Windows machine. When Windows 10 went EOL I bailed. I could check if my work laptop can be opened and has a second M.2 slot but only as a last resort.

Actually, technically, I think windows came pre-installed on the eMMC which I don’t think I’ve touched. I have another NAS with everything backed up there so I guess I can risk blowing things up by trying to boot Windows 11 from the eMMC.

6

Let’s start from the most basic and bare setup!

Disconnect all non-boot drives.

Stop ALL apps, VMs, etc.

Basically, we want to test the system as if had been recently installed (no pools, apps or VMs configured).

See if it still hangs.

Me myself, I would install another OS to test the machine MY WAY.

Windows takes 15 min to install and have a GUI ready, but I usually test with a Windows PE that gets me Windows in 1-3 min (depends on storage speed).

7

See if it still hangs.

If I wasn’t clear, it randomly has a process/job hang like once or twice a week (or sometimes after more than a week) and the only time I find out is because something other than the entire GUI becomes unresponsive. The general GUI itself is not unresponsive, at least not without me trying to reboot to fix the unresponsiveness. I don’t really know how to check if something is wrong when there is nothing that normally goes wrong to go wrong? I’ve noticed my apps won’t update or the job to update them gets hung, or I can’t reach the smb share from another computer. No pools and no apps would make that kind of hard.

A typical instance of what I’m trying to troubleshoot might look like this:

  1. It’s been a few days, I log in because it’s a new fun toy.
  2. I check the apps. jellyfin and navidrome say “deploying”… indefinitely.
  3. There is a running job that started 12 hours ago still at 5%.
  4. I try to restart jellyfin.
  5. Now a second job is stuck running indefinitely.
  6. I get frustrated and use the GUI to reboot.
  7. The webpage goes down.
  8. 15 minutes later, I still can’t get it to display.
  9. I hold down the power button, wait 10 seconds, push the power button to turn it back on.
  10. Everything is fine again for another random.randint(2, 10) days.

I usually test with a Windows PE

OMG I haven’t used Windows PE in forever - I’d forgotten about that! I’ll see if I can find something.

I’m just kind of… I don’t know, disappointed that there is no way anyone is recommending to even find out what is actually happening or what process is hung when something hangs.

I just need to basically take it all apart and test everything.

8

Are you able to access the Console when the GUI is hung or SSH into TrueNAS and check using the command line?

9

Usually, it’s just part of the GUI. “Jobs” are stuck running indefinitely and new ones, that seem to be TrueNAS trying to execute something I’ve told it to do via the GUI (like restart an App), just wait for the existing job to finish… which doesn’t happen. It only becomes completely unresponsive when I try to restart it from the GUI during times like that.

I believe I can get into the shell, and I can certainly enable SSH and try to get in that way. I just don’t know what to do from there beyond the log, which hasn’t helped.

10

After my last post, I ran ~24 hours of a memtest with zero errors so I’m ruling out memory errors. I didn’t have time to do much more, so I just powered it up ~2026-06-23 07:48:51 (Eastern) and left it alone.

On Thursday I found the Jellyfin server unresponsive. Beyond having a file browser tab in Dolphin with an smb folder open, this is the first time I’d had any active interaction with any part of the system. I logged in to TrueNAS and, as usual, I found a stuck job ‘catalog.sync’ running at 5%, having started 2026-06-24 07:49:39 - approximately 36 hours previously and almost exactly 24 hours since the system was started after the memtest.

I finally have some time tonight to investigate. No issues reaching the smb shares, no issues logging in via ssh, no issues logging in to the TrueNAS Web GUI. The job is still stuck, no other jobs with any issues, though the catalog.sync task doesn’t run again after it is hung.

Name Progress State Started Finished
zfs.dataset.bulk_process Success 2026-06-26 07:49:39 2026-06-26 07:49:39
pool.dataset.sync_db_keys Success 2026-06-26 07:49:38 2026-06-26 07:49:39
certificate.renew_certs Success 2026-06-26 07:49:38 2026-06-26 07:49:39
directoryservices.cache.refresh_impl Success 2026-06-26 03:30:01 2026-06-26 03:30:01
directoryservices.cache_refresh Success 2026-06-26 03:30:01 2026-06-26 03:30:01
update.download Success 2026-06-26 01:47:01 2026-06-26 01:47:01
zfs.dataset.bulk_process Success 2026-06-25 07:49:39 2026-06-25 07:49:39
pool.dataset.sync_db_keys Success 2026-06-25 07:49:38 2026-06-25 07:49:39
certificate.renew_certs Success 2026-06-25 07:49:38 2026-06-25 07:49:39
directoryservices.cache.refresh_impl Success 2026-06-25 03:30:02 2026-06-25 03:30:02
directoryservices.cache_refresh Success 2026-06-25 03:30:02 2026-06-25 03:30:02
update.download Success 2026-06-25 01:47:01 2026-06-25 01:47:01
catalog.sync 5.00% Running 2026-06-24 07:49:39 N/A
zfs.dataset.bulk_process Success 2026-06-24 07:49:39 2026-06-24 07:49:39
pool.dataset.sync_db_keys Success 2026-06-24 07:49:38 2026-06-24 07:49:39
certificate.renew_certs Success 2026-06-24 07:49:38 2026-06-24 07:49:39
Scrub of pool ‘boot-pool’ Success 2026-06-24 03:45:09 2026-06-24 03:45:13
directoryservices.cache.refresh_impl Success 2026-06-24 03:30:01 2026-06-24 03:30:01
directoryservices.cache_refresh Success 2026-06-24 03:30:01 2026-06-24 03:30:01
update.download Success 2026-06-24 01:47:09 2026-06-24 01:47:10
core.bulk Success 2026-06-23 07:50:13 2026-06-23 07:50:13
service.control Success 2026-06-23 07:50:07 2026-06-23 07:50:12
catalog.sync Success 2026-06-23 07:50:07 2026-06-23 07:50:20
service.control Success 2026-06-23 07:49:42 2026-06-23 07:49:42
service.control Success 2026-06-23 07:49:39 2026-06-23 07:49:42
service.control Success 2026-06-23 07:49:38 2026-06-23 07:49:38
service.control Success 2026-06-23 07:49:38 2026-06-23 07:49:42
directoryservices.initialize Success 2026-06-23 07:49:38 2026-06-23 07:49:38
smb.synchronize_group_mappings Success 2026-06-23 07:49:37 2026-06-23 07:49:38
smb.synchronize_passdb Success 2026-06-23 07:49:37 2026-06-23 07:49:38
service.control Success 2026-06-23 07:49:37 2026-06-23 07:49:37

I exported the log to an the smb share and copied to my laptop for future reference. The following are the entries considered err level by journalctl:

Jun 23 07:48:51 Mini-NAS kernel: proc_thermal_pci 0000:00:04.0: error: proc_thermal_add, will continue
Jun 23 07:48:51 Mini-NAS kernel: Error: Driver 'pcspkr' is already registered, aborting...
Jun 23 07:49:34 Mini-NAS systemd[1]: ipa-epn.timer: Refusing to start, unit ipa-epn.service to trigger not loaded.
Jun 23 07:49:34 Mini-NAS systemd[1]: Failed to start ipa-epn.timer - Execute IPA Expiring Password Notification (EPN) every day at 1AM.
Jun 23 19:35:29 Mini-NAS kernel: INFO: task txg_sync:1243 blocked for more than 120 seconds.
Jun 23 19:35:29 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 23 19:35:29 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 19:37:30 Mini-NAS kernel: INFO: task txg_sync:1243 blocked for more than 241 seconds.
Jun 23 19:37:30 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 23 19:37:30 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 19:41:32 Mini-NAS kernel: INFO: task python.d.plugin:3959 blocked for more than 120 seconds.
Jun 23 19:41:32 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 23 19:41:32 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 19:43:33 Mini-NAS kernel: INFO: task python.d.plugin:3959 blocked for more than 241 seconds.
Jun 23 19:43:33 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 23 19:43:33 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 19:45:33 Mini-NAS kernel: INFO: task python.d.plugin:3959 blocked for more than 362 seconds.
Jun 23 19:45:33 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 23 19:45:33 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 19:47:34 Mini-NAS kernel: INFO: task python.d.plugin:3959 blocked for more than 483 seconds.
Jun 23 19:47:34 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 23 19:47:34 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 19:49:35 Mini-NAS kernel: INFO: task python.d.plugin:3959 blocked for more than 604 seconds.
Jun 23 19:49:35 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 23 19:49:35 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 19:51:36 Mini-NAS kernel: INFO: task python.d.plugin:3959 blocked for more than 724 seconds.
Jun 23 19:51:36 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 23 19:51:36 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 19:53:37 Mini-NAS kernel: INFO: task python.d.plugin:3959 blocked for more than 845 seconds.
Jun 23 19:53:37 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 23 19:53:37 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 23 19:53:37 Mini-NAS kernel: INFO: task dockerd:5068 blocked for more than 120 seconds.
Jun 23 19:53:37 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 23 19:53:37 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Looking at the entire log, I see a LOT of those ‘deadman‘ entries - 4,294 of them. All I can find is some github discussions that I don’t understand. It seems that it is zfs related and basically warns about things taking too long. Again, any information about it might be helpful.

The first record:

Jun 23 20:02:13 Mini-NAS zed[181480]: eid=131 class=deadman pool=‘zpool’ vdev=f6730fb0-c854-4088-923a-60eddd72c850 size=8192 offset=756143386624 priority=0 err=0 flags=0x300080 delay=43365555ms bookmark=438:153324:0:32

The most recent record:

Jun 26 21:18:15 Mini-NAS zed[1167034]: eid=4428 class=deadman pool=‘zpool’ vdev=f6730fb0-c854-4088-923a-60eddd72c850 size=8192 offset=756143386624 priority=0 err=0 flags=0x300080 delay=43365555ms bookmark=438:153324:0:32

These deadman entries reference a ‘delay’ that would approximately start around 12 hours earlier - the time of the kernel messages/errors about blocked tasks. Just before the first blocked task error, I found the following entries:

Jun 23 19:33:18 Mini-NAS kernel: nvme nvme0: I/O tag 209 (00d1) opcode 0x0 (I/O Cmd) QID 3 timeout, aborting req_op:FLUSH(2) size:0
Jun 23 19:33:49 Mini-NAS kernel: nvme nvme0: I/O tag 209 (00d1) opcode 0x0 (I/O Cmd) QID 3 timeout, reset controller
Jun 23 19:34:12 Mini-NAS kernel: nvme nvme0: Abort status: 0x0

I’ve found that most of the blocked task error blocks throughout my log history, there is a similar set of errors, but not always for the same nvme drive.

Jun 13 11:43:22 Mini-NAS kernel: nvme nvme2: I/O tag 611 (f263) opcode 0x0 (I/O Cmd) QID 2 timeout, aborting req_op:FLUSH(2) size:0
Jun 13 11:43:53 Mini-NAS kernel: nvme nvme2: I/O tag 611 (f263) opcode 0x0 (I/O Cmd) QID 2 timeout, reset controller
Jun 13 11:43:59 Mini-NAS kernel: nvme nvme2: Abort status: 0x0
Jun 13 11:44:22 Mini-NAS systemd[1]: Cannot find unit for notify message of PID 1798177, ignoring.
Jun 13 11:46:22 Mini-NAS kernel: INFO: task txg_sync:1270 blocked for more than 120 seconds.
Jun 13 11:46:22 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 13 11:46:22 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Jun 18 12:19:33 Mini-NAS kernel: nvme nvme4: I/O tag 532 (f214) opcode 0x0 (I/O Cmd) QID 4 timeout, aborting req_op:FLUSH(2) size:0
Jun 18 12:20:03 Mini-NAS kernel: nvme nvme4: I/O tag 532 (f214) opcode 0x0 (I/O Cmd) QID 4 timeout, reset controller
Jun 18 12:20:03 Mini-NAS systemd[1]: Starting sysstat-collect.service - system activity accounting tool...
Jun 18 12:20:03 Mini-NAS systemd[1]: sysstat-collect.service: Deactivated successfully.
Jun 18 12:20:03 Mini-NAS systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
Jun 18 12:20:12 Mini-NAS kernel: nvme nvme4: Abort status: 0x0
Jun 18 12:22:59 Mini-NAS kernel: INFO: task txg_sync:1247 blocked for more than 120 seconds.
Jun 18 12:22:59 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 18 12:22:59 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Jun 20 02:43:34 Mini-NAS kernel: nvme nvme1: I/O tag 751 (42ef) opcode 0x0 (I/O Cmd) QID 4 timeout, aborting req_op:FLUSH(2) size:0
Jun 20 02:44:05 Mini-NAS kernel: nvme nvme1: I/O tag 751 (42ef) opcode 0x0 (I/O Cmd) QID 4 timeout, reset controller
Jun 20 02:44:54 Mini-NAS kernel: nvme nvme1: Abort status: 0x0
Jun 20 02:46:39 Mini-NAS kernel: INFO: task txg_sync:1260 blocked for more than 120 seconds.
Jun 20 02:46:39 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 20 02:46:39 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Jun 20 20:53:15 Mini-NAS kernel: nvme nvme1: I/O tag 155 (c09b) opcode 0x0 (I/O Cmd) QID 2 timeout, aborting req_op:FLUSH(2) size:0
Jun 20 20:53:46 Mini-NAS kernel: nvme nvme1: I/O tag 155 (c09b) opcode 0x0 (I/O Cmd) QID 2 timeout, reset controller
Jun 20 20:54:46 Mini-NAS kernel: nvme nvme1: Abort status: 0x0
Jun 20 20:56:05 Mini-NAS kernel: INFO: task txg_sync:1270 blocked for more than 120 seconds.
Jun 20 20:56:05 Mini-NAS kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jun 20 20:56:05 Mini-NAS kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

So that’s nvme0, 1, 2, and 4 so far. That’s notably not consistent with the drives reporting segment errors from SMART tests.

At this point, I’m going to pull several of the nvme drives and test an install with only the two identical drives with no segment errors for the zpool. Rule out possible excessive power draws.

This is obnoxious and exhausting.

11

It does not seem like a power issue. Half of the nvme drives have been removed, TrueNAS was reinstalled from scratch. I created a single smb share and added two apps that were never installed on the system before. I played around with the apps and then left the system alone. I checked back in a few days but didn’t find any stuck jobs. Then I tried to update one of the apps… and the update-related jobs were hung. I tried to shutdown, and the system started the shutdown process and the device was unreachable but never shut down - same as the last time. Eventually, I long pressed the power button for a hard shutdown.

I booted back up, checked the log, and found that the same kind of message about one of the drives and resetting its controller was in the log.

Jul 02 02:59:12 truenas kernel: nvme nvme1: I/O tag 303 (612f) opcode 0x0 (I/O Cmd) QID 4 timeout, aborting req_op:FLUSH(2) size:0
Jul 02 02:59:42 truenas kernel: nvme nvme1: I/O tag 303 (612f) opcode 0x0 (I/O Cmd) QID 4 timeout, reset controller
Jul 02 03:00:08 truenas systemd[1]: Starting sysstat-collect.service - system activity accounting tool...
Jul 02 03:00:08 truenas systemd[1]: sysstat-collect.service: Deactivated successfully.
Jul 02 03:00:08 truenas systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
Jul 02 03:00:16 truenas kernel: nvme nvme1: Abort status: 0x0
Jul 02 03:01:59 truenas kernel: INFO: task txg_sync:1162 blocked for more than 120 seconds.
Jul 02 03:01:59 truenas kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jul 02 03:01:59 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 02 03:01:59 truenas kernel: task:txg_sync        state:D stack:0     pid:1162  tgid:1162  ppid:2      flags:0x00004000
Jul 02 03:01:59 truenas kernel: Call Trace:
Jul 02 03:01:59 truenas kernel:  <TASK>
Jul 02 03:01:59 truenas kernel:  __schedule+0x461/0xa10
Jul 02 03:01:59 truenas kernel:  schedule+0x27/0xd0
Jul 02 03:01:59 truenas kernel:  schedule_timeout+0x9e/0x170
Jul 02 03:01:59 truenas kernel:  ? __pfx_process_timeout+0x10/0x10
Jul 02 03:01:59 truenas kernel:  io_schedule_timeout+0x51/0x70
Jul 02 03:01:59 truenas kernel:  __cv_timedwait_common+0x129/0x160 [spl]
Jul 02 03:01:59 truenas kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Jul 02 03:01:59 truenas kernel:  __cv_timedwait_io+0x19/0x20 [spl]
Jul 02 03:01:59 truenas kernel:  zio_wait+0x11a/0x240 [zfs]
Jul 02 03:01:59 truenas kernel:  vdev_config_sync+0xe0/0x2a0 [zfs]
Jul 02 03:01:59 truenas kernel:  spa_sync_rewrite_vdev_config+0x1ae/0x1d0 [zfs]
Jul 02 03:01:59 truenas kernel:  spa_sync+0x276/0x460 [zfs]
Jul 02 03:01:59 truenas kernel:  txg_sync_thread+0x1ec/0x270 [zfs]
Jul 02 03:01:59 truenas kernel:  ? __pfx_txg_sync_thread+0x10/0x10 [zfs]
Jul 02 03:01:59 truenas kernel:  ? __pfx_thread_generic_wrapper+0x10/0x10 [spl]
Jul 02 03:01:59 truenas kernel:  thread_generic_wrapper+0x5a/0x70 [spl]
Jul 02 03:01:59 truenas kernel:  kthread+0xcf/0x100
Jul 02 03:01:59 truenas kernel:  ? __pfx_kthread+0x10/0x10
Jul 02 03:01:59 truenas kernel:  ret_from_fork+0x31/0x50
Jul 02 03:01:59 truenas kernel:  ? __pfx_kthread+0x10/0x10
Jul 02 03:01:59 truenas kernel:  ret_from_fork_asm+0x1a/0x30
Jul 02 03:01:59 truenas kernel:  </TASK>
Jul 02 03:06:01 truenas kernel: INFO: task python.d.plugin:3874 blocked for more than 120 seconds.
Jul 02 03:06:01 truenas kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jul 02 03:06:01 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 02 03:06:01 truenas kernel: task:python.d.plugin state:D stack:0     pid:3874  tgid:2938  ppid:2924   flags:0x00000002
Jul 02 03:06:01 truenas kernel: Call Trace:
Jul 02 03:06:01 truenas kernel:  <TASK>
Jul 02 03:06:01 truenas kernel:  __schedule+0x461/0xa10
Jul 02 03:06:01 truenas kernel:  schedule+0x27/0xd0
Jul 02 03:06:01 truenas kernel:  schedule_preempt_disabled+0x15/0x30
Jul 02 03:06:01 truenas kernel:  __mutex_lock.constprop.0+0x34c/0x6a0
Jul 02 03:06:01 truenas kernel:  nvme_hwmon_read+0x5e/0x1b0 [nvme_core]
Jul 02 03:06:01 truenas kernel:  hwmon_attr_show+0x43/0xf0
Jul 02 03:06:01 truenas kernel:  dev_attr_show+0x19/0x60
Jul 02 03:06:01 truenas kernel:  sysfs_kf_seq_show+0xa4/0x100
Jul 02 03:06:01 truenas kernel:  seq_read_iter+0x120/0x470
Jul 02 03:06:01 truenas kernel:  vfs_read+0x244/0x370
Jul 02 03:06:01 truenas kernel:  ksys_read+0x6f/0xf0
Jul 02 03:06:01 truenas kernel:  do_syscall_64+0x87/0x1a0
Jul 02 03:06:01 truenas kernel:  ? audit_filter_rules.constprop.0+0x140/0x1120
Jul 02 03:06:01 truenas kernel:  ? __audit_filter_op+0xaf/0x110
Jul 02 03:06:01 truenas kernel:  ? audit_reset_context+0x232/0x300
Jul 02 03:06:01 truenas kernel:  ? syscall_exit_to_user_mode_prepare+0x148/0x170
Jul 02 03:06:01 truenas kernel:  ? syscall_exit_to_user_mode+0x37/0x1b0
Jul 02 03:06:01 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:06:01 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:06:01 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:06:01 truenas kernel:  ? syscall_exit_to_user_mode+0x37/0x1b0
Jul 02 03:06:01 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:06:01 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:06:01 truenas kernel:  ? __sysvec_apic_timer_interrupt+0x55/0x120
Jul 02 03:06:01 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jul 02 03:06:01 truenas kernel: RIP: 0033:0x7f35a148a29c
Jul 02 03:06:01 truenas kernel: RSP: 002b:00007f359dda99f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Jul 02 03:06:01 truenas kernel: RAX: ffffffffffffffda RBX: 00007f359ddaa640 RCX: 00007f35a148a29c
Jul 02 03:06:01 truenas kernel: RDX: 0000000000001001 RSI: 000000002dd1f090 RDI: 000000000000000b
Jul 02 03:06:01 truenas kernel: RBP: 000000002dcb2870 R08: 0000000000000000 R09: 0000000000000000
Jul 02 03:06:01 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f359fe3f380
Jul 02 03:06:01 truenas kernel: R13: 000000002dcb2870 R14: 000000002dd1f090 R15: 000000000000000b
Jul 02 03:06:01 truenas kernel:  </TASK>
Jul 02 03:08:02 truenas kernel: INFO: task python.d.plugin:3874 blocked for more than 241 seconds.
Jul 02 03:08:02 truenas kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jul 02 03:08:02 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 02 03:08:02 truenas kernel: task:python.d.plugin state:D stack:0     pid:3874  tgid:2938  ppid:2924   flags:0x00000002
Jul 02 03:08:02 truenas kernel: Call Trace:
Jul 02 03:08:02 truenas kernel:  <TASK>
Jul 02 03:08:02 truenas kernel:  __schedule+0x461/0xa10
Jul 02 03:08:02 truenas kernel:  schedule+0x27/0xd0
Jul 02 03:08:02 truenas kernel:  schedule_preempt_disabled+0x15/0x30
Jul 02 03:08:02 truenas kernel:  __mutex_lock.constprop.0+0x34c/0x6a0
Jul 02 03:08:02 truenas kernel:  nvme_hwmon_read+0x5e/0x1b0 [nvme_core]
Jul 02 03:08:02 truenas kernel:  hwmon_attr_show+0x43/0xf0
Jul 02 03:08:02 truenas kernel:  dev_attr_show+0x19/0x60
Jul 02 03:08:02 truenas kernel:  sysfs_kf_seq_show+0xa4/0x100
Jul 02 03:08:02 truenas kernel:  seq_read_iter+0x120/0x470
Jul 02 03:08:02 truenas kernel:  vfs_read+0x244/0x370
Jul 02 03:08:02 truenas kernel:  ksys_read+0x6f/0xf0
Jul 02 03:08:02 truenas kernel:  do_syscall_64+0x87/0x1a0
Jul 02 03:08:02 truenas kernel:  ? audit_filter_rules.constprop.0+0x140/0x1120
Jul 02 03:08:02 truenas kernel:  ? __audit_filter_op+0xaf/0x110
Jul 02 03:08:02 truenas kernel:  ? audit_reset_context+0x232/0x300
Jul 02 03:08:02 truenas kernel:  ? syscall_exit_to_user_mode_prepare+0x148/0x170
Jul 02 03:08:02 truenas kernel:  ? syscall_exit_to_user_mode+0x37/0x1b0
Jul 02 03:08:02 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:08:02 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:08:02 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:08:02 truenas kernel:  ? syscall_exit_to_user_mode+0x37/0x1b0
Jul 02 03:08:02 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:08:02 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:08:02 truenas kernel:  ? __sysvec_apic_timer_interrupt+0x55/0x120
Jul 02 03:08:02 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jul 02 03:08:02 truenas kernel: RIP: 0033:0x7f35a148a29c
Jul 02 03:08:02 truenas kernel: RSP: 002b:00007f359dda99f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Jul 02 03:08:02 truenas kernel: RAX: ffffffffffffffda RBX: 00007f359ddaa640 RCX: 00007f35a148a29c
Jul 02 03:08:02 truenas kernel: RDX: 0000000000001001 RSI: 000000002dd1f090 RDI: 000000000000000b
Jul 02 03:08:02 truenas kernel: RBP: 000000002dcb2870 R08: 0000000000000000 R09: 0000000000000000
Jul 02 03:08:02 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f359fe3f380
Jul 02 03:08:02 truenas kernel: R13: 000000002dcb2870 R14: 000000002dd1f090 R15: 000000000000000b
Jul 02 03:08:02 truenas kernel:  </TASK>
Jul 02 03:10:03 truenas kernel: INFO: task python.d.plugin:3874 blocked for more than 362 seconds.
Jul 02 03:10:03 truenas kernel:       Tainted: P           OE      6.12.91-production+truenas #1
Jul 02 03:10:03 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 02 03:10:03 truenas kernel: task:python.d.plugin state:D stack:0     pid:3874  tgid:2938  ppid:2924   flags:0x00000002
Jul 02 03:10:03 truenas kernel: Call Trace:
Jul 02 03:10:03 truenas kernel:  <TASK>
Jul 02 03:10:03 truenas kernel:  __schedule+0x461/0xa10
Jul 02 03:10:03 truenas kernel:  schedule+0x27/0xd0
Jul 02 03:10:03 truenas kernel:  schedule_preempt_disabled+0x15/0x30
Jul 02 03:10:03 truenas kernel:  __mutex_lock.constprop.0+0x34c/0x6a0
Jul 02 03:10:03 truenas kernel:  nvme_hwmon_read+0x5e/0x1b0 [nvme_core]
Jul 02 03:10:03 truenas kernel:  hwmon_attr_show+0x43/0xf0
Jul 02 03:10:03 truenas kernel:  dev_attr_show+0x19/0x60
Jul 02 03:10:03 truenas kernel:  sysfs_kf_seq_show+0xa4/0x100
Jul 02 03:10:03 truenas kernel:  seq_read_iter+0x120/0x470
Jul 02 03:10:03 truenas kernel:  vfs_read+0x244/0x370
Jul 02 03:10:03 truenas kernel:  ksys_read+0x6f/0xf0
Jul 02 03:10:03 truenas kernel:  do_syscall_64+0x87/0x1a0
Jul 02 03:10:03 truenas kernel:  ? audit_filter_rules.constprop.0+0x140/0x1120
Jul 02 03:10:03 truenas kernel:  ? __audit_filter_op+0xaf/0x110
Jul 02 03:10:03 truenas kernel:  ? audit_reset_context+0x232/0x300
Jul 02 03:10:03 truenas kernel:  ? syscall_exit_to_user_mode_prepare+0x148/0x170
Jul 02 03:10:03 truenas kernel:  ? syscall_exit_to_user_mode+0x37/0x1b0
Jul 02 03:10:03 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:10:03 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:10:03 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:10:03 truenas kernel:  ? syscall_exit_to_user_mode+0x37/0x1b0
Jul 02 03:10:03 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:10:03 truenas kernel:  ? do_syscall_64+0x93/0x1a0
Jul 02 03:10:03 truenas kernel:  ? __sysvec_apic_timer_interrupt+0x55/0x120
Jul 02 03:10:03 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jul 02 03:10:03 truenas kernel: RIP: 0033:0x7f35a148a29c
Jul 02 03:10:03 truenas kernel: RSP: 002b:00007f359dda99f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Jul 02 03:10:03 truenas kernel: RAX: ffffffffffffffda RBX: 00007f359ddaa640 RCX: 00007f35a148a29c
Jul 02 03:10:03 truenas kernel: RDX: 0000000000001001 RSI: 000000002dd1f090 RDI: 000000000000000b
Jul 02 03:10:03 truenas kernel: RBP: 000000002dcb2870 R08: 0000000000000000 R09: 0000000000000000
Jul 02 03:10:03 truenas kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f359fe3f380
Jul 02 03:10:03 truenas kernel: R13: 000000002dcb2870 R14: 000000002dd1f090 R15: 000000000000000b
Jul 02 03:10:03 truenas kernel:  </TASK>

nvme1 has zero smart errors in its testing history even after running an extended test from the last installation’s troubleshooting.

Not too long after that nvme error, the deadman entries started showing up in the log. 705 of them, starting some time after the above nvme error.

Jul 02 09:40:52 truenas zed[593015]: eid=112 class=deadman pool='zpool' vdev=85563d69-3820-4762-9741-41988d8d1316 size=16384 offset=154638540800 priority=0 err=0 flags=0x300080 delay=384075095ms bookmark=170:134:0:10

...

Jul 02 21:41:46 truenas zed[654077]: eid=816 class=deadman pool='zpool' vdev=85563d69-3820-4762-9741-41988d8d1316 size=16384 offset=154638540800 priority=0 err=0 flags=0x300080 delay=384075095ms bookmark=170:134:0:10

I guess now I need try to run some diagnostics and stress tests to rule out the never-had-smart-errors nvme to make sure there isn’t something wrong with it that can’t be caught with smart?

12

You might want to take a look at the manufacture website for the NVMe and see if they have diag software. That may be a more reliable way to check even though you may need to install on Windows and test on that system

13

I booted and completed the Windows 11 installation that was still present on the eMMC. It ran like absolute trash but eventually after several Windows update attempts, where one update or another sat at “installing 0%” for literal hours without progressing until I lost patience and restarted, it was finally updated and stable. Windows sees the drives and brings up the correct model number in Device Manager, though I have not formatted them - they were left as-is from TrueNAS.

I installed the PNY PCIe SSD Toolbox. It lists no drives, even when launched as admin. I didn’t have any more patience in me this weekend.

My next effort with this NAS is going to see if it is of the flawed initial batch. I found a forum thread about inconsistent power and unexplained power drops causing nvme controllers to reset and fixed by RMAing the device. The disks aren’t dropping from the pool (at least the are zero entries in the log that indicate such) and I’m absolutely not taxing the system like those who reported issues but it sounds like similar issue so maybe that’s my issue.

Either way, I’m done with this thing. It seemed like an interesting experiment at the time but achieved absolutely nothing for me beyond a slightly smaller NAS, even if it wasn’t a waste of a large portion of my free time for well over a month. Now that SSDs have become unaffordable for the foreseeable future I’m just going to move on. I’m now troubleshooting this just so that I can try to get rid of it without feeling guilty that I’m selling or giving away a broken device. I’ll use the m.2 drives for something else - maybe as cache or a smaller second pool for a mostly mechanical drive system and hope that mechanical drives don’t skyrocket in price 6x+ before I run out of replacements.

14

I run this device in its original 12 GB memory variant with

  • BIOS version 307
  • eMMC disabled
  • WLAN removed
  • 6x Crucial P3 1TB SSDs
  • FreeBSD 15.1
  • 3x mirror vdevs in a single zpool

No stability problems whatsoever. Going to use it at EuroBSDCon 2026 for the jail tutorial I will be giving.

Of course there are too many variable parts involved to easily point a finger at the cause of your problems.