Hello everyone,
I’m running into an issue where the middlewared service on my TrueNAS SCALE system fails to start, timing out after several attempts.
Here’s a summary of the issue:
Symptoms:
Middleware attempts to load, but it times out during startup.
Once the timeout occurs, the asyncio loop appears to peg the CPU usage to 100%.
I see repeated errors in the logs like:
RuntimeError: Event loop stopped before Future completed
State ‘stop-sigterm’ timed out. Skipping SIGKILL
Failed with result ‘timeout’
After a certain amount of time, the middlewared service fails entirely and won’t restart, consuming high CPU during the process.
System Info:
TrueNAS SCALE (24.04.2.1)
Hardware specs (Truenas Mini XL)
Troubleshooting Steps Taken:
Checked CPU, Memory, and Disk Space: No obvious resource shortages .
Killed stuck processes: Identified and killed any lingering middleware-related processes, especially asyncio-related, but they keep respawning after retries.
Cleared old logs and rotated journal: Freed up space in the /var/log/journal, but the issue persists.
Reinstalled middleware: Attempted to reinstall freenas-middleware to reset any corrupt dependencies but no success.
Rebooted multiple times: Same results on reboot, with middlewared timing out and asyncio_loop consuming CPU.
Logs: The most notable log messages include:
RuntimeError: Event loop stopped before Future completed in /usr/lib/python3.11/asyncio/base_events.py
State ‘stop-sigterm’ timed out. Skipping SIGKILL
Failed to start middlewared.service - TrueNAS Middleware
Thanks for the recommendation.
I tired to boot into my previous boot instance but i had just migrated off of core recently. I booted into core but none of the data was available. Not able to go through a reinstall from the core instance.
This all started to happen when i attempted to boot into the webui and got a weird message from truenas to check to make sure the nas was available. SMB and SSH was working, I attempted to reboot it via ssh and it hung, hard reboot ended up with the middleware unable to load message on ipmi
Attached are some screenshots of the logs\status.
Does anyone have suggestions on how to resolve this issue or further troubleshoot the root cause?
Any help would be greatly appreciated!
Thank you in advance for any advice or guidance!