ongoing problem before ee . job middlewared.service/ start running.
middlewared: setting up plugins (audit) [47/111]. then it runs for about 7 minutes then goes to middleware not running press enter. Any help would be appreciated to restart from cli screen . If i go back to 24.0 it loads and if i do a fresh install. this been driving me crazy.
Same issue, happened when I rebooted into the EE .0.2 update, but not sure if it’s related.
I seem to be having a similar issue. My TrueNAS seemed to have crashed. I couldn’t get a video output via my KVM console so I rebooted.
Afterwards TrueNAS did not come up and I still did not have any video output beyond the GRUB screen. It just went black.
Out of desperation I plugged a monitor into my Intel ARC GPU which I use for transcoding.
There I got an output and I could see, that middlewared hangs for me as well:
middlewared: setting up plugins (audit) [67/111]
There it just hangs. This is still a TrueNAS 24.10.0, I will now try to update to 24.10.0.2 but if it happened for you after updating to this version I have little hope.
I think this might actually be the first reboot of my NAS since updating to 24.10 so I think after the initial boot it did not have to boot up a single time.
I hope you resolved your issue mine was running for awhile again after a new installation even updated 01-02 then shutdown today for disk upgrade. BAM it stuck again middlewared: setting up plugins (audit) [74/111]. This is getting annoying i would love to here from the admins regardind this issue please thanks.
same thing here.
It gets stuck there then I get dropped into a shell and cant start the service
I re-created my VM and restore config and got it all restored. But I just updated and rebooted my host, and it got stuck at the same point again. Had to re-create VM and restore config again.
Has anyone found a fix for this yet? I
Have any of you reported a bug and debug dump?
Top right of the TrueNAS dashboard is a SMILE. Feedback / Report a Bug.
As someone currently also experiencing this issue, no, I have not, as without middlewared, there is no Dashboard for us to log into.
Edit: After a nearly 14 minute long boot the system has now started in read-only mode, without my storage pool (thankfully just a symptom of the latter as far as I can tell). Not sure exactly where I might find logs worth uploading, but if someone may assist on that front, I’d be more than happy to forward them to iX.
Edit2: Upon closer inspection, it would appear zfs-mount.service fails just before middlewared, complaining about the root file-system being read-only. Oddly enough, it appears to be trying to mount my main zfs pool (confusingly named pool, whoops) to /pool, as opposed to /mnt/pool where TrueNAS typically mounts it to…
I’m not from iX, but /var/log/middlewared.log would interest me.
TL;DR: ipa-epn.service appears to be broken and causing middlewared.service to fail. Disabling it and restarting the system appears to fix everything unless you’re like me and messed things up in the meantime.
Okay, I believe I’ve made some somewhat promising progress:
After looking through systemctl for a while (assuming , I believe what’s actually failing on boot to be zfs-mount.service to be a red herring for what’s actually going wrong for now)ipa-epn.service.
It fails on boot and spits the following error in it’s log-file (/var/log/ipaepn.log):
2024-11-29T13:03:40Z DEBUG The IPA-EPN command failed, exception: ScriptError:
2024-11-29T13:03:40Z ERROR The IPA-EPN command failed. See /var/log/ipaepn.log for more information
2024-11-29T13:12:11Z DEBUG Logging to /var/log/ipaepn.log
2024-11-29T13:12:11Z DEBUG IPA-EPN was invoked with arguments [] and options: {'verbose': False, 'quiet': False, 'log_file': None, 'from_nbdays': None, 'to_nbdays': None, 'dry_run': False, 'mailtest': False}
2024-11-29T13:12:11Z DEBUG IPA version 4.12.1
2024-11-29T13:12:11Z DEBUG IPA platform debian
2024-11-29T13:12:11Z DEBUG IPA os-release Debian GNU/Linux 12 (bookworm)
2024-11-29T13:12:11Z DEBUG Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
2024-11-29T13:12:11Z DEBUG Loading StateFile from '/var/lib/ipa-client/sysrestore/sysrestore.state'
2024-11-29T13:12:11Z DEBUG Loading StateFile from '/var/lib/ipa-client/sysrestore/sysrestore.state'
2024-11-29T13:12:11Z ERROR IPA client is not configured on this system.
2024-11-29T13:12:11Z DEBUG File "/usr/lib/python3/dist-packages/ipapython/admintool.py", line 180, in execute
return_value = self.run()
^^^^^^^^^^
File "/usr/lib/python3/dist-packages/ipaclient/install/ipa_epn.py", line 299, in run
raise admintool.ScriptError()
2024-11-29T13:12:11Z DEBUG The IPA-EPN command failed, exception: ScriptError:
2024-11-29T13:12:11Z ERROR The IPA-EPN command failed. See /var/log/ipaepn.log for more information
After disabling it (systemctl disable ipa-epn) and restarting the system, now only zfs-mount appears to fail, again complaining about the read-only filesystem Is the TrueNAS root filesystem read-only by default? I recall hearing something about that, if so, how are the zfs filesystems mounted? Is it really through the zfs-mount service? and if so, why isn’t it mounting it where it’s supposed to? I did try disabling the zfs-mount service as well to see if TrueNAS would mount it through it’s own magic, but appears not to be so.
Edit: It appears this last part of it not mounting correctly may have been caused by me running zpool import pool early on. This was easily fixed by simply exporting & re-importing the dataset from the Dashboard that’s now finally accessible with ipa-epn Disabled
Some parts of the boot pool are mounted read-only.
Import non-boot pool is usually done by the middleware. The middleware does use -R /mnt flag to specify alt root. In your case the import wasn’t done using alt-root, which may indicates that something else is importing the pool.
The middleware also does the mounting using zfs mount -R <pool>.
ipa-epn failing is normal.
The only thing that sounds like it might import pools is zfs-import.service, which in a normal installation is disabled:
$ sudo systemctl status zfs-import.service
○ zfs-import.service
Loaded: masked (Reason: Unit zfs-import.service is masked.)
Active: inactive (dead)
have you guys found a solution for this ?
after initially solving this by just installing TrueNAS fresh onto the boot drive and restoring configuration I am now facing the same issue again
I also still cannot see video output after GRUB finishs via my BMC - I had to wait for coming home and plugging in a monitor to diagnose this error.
i disabled that service mentioned and restarted and still had the issue
i redeployed a 24.10.00 VM and wont update to 24.10.02, im just thankful, I just use tnas as NAS and have proxmox running lxc’s
still would be intersted to understand what went wrong…
Yes, this has to happen.
There’s a “Report a Bug”-link at the very top of the forum so you do not need to be able to successfully load the TrueNAS Dashboard.
For that to be useful I‘d have to extract some logs to get an actually useful ticket done.
I’ve got to admit that I did not have the nerves to mount a USB stick and copy log files or something of that kind.
I just reinstalled TrueNAS, restored my config and now I hope that this does not happen again after 24.10.1 comes out…