Naturally I woke up this morning to an email alert about a “unrecoverable error”. And trying to log into the web gui just shows “Dead Host“. For some reason there is literally no information on this situation, beyond generic “my server died” posts. The unrecoverable error alert was cleared on a reboot anyways, yet the UI still says “Dead Host”. Do people have any next steps as to how I can figure out what’s actually wrong??
Alert:
Pool Starboard state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
Some additional info, this system has been running for roughly 3 years. Old ssd is the os drive I believe, and 4x wd red plus 4tb drives in a 2x mirror, ECC memory and a UPS and everything. Applications indeed seem unaffected as Jellyfin is moving along just fine. I’ll have to come back to this after waking up a bit more to see if remoting in through the BMC gives me anything useful. But considering the error I’m hoping there’s some order of operations to safely recover, instead of just trying random stuff.
Alright I’m sorry guys. The issue has now fixed itself…
So logging in via the BMC showed the standard “Console setup” info, with no errors. So I figure I’d directly connect to the server instead of using my DNS alias. And that indeed allowed me to connect just fine, and no pools show any errors, and the only alert is about an expiring certificate and system update (25.10.2 > 25.10.2.1). So I tried connecting via my nginx path, and it just worked too! And then the original domain started working after like 10 minutes of aimlessly looking to see if something was wrong with nginx! What the heck? The BMC IPMI logs don’t show any unusual errors, and everything seems fine. This is literally comically awful. And keep in mind this “Dead Host” screen was from truenas, theming and everything, not nginx or another generic connection error message.
I’ll try doing a bios update if available, and update truenas and the apps. But what a thing to wake up to.
Alright here’s a TDLR:
Random “unrecoverable error” alert email recieved. Tried to log-in via the server’s ip directly (no port), and it showed a “Dead Host” screen from TrueNas. Tried a restart of the server and the error was cleared. Turns out somehow, possibly after an update or after I setup nginx, the server ip just goes to this “Dead Host” screen if you don’t specify a port. When specifying port 81 or 444, it works just fine, or when using my nginx proxy.
Request: CHANGE THIS SCREEN. Literally any search about it just leads to “my server is dead” posts, despite it just seemingly being a “page not found” sort-of thing. Hopefully this post can out SEO the others for anyone else getting this error.
Still no idea what caused that other error in the first place, so I guess I’ll hope it was a fluke, sweats profusely.
