Attack of the zombie alerts!

TrueNAS General
1

I’m on (I think) the most recent non-SCALE TrueNAS, running on a first gen FreeNAS Mini.

Late summer I had a whole bunch of alerts pile up… temp running high, and then battery voltage too low. I blew out dust and replaced the battery and all has been well, but this mass of old alerts keeps popping back up… emailing me… appearing in the UI… and when i dismiss them, they appear to actually be dismissed, and then sometime later (sometimes hours, sometimes days) they are back.

Any thoughts what I might be able to do about this?

2

You have a better chance of getting help if you provide more information about your system, an literal example of one of the zombie alerts, and so on.

A good starting point is to read these guidelines:

3

I’m wondering if these alerts are from ipmi & need to be cleared from there?..

4

Sorry about the lack of info… I was at work and didn’t have better details, but wanted to try to start the conversation because I was getting so many emails. :slight_smile:

The only add’l I have right now (at work again) are it’s TrueNAS-13.0-U6.2 and the system disk is 15GB. But yeah, I don’t have other details. The alerts I keep getting re-alerted about look like this:

TrueNAS @ freenas.local

These alerts have been cleared:
* Voltage BAT Deasserted Lower Non-recoverable going low : Reading 3.26 <
Threshold 2.38 Volts.

* Voltage BAT Deasserted Lower Critical going low : Reading 3.26 < Threshold
2.50 Volts.

Current alerts:
* freenas.local had an unscheduled system reboot. The operating system
successfully came back online at Wed Sep 11 10:00:36 2024.

* freenas.local had an unscheduled system reboot. The operating system
successfully came back online at Sun Nov 3 14:14:02 2024.

* freenas.local had an unscheduled system reboot. The operating system
successfully came back online at Mon Nov 4 11:16:52 2024.

* freenas.local had an unscheduled system reboot. The operating system
successfully came back online at Mon Nov 4 13:10:36 2024.

* freenas.local had an unscheduled system reboot. The operating system
successfully came back online at Mon Nov 4 16:18:19 2024.

* freenas.local had an unscheduled system reboot. The operating system
successfully came back online at Wed Nov 6 19:34:24 2024.

* freenas.local had an unscheduled system reboot. The operating system
successfully came back online at Wed Nov 6 21:51:21 2024.

* freenas.local had an unscheduled system reboot. The operating system
successfully came back online at Thu Nov 7 07:04:04 2024.

* New ZFS version or feature flags are available for pool(s) Pool1. Upgrading
pools is a one-time process that can prevent rolling the system back to an
earlier TrueNAS version. It is recommended to read the TrueNAS release notes
and confirm you need the new ZFS feature flags before upgrading a pool.
5

I see you’re testing ZFS’ resilience when faced with repeated unplanned reboots. Interesting.

Anyway, if it’s the top two alerts you’re referring to, those are likely from the motherboard IPMI.
Your BAT error message suggests your CMOS battery was running out of juice. If you have indeed replaced it, it’s possible you need to clear the log in the IPMI interface to get it to fully clear out.

1 Like
6

Thanks for the pointer… I logged into IPMI and was able to clear the 2000+ alerts that had built up over the last 8 years. I had replaced the battery somewhat recently, and yeah, those power problems were from some storms this spring, if I remember right.

Anyway, I suspect I’ll be back to mark this as the solution… thank you!

1 Like
7

I do see some of these remaining in the alerts in the TrueNAS UI… about once a day for the last few days…

freenas.local had an unscheduled system reboot. The operating system successfully came back online at Wed Nov 6 19:34:24 2024.

But I’m not seeing anything that appears unhealthy… memory seems good… temps good… I’ll give it a reboot to see if that helps get its mind right.

8

Are you certain that it’s not rebooting when you’re not looking?

If the BIOS is set to start the system back up after power loss it could be masking that your power delivery is in bad shape, possibly because of the storms you mentioned. It could be the PSU or something further up the chain.

Either way, investing in a UPS is something you should probably look into.

9

Yeah it might be. FWIW, I do have it on a battery backup, so it has rarely if ever gone down hard. It’s got something like 20-30 minutes then gets the shutdown signal. Fairly certain that all has been working.

Nevertheless, maybe the PSU is just plain old? Now that my IPMI alerts are non-absurd, could I expect to see some fresh alerts there if that were actually happening?

10

Interestingly (I think), while I have a few of those “unscheduled system reboots” over the last few days, my UniFi has no matching connection logs… the most recent I see is last Sunday, which I think was actually me.