Issues after power loss

iTsMaaT · April 4, 2024, 2:30pm

Hi, every time I get a power loss, my server doesn’t automatically boot back up after the power comes back, but when I power it on manually, I have no issues so it doesn’t seem to be corruption. Does anyone know why that might happen?

winnielinnie · April 4, 2024, 2:33pm

This is usually configured in your motherboard’s BIOS / EFI.

I would check there first, to see if your motherboard even has this feature.

iTsMaaT · April 4, 2024, 2:41pm

Thanks, i’ll check that when i have access to the machine and get back if i have any issues

winnielinnie · April 4, 2024, 2:43pm

Consider investing in a UPS (battery backup), even a low-end one that can at least mitigate against brief powerloss events.

iTsMaaT · April 4, 2024, 2:47pm

I do have one, but I haven’t set it up yet, I will tomorrow, I just need to make sure it properly shuts down both of my servers, so I don’t know if ill plug in the USB of the UPS to the NAS or to my Ubuntu server yet.

Protopia · April 4, 2024, 2:48pm

I couldn’t agree more. Although ZFS pools are pretty resilient against sudden failures, we have had several brief outages in the 6 months since I implemented our NAS. Additionally it smooths out any brownouts and spikes and protects the electronics of your disks - and you really don’t want one disk much less several disks having their electronics fried at the same time.

iTsMaaT · April 4, 2024, 2:52pm

What would the setting in the bios be? Because the only thing close i can think of is WOL, which after some research seems to not be well supported by truenas

winnielinnie · April 4, 2024, 2:55pm

“Restore on AC Power Loss” or something like that.

The wording might be different.

Could be under “Advanced” or “ACPI” settings.

Constantin · April 4, 2024, 4:31pm

You want to look into NUT and using same to broadcast from your NUT server to other devices that it’s time to shut down gracefully as the battery goes low. This can be a very frustrating process to set up and test but eventually it will work as intended.

Constantin · April 4, 2024, 4:33pm

It’s definitely not WOL. @winnielinnie has got it 100% right, but it varies by OEM. Supermicro tech support replied to a similar question like this:

Enter the BIOS setup and select the Advanced > Boot Feature and set the Restore on AC Power Loss setting to “Last State”. You will know there was an AC power loss if the system starts automatically when you plug the AC power cord in. If you plug the power card in and the system does not start (and you have to manually press a button to start the system), that means the system was normally shut down from the operating system.

iTsMaaT · April 4, 2024, 4:43pm

What i saw some people did is use the TrueNAS support for UPS to shut it down, and make other servers ping the TrueNAS server and if its shut down, shut down the server as well

chuck32 · April 4, 2024, 4:49pm

Yes, that’s what I’m doing. It’s not that much of a hassle to setup honestly.

What UPS do you have? If it works with truenas it’s not that much of a setup.

If needed, I can copy the appropriate settings / configs from my Ubuntu VM and supply it to you.

iTsMaaT · April 4, 2024, 4:59pm

That would be great, I use a APC Back-UPS NS 700

Constantin · April 4, 2024, 5:03pm

That’s the theory. In practice, some of the settings are not as simple to implement as one would think. Sort of like ACLs.

Davvo · April 4, 2024, 5:17pm

Often NUT feels like a lottery.

chuck32 · April 4, 2024, 5:35pm

This is mostly from my bookstack instance (unfortunately you cannot simply copy and paste). It’s been a while since I configured it. If someone has remarks / catches an error or has an improvement, feel free to chime in.

TrueNAS SCALE Settings

Set a user and a password

Settings on Ubuntu Server

Install NUT

sudo apt update
sudo apt install nut-client

Create /etc/nut/nut.conf with the contents of

MODE=netclient

Create /etc/nut/upsmon.conf

RUN_AS_USER root

MONITOR ups@<YOUR-TRUENAS-IP>:3493 1 <USERNAME> <PASSWORD> slave

MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h"
NOTIFYCMD /usr/sbin/upssched
POLLFREQ 2
POLLFREQALERT 1
HOSTSYNC 15
DEADTIME 15
POWERDOWNFLAG /etc/killpower

NOTIFYMSG ONLINE    "UPS %s on line power"
NOTIFYMSG ONBATT    "UPS %s on battery"
NOTIFYMSG LOWBATT   "UPS %s battery is low"
NOTIFYMSG FSD       "UPS %s: forced shutdown in progress"
NOTIFYMSG COMMOK    "Communications with UPS %s established"
NOTIFYMSG COMMBAD   "Communications with UPS %s lost"
NOTIFYMSG SHUTDOWN  "Auto logout and shutdown proceeding"
NOTIFYMSG REPLBATT  "UPS %s battery needs to be replaced"
NOTIFYMSG NOCOMM    "UPS %s is unavailable"
NOTIFYMSG NOPARENT  "upsmon parent process died - shutdown impossible"

NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC
NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC
NOTIFYFLAG LOWBATT  SYSLOG+WALL
NOTIFYFLAG FSD      SYSLOG+WALL+EXEC
NOTIFYFLAG COMMOK   SYSLOG+WALL+EXEC
NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC
NOTIFYFLAG SHUTDOWN SYSLOG+WALL+EXEC
NOTIFYFLAG REPLBATT SYSLOG+WALL
NOTIFYFLAG NOCOMM   SYSLOG+WALL+EXEC
NOTIFYFLAG NOPARENT SYSLOG+WALL

RBWARNTIME 43200

NOCOMMWARNTIME 600

FINALDELAY 5

Replace <YOUR-TRUENAS-IP>, <USERNAME>, <PASSWORD> accordingly. The latter two are the name and password you set in TrueNAS under the UPS service tab.

Create the paths for pipefn and lockfn, i.e.

mkdir /etc/nut/upssched/

Create /etc/nut/upssched.conf

CMDSCRIPT /etc/nut/upssched-cmd

PIPEFN /etc/nut/upssched/upssched.pipe
LOCKFN /etc/nut/upssched/upssched.lock

AT ONBATT * START-TIMER onbatt 60
AT ONLINE * CANCEL-TIMER onbatt online
AT ONBATT * START-TIMER earlyshutdown 60
AT ONLINE * CANCEL-TIMER earlyshutdown online
AT LOWBATT * EXECUTE onbatt
AT COMMBAD * START-TIMER commbad 120
AT COMMOK * CANCEL-TIMER commbad commok
AT NOCOMM * EXECUTE commbad

Change the timings to your liking here.

Create /etc/nut/upssched-cmd and make it executeable (chmod +x)

#!/bin/sh
 case $1 in
       onbatt)
          logger -t upssched-cmd "UPS running on battery"
          ;;
       earlyshutdown)
          logger -t upssched-cmd "UPS on battery too long, early shutdown"
          /usr/sbin/upsmon -c fsd
          ;;
       shutdowncritical)
          logger -t upssched-cmd "UPS on battery critical, forced shutdown"
          /usr/sbin/upsmon -c fsd
          ;;
       upsgone)
          logger -t upssched-cmd "UPS has been gone too long, can't reach"
          ;;
       *)
          logger -t upssched-cmd "Unrecognized command: $1"
          ;;
 esac

Profit

sudo systemctl restart nut-client restart the client after making all the changes.

You can then use sudo systemctl status nut-client to check the status.

Some remarks

Shutdown timer in TrueNAS should be long enough for the other server to have all timers passed and shutdown gracefully.
Make sure your UPS has enough power to supply both machines with energy until they are shutdown.
Test this setup before relying on it.

iTsMaaT · April 6, 2024, 4:07am

Hi, I did all the configurations, and i get that, so you know how I could fix it?

chuck32 · April 7, 2024, 8:53am

Try rebooting the server. Maybe not all services needed were restarted.

You could also kill power shortly and wait a few seconds while SSHing into the server to see whether you get a notification and see if it would work nonetheless.

I googled for that error and didn’t find a definite answer. Sorry I’m not of more help here.

iTsMaaT · April 7, 2024, 6:19pm

Looks like those errors aren’t causing any issues, as i’ve tested everything and everything works as expected, the only thing i’m missing is maybe make it so when the power comes back, it wakes up my servers, if it’s even possible

Constantin · April 13, 2024, 11:58am

That’s not something NUT does, AFAIK. It is something you can enable via the BIOS, however. For supermicro here is a description. Other OEMs will have different ways of auto-powering back on. It’s been working pretty flawlessly here.