I just need a command to manually shutdown of system1 from system2

[jackdinn]

I am running two TrueNAS systems:

1. Master: TrueNAS SCALE - Handles all primary storage and server applications.
2. Slave: TrueNAS Core - Acts as a backup server for the Master system.

I need a way to issue a manual shutdown command from the Master (SCALE) to the Slave (CORE) without affecting the Master itself. This command is intended for maintenance or other scenarios where I want to power down the Slave while keeping the Master operational.

Additionally, both systems are connected to a UPS, and during a power failure, both should shut down gracefully, with the Slave shutting down first and the Master following later.

---

What We’ve Tried:

• UPS Command Attempt:
  • Tried using upsmon -c fsd ups@<IP> to initiate a shutdown of the Slave.
  • However, this resulted in both the Master and Slave shutting down, which is not the intended behavior.

1. SSH-Based Shutdown:

   • Configured SSH keys for passwordless access from the Master to the Slave.
   • Encountered persistent issues with authentication and managing authorized_keys.

2. API Approach:

   • Investigated TrueNAS Core’s API to see if shutdown commands could be issued.
   • Could not find an accessible API endpoint for this task.

3. NUT (Network UPS Tools):

   • Attempted to use the existing NUT UPS service to send a remote shutdown command like upscmd -u upsmon -p fixmepass ups@192.168.1.100 shutdown.return from the Master to the Slave.
   • Configured NUT on both systems but faced issues with missing configuration files and upscmd returning “Connection refused” errors.

4. UPS Timer Settings:

   • Confirmed that both systems have independent shutdown timers configured through the UPS integration.
   • This ensures proper shutdown during a power failure but does not allow for manual control outside of UPS events.

---

The Problem:

1. I need a reliable way to manually shut down the Slave (Core) from the Master (Scale), regardless of the method.
2. This functionality should not interfere with the existing UPS-triggered shutdown configuration.

---

Questions:

1. What is the simplest way to achieve this manual shutdown command?
2. Are there other methods (besides SSH, API, or NUT) that might be more straightforward for this use case?
3. Has anyone successfully configured a similar setup and can share their approach?

---

Current State:

• Both systems are operational but have residual configurations from previous attempts (e.g., leftover SSH keys and partial NUT setups).
• Manual shutdown via upscmd is still not functional, and SSH into root based methods remains unachievable.

---

Any guidance or recommendations would be greatly appreciated. Thanks in advance for your help!

[neofusion]

Just posting a few things that come to mind, mostly as breadcrumbs:

• Use a shell script to ssh to the slave-system and initiate an orderly shutdown using /usr/bin/midclt call system.shutdown.
• Use websocket to issue the same command remotely. The API docs explains this to some degree. The “system.shutdown” call is defined as a “Job” and is called appropriately.

Maybe that will help, nor not.

[jackdinn]

@neofusion

Use a shell script to ssh to the slave-system and initiate an orderly shutdown using /usr/bin/midclt call system.shutdown.

well iv tried a hundred times to create a ssh to root but it wont let me. but i only did that because i thought any shutdown command would need sudo. Maybe yours does not?

@neofusion

Use websocket to issue the same command remotely. The API docs explains this to some degree. The “system.shutdown” call is defined as a “Job” and is called appropriately.

Well this is an odd one. I can not find the API in the gui anywhere at all. Its in Scale UI but not Core. But iv just found that if i use command sudo -i it gave this splash indicating that it does indeed have an API

jack@TruenasCore ~ $ sudo -i
Password:
Sorry, try again.
Password:

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

[neofusion]

@jackdinn

well iv tried a hundred times to create a ssh to root but it wont let me. but i only did that because i thought any shutdown command would need sudo. Maybe yours does not?

Don’t ssh as root, ssh as your administrative user and then sudo the command, if needed.

@jackdinn

Well this is an odd one. I can not find the API in the gui anywhere at all. Its in Scale UI but not Core. But iv just found that if i use command sudo -i it gave this splash indicating that it does indeed have an API

Top Menu | TrueNAS Documentation Hub states that the button to make an API key is in the top menu. That is also where you will find a link to a local copy of the API documentation.

You will need to figure out how to wrangle the JSON to authenticate and then issue the call. I don’t know how to do that.

[jackdinn]

o god, i completely missed that there.

I shall call this a solution in the hope that i can get the api to work. im sure i can.

Thank you

[John]

I’ve never tried this myself but I would imagine my first attempt would be to use the ssh command.

EX:
ssh ricky@remote.local -t 'shutdown -p now'

I know you said you had problems with ‘persistent issues with auth and managing keys’ but I’m not really sure I know what your dealing with so my answer assumes you’d have a set of users–or one–(added to the operator group, which can manage their keys properly and/or setup their ssh.confg file) that can issue the command to shutdown the system.

[jackdinn]

Hi, yeah I’ve set up key pairs for a lot of SSH stuff but I just could not get it to work when I tried to SSH to root. It’s either just me late at night with blurry vision or it’s a feature of Core that you can’t SSH directly into root.

You need sudo to run shutdown.

I have used it now with the API, but if I could not have done so I would have tried another user as suggested above as I don’t need to be root to execute root commands on a remote system. I just need the right user account with the right permissions and access.

[John]

Okay, let’s back up a step because I feel like this could be a “tempest in a teacup” situation. You are certainly allowed to use whatever method you want (if that’s the API, then sure, knock yourself out) but I think a few simple steps are messing you up.

1. never ssh in as root. I know there are several users here that recommend it–and I hope they are not sysadmins (I’m not but I’m almost 1000pct positive that is a bad idea) but I’ve argued with them about that on other occations. It’s far easier to keep your configs and permissions tied to users and not pass out root info to bob, ricky, and jane. …just don’t. You create/destroy users and add them to groups (done. simple. easy. do it this way).

2. to issue the shutdown command, you–or the user–needs to be added to the operator group. So, just add a user to that group and they can use the utility.

SHUTDOWN(8) FreeBSD System Manager’s Manual SHUTDOWN(8)

NAME
shutdown, poweroff – close down the system at a given time

SYNOPSIS
shutdown [-] [-c | -h | -p | -r | -k] [-o [-n]] time
[warning-message …]
poweroff

DESCRIPTION
The shutdown utility provides an automated shutdown procedure for super-
users to nicely notify users when the system is shutting down, saving
them from system administrators, hackers, and gurus, who would otherwise
not bother with such niceties. In order to use the shutdown command, the
user must have root privileges or be a member of the operator group.
…

3. Users should keep their own ~/.ssh/config file which looks like:

Host server.local      # or whatever your server is called.
            User ricky
            Hostname 192.168.x.x
            IdentityFile ~/.ssh/id_rsa
# you could have 50 other entries below,
# ...after cleaning, I have 15 server entries in mine...

4. Users manage their own keys. They can upload their public keys like:
   % ssh-copy-id -i ~/.ssh/id_rsa.pub server

So, for my method to possibly work, you–the admin, root, theman, etc–need to add the users you want to be able to shutdown or poweroff to the operator group. That it. They, can then use the one-liner above or create a function or a shell script out of it. You–or the admin, root, or theman–is totally hands-off. I, myself, strive for the KISS method and try to use it for as much as I can. For example, I don’t have to remember or rely on any API function calls or methods and almost fool proof (answers to questions like: “I need to input a password every time I want to shutdown server0” are dead simple to answer, like: “well, did you upload your private key, dummy?”). not to mention the bus-factor (question you should ask: “if you get hit by a bus, can the server still be shutdown by another user?”).

[John]

FYI (because I was curious–I was almost positive an engineered system like BSD wouldn’t allow root login via ssh by default but just wanted to check anyways):
% man sshd_config

 ...
 PermitRootLogin
         Specifies whether root can log in using ssh(1).  The argument
         must be yes, prohibit-password, forced-commands-only, or no.  The
         default is no. 
         ...

So, if that’s not clear enough for everyone reading along: “If you allow root access via ssh, you are at best a hackers wet-dream.”

[jackdinn]

ofc you are right, but just fyi i did try al that

jack@TruenasCore ~ $ sudo cat /etc/ssh/sshd_config | grep Perm
#PermitRootLogin no
#PermitEmptyPasswords no
# the setting of "PermitRootLogin prohibit-password".
#PermitTTY yes
#PermitUserEnvironment no
#PermitTunnel no
#       PermitTTY no
jack@TruenasCore ~ $ 

I created the key-pair, I put the pub key in the remote roots authorized_keys, I connect (tried), I got the known_hosts warning, I type yes to add the host to known_hosts. I change the PermitRootLogin to yes but still I get root@192.168.1.117: Permission denied (publickey).

I have really struggled with SSH by key pair before so even though I was not actually going to be using it, I wanted to know why. Why would it not connect?

Anyhow, I spent a few hours trying. I failed, and could not get past that error. I have many other ssh key-pair connections that work so i just assumed maybe they made it so you can not even ssh into root at all.

I have put everything back how it was now as the API was so much easier, but I’m sure I will be back at this SSH and key-pairs again with a normal user some time in the future (just not with root).

[neofusion]

@jackdinn

I change the PermitRootLogin to yes but still I get root@192.168.1.117: Permission denied (publickey).

As @John and I have told you in this thread, please don’t do this. You should never have to SSH as root like that. Never ever. This is bad practice in any *nix based OS.

Directly editing a /etc-file is also the wrong way to go about things in this appliance OS, as those changes won’t stick around in an update and some won’t even survive a reboot.

SSH as your administrative user and then use sudo command to do whatever you need. Or use the the -t flag like John mentioned and run the command (with sudo) directly in the same ssh-command.

@John

to issue the shutdown command, you–or the user–needs to be added to the operator group. So, just add a user to that group and they can use the utility.

White shutdown does indeed shut the system down, it should not be used in TrueNAS under normal circumstances because it’s akin to you pulling the rug out of the middleware. During that fire sale of a shutdown there’s a fair risk things won’t get preserved as they should.

The correct shutdown command is, as I said earlier, midclt call system.shutdown (if you have midclt in your path).

[jackdinn]

@neofusion

As @John and I have told you in this thread, please don’t do this. You should never have to SSH as root like that. Never ever. This is bad practice in any *nix based OS.

yea and as i have said, im not trying to.

I was never going to leave it that way. It was just that it really bugged me as to why it didn’t work. I’m hazy with SSH key pairs as it is and I thought I had misunderstood something that would crop up later, so I wanted to know before removing it all.

@neofusion

The correct shutdown command is, as I said earlier, midclt call system.shutdown (if you have midclt in your path).

hmm, I shall see if I can implement it but I could be back to really struggling with the API if that command is not exposed.

Thx for this info though

[HoneyBadger]

@jackdinn

hmm, I shall see if I can implement it but I could be back to really struggling with the API if that command is not exposed.

Even if you’re implementing this through an SSH connection, the midclt call is a way to locally make that API call once you’re hooked in.

[John]

@neofusion

White shutdown does indeed shut the system down, it should not be used in TrueNAS under normal circumstances because it’s akin to you pulling the rug out of the middleware. During that fire sale of a shutdown there’s a fair risk things won’t get preserved as they should.

As a programmer I find that very concerning (I would imagine that a room of engineers would have used the rc.d system because “why reinvent a new init?”). Sorry, but I call BS on that statement but if it is true, that seems like some low-hanging-fruit. I’d kind like to investigate that (but honestly, if I did find some time to look into that it would only be out of curiosity because I don’t really have a need for this). …I dunno, seems fishy (sorry).

For all annon guests, and future people reading this thread: never ssh in as root (a certified true statement by at least two people :] ).

[neofusion]

Directly from HoneyBadger’s mouth:

Is it possible to shutdown multiple hosts from a single UPS?

If you’re going to drive a scripted shutdown of TrueNAS remotely, please either use the remote API for system.shutdown or midclt call system.shutdown over SSH to give the middleware a chance to clean up. [TrueNAS Tech Talk (T3) Podcast - E005 - Cache Rules Everything Around Me - November 22, 2024] Timestamp 14:34 if it doesn’t come through for some reason.

[John]

Well, thanks for tracking that down but, that still seems like a huge overstep if you ask me.

[John]

My curiosity was killing me. So, from FreeBSD rc.d scripty stuff…

shutdown - This keyword is to be listed explicitly if the service needs to be stopped before system shutdown.

Note: When the system is going to shut down, /etc/rc.shutdown runs. It assumes that most rc.d scripts have nothing to do at that time. Therefore /etc/rc.shutdown selectively invokes rc.d scripts with the shutdown keyword, effectively ignoring the rest of the scripts. For even faster shutdown, /etc/rc.shutdown passes the faststop command to the scripts it runs so that they skip preliminary checks, e.g., the pidfile check. As dependent services should be stopped before their prerequisites, /etc/rc.shutdown runs the scripts in reverse dependency order.

If writing a real rc.d script, you should consider whether it is relevant at system shutdown time. E.g., if your script does its work in response to the start command only, then you need not include this keyword. However, if your script manages a service, it is probably a good idea to stop it before the system proceeds to the final stage of its shutdown sequence described in halt(8). In particular, a service should be stopped explicitly if it needs considerable time or special actions to shut down cleanly. A typical example of such a service is a database engine.

I don’t have access to TrueNAS at the moment, but I spent 5 minutes on github and found some rc.d scripts in the system (i.e. not, /usr/local/etc/rc.d) rc.d directory (specifically at: /etc/ix.rc.d/) and from a real quick look, there appears to be a “shutdown” script. However, if you people are telling me a user still cannot shutdown the system using shutdown/halt/etc. that’s really too bad iX couldn’t get the middleware to shutdown using the built-in method.

[jackdinn]

@neofusion

midclt call system.shutdown

Well, all I can say as a layman is I hope midclt call system.shutdown will be okay for me on a regular basis. If not, it will have to be 2 servers on 24/7, which is not necessary since one is purely a NAS and only receives replications once every 3 days.

[fchk]

When building server systems I insist having IPMI hardware management. All decent server mainboards have this. This is completely independent of the PC software itself and works even on the BIOS/UEFI Setup level.

IPMI allows me to turn the system on and off remotely, boot from a virtual CD and have a remote, hardware-based VNC session.

[John]

@fchk

When building server systems I insist having IPMI hardware management. All decent server mainboards have this. This is completely independent of the PC software itself and works even on the BIOS/UEFI Setup level.

IPMI allows me to turn the system on and off remotely…

I don’t know IPMI (forgive me, I’m not a “professional sysadmin”) but I read the wikipedia quick, and if I understand the situation; you would NOT be allowed to use IPMI on a server running TrueNAS then (because in the case of TN Core, the IPMI driver uses shutdown(8) and this is NOT recommended by iX, apparently). Does that sound about right?

Assuming you are a professional; do you have TN instances in “production” and have you used IPMI to shutdown/reboot a system?