So, it “appears”, that the only “proper” way to shutdown a TrueNAS Scale machine, is with:
/usr/bin/midclt call system.shutdown
This came from the very developers in their youtube channel, BUT,
I’m having a hard time believing that IX-Systems did not use the proper systemd setup in order to properly shutdown their middleware for ANY shutdown or reboot commands (power button, poweroff, reboot, etc.).
So, I’m here asking for the “official” answer to this important process:
Should all people with scripts CHANGE their shutdown commands to the above middleware call?
In another thread about a similar task on CORE, neofusion tracked that it will have to be an API call on SCALE but I tracked down that it’s safe to call shutdown on CORE (the rc.d system will do what’s right).
Its always been recommended to try and use the API calls for specific tasks on TrueNAS, even back on the CORE days. TrueNAS is not some generic BSD/Linux distro. It’s a pretty specifically built appliance, designed to be used through the UI and APIs. While there is access to a shell prompt, it is always recommended to use the API commands for those kinds of operations, since that’s how we test and validate the right things happen here internally. We don’t spend cycles wiring up every shell command (of which their are many) to hook back into the API to do the correct and tested shutdown procedures.
Calling the API will essentially run the exact shutdown procedure you would get had you clicked the “Shutdown” button in the UI. Calling that means as we make changes and improvements to the backend across new versions, your scripts will always be following the recommended and tested procedures for whatever task you are trying to automate.
Out of curiosity (mainly because I found this a fun “thread to pull on”/diversion/sidebar), why did you choose to not use the ready-made solution (rc.d) (I mean, I know the rc.d scripts are there, did you find those “not enough” -i.e., did you find a deficiency)?
We used selected rc.d scripts in various places for specific parts of the middleware. But relying on it to do everything right was out of scope and not really feasible. We already had to fork rc.d for that limited usage in the product as it was. So the version in BSD base was already insufficient for our use-case and it was apparent years gone past that we just needed to have finer grained control over what happens, what order, and timers / triggers, etc.
Same goes for systemd. Systemd is far more powerful than rc.d and we possibly could have used it “out of box” to do what we needed here, but its an effort vs impact calculation. Would take a fair bit of work to make that all work properly, and become the supported thing, but we already had a working API which did the right things in the right sequences, so not worth the effort at this time.
Thanks. I’m going to try and find that fork and play around a bit (just for the sake of playing around). Seems like fun. Any and all tips welcome (if you remember any “odd ducks”, feel free to point them out). …not sure why that was such a fun diversion though; it was kind of like the first time you wrote a program to parse a sentence.
Worth noting; I’m not sure how SCALE does it, but in Proxmox, it helps to have qemu-guest-agent installed on all your VM’s to make the shutdown process a lot smoother and more graceful as well.
Didn’t know what that was, googled and found basic information.
See! Fascinating.
The only role QEMU plays in guest commands is unmarshalling and remarshalling the input and output. This means that data from the guest is not being sent directly to a management tool which significantly decreases the guest attack surface.
In the inner workings of a shutdown/startup “thing” I can foresee problems with dealing with things like interrupts and/or hangs. And after a very quick github search on freebsd’s rc.d I found something touching on that topic right away.
# If requested, start a watchdog timer in the background which
# will terminate rc.shutdown if rc.shutdown doesn't complete
# within the specified time.
#
_rcshutdown_watchdog=
if [ -n "$rcshutdown_timeout" ]; then
debug "Initiating watchdog timer."
Obviously, I don’t have any context at the moment (I’ve spent longer typing this post than searching/reading) but, this is fun.
The shutdown command used when the UPS service in SCALE needs to bring the system down appears to be /sbin/shutdown -p now, unless the user has overridden it with their own command, is that intentional or an oversight?
AFAIK, ASPI shutdowns (ie power buttons and VMs) also trigger /sbin/shutdown, rather than the API.
(Actually that’s a simplification, it’s really handled internally by systemd)
The point is, the power button should trigger a proper shutdown, and if a proper shutdown should involve the API, then the API needs to be tied back into systemd via a before/after type service.
im going to follow this, I have a need to shutdown my NAS at midnight everynight and using the built in cronjob im using shutdown -h now, if thats not correct I need to update that.
They tell you to use the midclt command in scripts in the T3 video, and kris even came here to say that you should use the midclt command to shut your system down instead of directly calling shutdown, poweroff or similar.
My Jira bug report is not related to that, it’s focused on a possible oversight in the UPS service.
I finally got around to following up on the UPS for CORE and I verified the shutdown command is in fact being called if the end user does not fill in the ‘shutdown command’ value in the UI.
shutdown_cmd = '/sbin/shutdown -p now'
History show this was added in 2019.
I checked the documentation and there is NOT a recomended value for this setting in the documentation
NOTE (for 13.3):
My recommendations (I know my recommendations are not worth much but this is a force of habit):
(lowest impact) Leave defult hard coded, and modify documentation for 13.0 and 13.3.
(highest impact) Change default.
NOTE: I typed and checked as much as I could but you may/should still quickly double check (links provided for your convience). I only spent 5 minutes on this task (I was moving quick and may have missed something).
Also, I was not able to locate the rc.d fork you mentioned. I know you’re busy but if you happen to have that location handy I’d still be interested in toying around.
EDIT: I forgot to mention; I said #1 was lowest impact becuase I previously verified there is a rc.d shutdown script which calls the middleware shutdown script (two links below).
On the one hand you have iX informing its users to invoke the midclt method of shutting down the server via scripting or the command-line. “Do not use the OS’s shutdown command.”
On the other hand, the default command for shutting down the server in the UPS configuration is /sbin/shutdown -P now
I had the same thought when we first starting pulling on this thread.