Hi, folks. I come from pretty large infrastructures, as in \~300k+ servers. I wrote https://jonnywhatshisface.github.io/systemd-shutdown-inhibitor/ to solve problems I've hit in some of those infrastructures, and figured I'd share with everyone in case you may potentially have a use-case for it as well.

We had serious challenges around patch maintenance and management when we switched from SystemV to SystemD on (RHEL 6 -> RHEL 7) quite a while back.

Given the size of our plant and the count of unique hosts in the infrastructure (thousands of departments and super orgs, 97k employees - all with their own server infra, and just 15 operations members and 7 engineers globally) - the entire plant was setup to do rolling reboots with dynamically controlled scheduling that the users set their maint. windows. They handled things such as their own shutdown scripts for handling scenarios like HA failover, service stops prior to package upgrades and etc.

With the switch to systemd, we had to leverage offline-reboots (system-update state) to align with those strategies, and that introduced dual-reboots to every system because the updates would happen on the way UP while in system-update state, instead of on the way DOWN when the shutdown/reboot was executed. Why that's a big issue in that plant is because POST on some of these servers can take more than 30 minutes (think boxes with more than 1TB RAM, 12 NIC's, RAID cards, JBOD's attached, etc). This was turning simple reboots and patching into an hour long adventure in some cases, particularly when a host was being rebooted specifically for the purpose of rolling back a set of patches.

So, I had addressed this using a similar methodology to terminusd (though, not as feature-rich), and it resolved that after many years of just dealing with the ridiculous dual reboots.

Now that I've left the company, I had rewritten it into a daemon with far more flexibility because I was bored and wanted to leverage it on my own systems.

Then, a colleague I got pinged by an old colleague inquiring about ways to handle dyamically disabling reboot/shut entirely on boxes so that normal systemctl and /sbin/shutdown commands wouldn't work - so I decided to extend that functionality into it as well. Apparently, an HA pair that looked as though the other side was up was shutdown by someone on the operations team, and it had serious financial impact because the other node was not in a seeded state and couldn't take the handover.

I decided to take that scenario and cover for it in terminusd as well.

What came out of it is terminusd - a lightweight daemon that gives full control and flexibility over shutdowns and reboots by leveraging a systemd delay inibitor, and a shutdown guard that can dynamically enable/disable shutdown, halt, reboot and kexec based on environmental factors determined by administrator scripts.

To handle shutdown actions before the system goes down - and before systemd is even in a shutdown state - it registers a delay inhibitor. During this time, all systemctl commands work as normal and systemd is still in a 100% fully running state, but has a pending shutdown. That pending state is controlled by the InhibitDelayMaxSec parameter in logind.conf - which terminusd can optionally configure for you. The delay is only held as long as the inibitor holds it, or until this timeout is reached - at which point the shutdown/reboot/halt proceeds regardless of whether the inihibitor has finished (to prevent a total dead-lock/hang).

Commands for shutdown actions are dynamically configured as drop-ins or in the config file. It allows setting a full command to run (with args), optionally setting the user/group to run as, in addition to optional env for it, and can be marked as critical. The actions are executed in ascending order "priority groups," meaning commands you set with equal priority will run in parallel. Any task marked "critical" failing will result in not running any further priority groups and the inibitor will be released.

This is currently being used on large storage clusters and HA kits where shutdowns require things such as trigger failovers, migrating services and VIP's and etc, as well as stopping various services before applying patches/upgrades.

The shutdown guard can disable system-wide reboots, shutdowns, halts and kexecs, even if the command is issued as root. It can either run your guard command/script/binary in timed intervals with a configured threshold for failure - oneshot mode - which simply requires a zero exit of the command to re-enable reboots, where a non-zero exit will disable them, or it can run in persist mode where it attaches a pipe to the stdio of your script/command/binary and monitors it, logging all stdio/stderr to syslog. With the persist mode, your app only needs to write the command out to enable or disable the shutdowns on the system.

Currently, the persist mode is being used on HA clusters that the script is monitoring the readiness of the servers to take the handoff if one of them is rebooted. If at any point one is not able to take the handoff for whatever reason (reboots, service failures, etc) - then the reboots are disabled on the other side to prevent accidental reboots.

terminusctl allows you to actually visualize the action order, see the status of shutdown enable/disable state, stop/start the shutdown guard and reload the configuration live without restarting the daemon. This is useful for working on developing your shutdown guard scripts, configuring your shutdown actions and being able to visualize the result without having to restart the daemon. It can also be used to enable/disable the system-wide shutdowns from the cli on the spot, including to override shutdown guard.

If you find it useful, I'd love to hear about it. It may not be for everyone, but I'm sure someone else out there has some kind of need for it given we did.