This is something I’ve seen quite often — automation doesn’t remove work, it shifts it.

Instead of doing tasks manually, you end up maintaining and validating the automation. And if that layer isn’t reliable or well-observed, it quickly turns into exactly what you described: “what broke overnight?”

From my experience, the missing piece is often visibility into the automation itself.

It’s not enough to automate onboarding, patching, etc. — you also need to monitor:

• whether workflows completed successfully end-to-end
• where they failed (and why)
• and whether the outcome is actually in the expected state

Otherwise, you’re stuck manually validating automated processes.

What helped us a lot here was using Checkmk to actually monitor the automation layer itself. Instead of just knowing that a job ran, we track if it really succeeded, if systems reached the desired state, and trigger alerts only when something is actionable.

Especially with things like mk-job (from checkmk), you can directly monitor automation jobs and workflows, which makes it much easier to see if something failed and where. Combined with proper checks and thresholds, this gives you a much clearer picture than just “job executed”.

For example:

•   failed patch jobs → alert with context instead of just “job finished”

•   onboarding workflows → monitored as services

•   dependencies → so you don’t get flooded with follow-up symptoms

That way, you move from “checking everything in the morning” to only reacting when something actually needs attention.

I’m also doing similar things in my homelab, and it’s a good reality check — if automation isn’t observable and reliable, it just creates hidden manual work.

So I’d say:

👉 automation without observability = hidden manual effort

Tools like Checkmk (and extensions like mk-job) really help close that gap by making automation outcomes visible and actionable — not just “executed.” 👍