What IT tasks are you comfortable letting automation handle end to end?
Posted by Muhammadusamablogger@reddit | sysadmin | View on Reddit | 74 comments
trying to sanity check how far people are going with automation.
What IT tasks are you comfortable letting run end to end today without human intervention? And where do you still insist on checkpoints?
We're debating how aggressive to be with access provisioning and onboarding. Some tools, including newer ones like Siit, make it easy to automate a lot quickly, but I've also seen similar pushes with ServiceNow and Freshservice that didn't always age well
Vivid_Rate_6118@reddit
We have big customer whose employees exclusevly use email as only communication media with us. Problem/issue management is real pain for us as it is hard to track numerous email threads with issue updates on their side. We have to read their emails and sync with our issue management system (Nifty SaaS platform). It is anoying and engineers are sometimes very angry becose of this.
We automated it by attaching automation platform to our email account and we do this:
- check if email originates from our customer
- check if email has important contacts at customer side (implies excalations and bring awarenes of our human operator)
- check email context with AI (automation tool can prompt to our AI - anthropic/openAI - automation platform allows you to connect to any LLM) - AI returns structured data
- retrieve opened issues from issue management platform
- check with AI if there are matchings with already reported issues
- depending on quality of match workflow can involve human to decide should existing ticket be updated or new would be created
- let AI check if we need more details from customer
- reply to customer about updates, further questions etc.
- updates the relevant team to their favourite channel (SLACK or MS Teams)
We do couple more automations but this one brought lot of happiness in our office :)
Weekly_Accident7552@reddit
Access provisioning runs end to end for us on new hires. Manifestly kicks off the checklist from HR ticket auto assigns AD account 365 groups VPN and app installs then pings for exceptions only. Been solid two years no major oops. Still checkpoint deprovisioning since offboarding edges cases bite hard.
Vegetable_Carpenter5@reddit
Device distribution, granting account accesses for newly onboarding employees, revoking accesses for offboarded employees, etc. if not just to save admins time and effort. These getting done consistently has reduced the security risk of having steps missed that are all too common when humans have to do the boring stuff, especially offboarding. No much oversight is required once these are set up. Rippling IT is another solution that can automate these, especially any tasks associated with employee changes like promotions, onboarding/offboarding, relocation -- Adding to the mix because I'm a Rippling employee.
Hotshot55@reddit
Everything should be handled end-to-end by automation. Feel free to add in approvals wherever you'd like, but don't rely on manual processes still.
fatmanwithabeard@reddit
not everything.
ticket closures, setting the alerting system to maintenance, and off boarding admins all need manual touch.
Hotshot55@reddit
I can't think of any good reason why those shouldn't also be handled via automation.
fatmanwithabeard@reddit
ticket closures should only be done by the person who worked the ticket.
the alerting system should automatically come out of maintenance, but going into maintenance should always be deliberate choice. I've never had an issue where getting alerts because someone forgot to set maintenance and that made things worse. I've been privy to several that went unreported because of automated maintenance.
Admins always have weird and usual access points. You want to go over everything (and unless they're being fired, you want to go over everything with them) to catch all the stuff. You're simply not going to find all the weird appliances, and non standard systems that were part of the X or Y project unless you have perfect documentation. Generally an admin leaving is a good excuse to review all of your systems documentation, and touch a lot of stuff you generally ignore. Unless you're in hell, you shouldn't have a terribly high turnover of your admins.
Hotshot55@reddit
The person can move it to a completed stage, and then automation can validate data quality and then move it to a closed state. Do you think automation is going to just close your tickets while you're working them?
Yeah, we schedule maintenance windows and then things go into maintenance mode automatically during that window. It's still a deliberate choice being made. This seems like more of an issue if you're doing ad-hoc work.
Sitting down with someone to do handovers is kinda irrelevant in the automation discussion. You can have that discussion whether you've automated user off-boarding or not. If automation is the standard, you don't have the opportunity for people to create these one-off accounts where you'd only know about it if you asked the person.
fatmanwithabeard@reddit
Data quality isn't validated before the close button is available? In either case, the idea is the same, a person had to take an action before the ticket was closed.
So, this is entirely in the worst case line, but I've been near party to an automated maintenance window coinciding with a power outage at a secondary site. The power loss caused a freezer to warm, and the loss of major sample collection. Since the remote system never lifted the maintenance window, no alert was ever sent.
Hand scheduled maintenance, with an enforced end time requirement is fine. Especially with a cultural requirement that a person is involved with the maintenance until they either hand release it, or after the monitoring system does so. Systems that can schedule maintenance without a human doing stuff is bad.
If you have automation for every weird appliance and tool in your environment, I envy you (unless you work for a federal lab, in which case, sorry). There are three of us who deal with the backend storage for the cluster. All of the storage systems have local accounts, because the cluster interconnect and management networks are not allowed to be directly connected to any other networks. There's a small number of total devices like that, and I can think of no valid reason to allow those networks to talk to the rest of the world. There are whole slews of medical devices and lab equipment that have special rules (and the argument about life dependent machines and local admin is long and stupid (if you've ever had someone suggest a laptop as a domain server for giant hospital, you've been in meetings at least as terrible as I have)
Murhawk013@reddit
Yup everything can be automated, it’s mostly politics that prevents these improvements from happening.
TheDaznis@reddit
Sure, I just have a ticket with a "service" provide whose automation broke, the ticket has been "stuck" without progress for 3 months now.
It's like everybody forgot what happened to a certain antivirus software a few years back, or facebook when it's network was gone from the internet. Or the almost monthly outages (not complete but location or certain services) of AWS, Azure and google.
Hotshot55@reddit
That sounds more like a service provider problem rather than an automation problem.
jakesps@reddit
100% Agree.
Caveats: have good error handling, notifications, and unit tests (or tests that inspect output and run edge cases).
deacon91@reddit
Automations that are solving clearly well understood problem within fine scoped end result.
https://xkcd.com/1319/
whythehellnote@reddit
Or more usefully https://xkcd.com/1205/
There are other benefits of automation other than timesaving though -- consistency for example.
deacon91@reddit
I don't think those 2 strips are at odds. Both time-saving and consistency is good, but automation has a cost too.
jhaant_masala@reddit
Building container images
deploying said images to Kubernetes clusters
creating VMs via Terraform and provisioning them via ansible
creating reproducible development environments using docker-compose and KIND clusters
TLS certificates for various TCP services
I do realise this is more DevOps than sysadmin.
The last part on certs cannot be emphasised more, but do realise - my scope of work does not involve “appliances” or similar systems.
fatmanwithabeard@reddit
I don't deploy anything that isn't automated. Configs in a test branch, verify tests from Jenkins, then isolate a node and do a single deploy of that brand there, and pick through it. Reboot the node, post the test results, ask for a +1, merge the branch, and the deploy goes live whenever restarts happen.
VM build and deployment is fully automated. I mean, fill in the worksheet, which generates the ticket, gets an admin review, and the build button is in the ticket, which will do all the work and close the ticket. It's nice.
Dev environments...really depend on what's being developed, and by whom. Mostly, those end up being workstations some place that I don't deal with.
DevOps is just a stupid name for a reasonably senior sysadmin. I'll admit I'm old, but I really hate the term; it feels too much like people want to give admin access to prod to devs (never, ever, ever)
jhaant_masala@reddit
I don’t think we’re the same, purely by virtue of experience.
Your VM creation is via a worksheet, my VM creation is using GitHub actions workflows.
DevOps doesn’t mean handing out prod access to devs - we just state that if you want to own it, we’re hands-off.
Under those conditions, if shit breaks, we’re not responsible. We thankfully have management buy-in on this because all senior management folks were once technical.
Because of this, there is a clear understanding - we own the infrastructure, the devs own the code.
fatmanwithabeard@reddit
The worksheet is part of the ticket, which feeds into github, which is where the ansible configs live. VMs are for interfaces and hosting data for papers, user level stuff. Our user interface people (they're like helpdesk, but they're all science phds with good tech skills) do the verify and button push that feeds it into ansible. Mostly they're checking to see if someone is looking for something to do post compute renders on, but asked for a webhost, or vise versa.
DevOps is still a stupid name. I remember arguing about it when it was mostly a bunch of senior linux backend people pissed off that some guy fixing exchange mailboxes had the same title tree as the guy running the entire backend infrastructure. It was stupid then, and remains so.
My favorite incident was a group that swore they could run their own servers as long as we built them to our internal standards come begging for help because someone had deleted /etc/. I bet my jr. admin I could have it back up before she could get the restore tape in place. She didn't even get to the library before I was done (this was 15 years ago, and the process in that place was worse than places I worked at in the 90s. It was almost sane when I left.)
mesaoptimizer@reddit
I try to automate everything but the automation. Any action that's repeatable, has known start parameters and known end state is a good candidate for automation. Despite what people will tell you, there is always enough work to do no matter how automated different processes are, and as requirements change, you have to update the automation, not everyone has the skills to do this, it's surprisingly difficult to automate your way out of your job.
Key thinks to automate, user lifecycle management, Ideally it should be to the point where IT doesn't have to DO anything when HR hires a new person to get that person an account, probably most of if not all of their access. When that user changes departments, or roles in the organization this should kick off automation to either review, revoke or change access. When that user is terminated in ERP automation should revoke their access and disable their account and delete it after a waiting period.
Things you shouldn't automate, or need human intervention at the beginning, anything where the source is untrustworthy, don't completely automate access requests, have them hit a review step first so they can be sanity checked.
Automate your build processes for servers and such, even one offs, the automation serves as documentation of what was done so it can be repeated if needed and the biggest advantage to automation is that it reduces human error, an automated workflow is never going to skip a step because it had a late night last night.
US_AxisOfMorons@reddit
What are you using to automate end-user on and off boarding?
ChevronEncoder@reddit
Depends on your environment. If you're M365 like many around here are, then PowerShell, Power Automate, and APIs are a good place to start.
fatmanwithabeard@reddit
How do you handle oddities, user access to obscure or non standard systems? Do you have security levels to deal with where your standard corporate environment can not talk to certain internal environments?
I've been frustrated by every single user off boarding tool and process I've dealt with in the last 30 years. It got worse when I went from doing backend work to HPC, and no matter where I've been, I've had to fight to get notified when people lose access to systems.
Onboarding is generally easier to deal with, I think.
mesaoptimizer@reddit
So, I use a homespun C# application, for offboarding the workflow is pretty simple. I have a service that checks a trigger table in our ERP, whenever a termination is processed in the ERP system (actually any time a substantial change like department or role changes as well). This includes a little about the change. If it was a department change, the service calls a proccess that removes the user from all groups in AD. If they were terminated it disables the user account, and adds another entry to the trigger table to delete the account and sets it for 2 weeks process runs ever 5 minutes. This means as soon as HR marks an employee terminated they lose their access, 2 weeks later we delete their account.
Don't rely on people communicating, don't rely on them automating communicating, take action on the business process and do whatever you have to do to tie in with your ERP system. You cannot do effective IAM if you do not have at least read access to the source of truth for who is and isn't an employee.
fatmanwithabeard@reddit
Heh.
I've worked in higher ed. The research computing group didn't have (or want) access to AD or the ERP system. We need someone else's automation to trigger a ticket to us so we know that someone has left. We could't even rely on access patterns for anyone but the major users. Some lab assistant will get their access, test it, and then not login for two years before slamming their lab's queue for a month. Then nothing again for who knows how long.
ChevronEncoder@reddit
If it can't be automated directly, the communication can be automated. Someone has to be the authority on access to any given system, and that person is going to be reachable online somehow. Your ticketing system could have something like this already or easily created, or in M365 you can make use of Graph, Power Automate, Planner, and/or other tools to automatically alert people do things and manually confirm they did it then trigger the next step.
If the problem is HR not telling IT when they've let people go or changed their role, there's no amount of automation that can fix that if you can't integrate your HR system into your greater automation scheme. That's a managerial problem.
fatmanwithabeard@reddit
This is the issue. We're not exactly IT either, and that really rankles the IT group. We're research support, and we're separate because the research groups got really tired of the specialists they needed help from being busy dealing with general issues.
IT pretends that it's impossible to know who has access to our stuff, but...cluster access includes specific AD groups, so, yeah.
Icedman81@reddit
You could always outsource that back to HR. Using something like Adaxes. Assuming a Hybrid environment though, don't know how it goes with Clöd-only.
cjbarone@reddit
This guy automates!
These actions were performed by a bot /s
Secret_Account07@reddit
Patching. Not only the patching itself (grabbing monthly patches and packaging baseline by OS) but also failures.
Any time patching fails we get a ticket to investigate. This stops the “good enough” mentality where 99% of servers are patched but this one hasn’t patched in 6 months.
It’s good to automate and reserve the remaining work for humans.
0zer0space0@reddit
What do you use to automate this? I’d be happy just getting a decent report of what servers are missing this month’s patches on certain dates. I kind of had this when we were using SCCM for patching. At least the SCCM job will tell you what succeeded and what failed. But if you happened to not have an oddball machine in SCCM devices or in a deployment, it wouldn’t show up. I have a nice query for Defender for Cloud but it just kind of stops there. I haven’t found a way to email the results on a certain day to me. So do I just get a list of VMs from vCenter so that I know I have all the VMs, and then ask it to login to the guest OS and check for latest installed patches and whether the VM has rebooted? Idk , looking for ideas here. Thanks
Secret_Account07@reddit
So we use big fix , which creates a baseline (MS February 2022 server 20xx) and targets all servers monthly during patching window. When any component fails (KB123454) it generates a ticket. It could just be a simple restart needed or it truly failed. Either way a tech gets that ticket.
I fear I won’t be much help for SCCM as it’s been almost 10 years for me lol. I could be hallucinating but I thought there was a way to use Powershell or WMI too and query using “get-hotfix” and seeing if servers are missing newest cumulative but I imagine you’d have to hardcore that each month.
TBH even a report showing missing cumulative on servers would probably be good but i don’t have the SCCM knowledge to know how you setup that report.
Patching is one of those things where if a server exploit is taken advantage of someone is going to want to blame IT so we’re pretty highly prioritize it. Just a CYA thing.
Carter-SysAdmin@reddit
basically all of onboardings and offboardings can be successfully done with automation no problem if you have the right tools or know what you're doing.
application access requests, approvals, and access controls can be done no problem with automation - this used to take a lot of code, but depending on your primary IAM and tech stack it can generally be done with no-code these days.
automation of group membership, OU membership, etc can all be fully automated if you leverage your actual HR data and your HR team actually updates things the right way.
workflows for things like reminders, checks, and a certain amount of audit-prep or audit considerations can also be automated so you're not filling out data manually.
keeping reports shared out to things like security teams/auditors, etc can be fully automated if the reports are getting good data.
make sure your MDM, Identity, Inventory, and HR data are all fully on lock with each other and things start to fall into place.
fatmanwithabeard@reddit
You've never worked with the government or a university have you?
I'm usually happy if I learn that someone is being hired/leaving the day it happens. I've had people switch labs twice before hearing about it, and which lab has access to which dataset is a huge deal, and because posix groups are stupid, which lab you work for is what data set you have access to.
Carter-SysAdmin@reddit
I worked for a "premier public, research-intensive flagship university" for 13 years before I entered the private sector, and I agree that automating offboardings there at the time could only get you so far with the way many things were handled.
AdmRL_@reddit
Better question really is why you'd be more comfortable with a person, who could be tired, poorly, distracted or could just be having a bad day, handling something critical or frequently done over code which suffers none of those problems, and what's only concern is whether it has power and an environment to run and access to whatever system it's accessing?
Stop looking at tools like that in my opinion. Unless you have decent budgets, lots of time, adequette FTE and management investment you aren't getting the most from them.
It's quicker to learn basic JSON and REST API norms & either Invoke-RestMethod syntax in PS, or learning Python's requests lib to set up some scripts in an Azure/AWS/etc than it is to try deal with ServiceNow's shit properly.
pdp10@reddit
Imbrex@reddit
I'd say this depends on how you're automating. If using a no code solution I'd say very little. If using code I've written, most things. Because I can fix it if there's an issue.
JuicedRacingTwitch@reddit
All of them, people make mistakes, computers do not. I have always explained to management/leadership "If you cared about this process then you would automate it."
fatmanwithabeard@reddit
computers just let humans make many mistakes really quickly.
automation doesn't reduce errors--testing reduces errors.
my favorite was a weird linux box that had some outside of normal user provisioning automation. it created users with spaces in their usernames. someone broke permissioning in /home. I wrote a script to fix it...that did not account for usernames with spaces. After fixing all of that, I got screamed at for two hours that it was too hard to explain to the people that used that tool that usernames could not have spaces. Which was weird because the guy who had to contact all the external customers and tell them that their usernames had changed didn't blink an eye.
Recent_Perspective53@reddit
Employee offloading, new employee setup. Repeatable actions by humans can easily be automated.
StrayHearth@reddit
Offboarding is the easiest one to fully automate. It's time sensitive and rule based. I'm much more cautious with onboarding.
-UncreativeRedditor-@reddit
Funnily enough, I’m the other way around. It’s perfectly fine if my automation script misconfigures a new user. I can just delete it all and try again. Automating offboarding can be a bit scarier because I don’t want to remove access or delete data from an active user by mistake.
Centimane@reddit
You could step-wise this to make it less risky if you're really concerned.
Then you get the immediate remediation (the user can't use the account), but you've not deleted anything. Step 2 only works if step 1 was done, which means you'd need to make the same mistake twice for a problem to manifest - far less likely
fatmanwithabeard@reddit
Depends on what your disable does. I've had all kinds of fun with people who run part of their department's workflow through stuff in their home directory, or through auto fire scripts run by their user. Doesn't matter that the user can't login, as long as the script fires.
I generally have my user destruction process as a disable phase 0; for admins, devs, and power users an investigate non standard access phase 0.5; a snapshot, redirect and rename phase 1, and a delete phase 2. While the process is scripted, it's not automated (triggered by non human input).
-UncreativeRedditor-@reddit
Yeah I’ve got it configured to disable and soft delete now, so in reality the only risk with our current automation would be a temporary disruption to our users if it were somehow triggered for the wrong person.
kissassforliving@reddit
I worked at a company where the off boarding script broke and ex employees had access to email and accounts for months after departure. Big Media company….
Arudinne@reddit
We automated that after HR decided terminating people after 5PM was not only acceptable, but something we should handle without being given prior notice.
As if we're glued to our company email 24/7.
PaidByMicrosoft@reddit
This vague post with zero interaction from OP, combined with his username and post history, looks like he is simply farming answers he can just repost on his blog.
2cats2hats@reddit
None. However, I am fine with automating all the things if notifiers are involved. Could be email/SMS/ntfy/smoke signal.
Point is automate but someone needs to keep an eye on logs and daily reports, in my opinion.
420GB@reddit
I mean, pretty much anything. I'm struggling to think of anything I wouldn't trust automation or a script with. A scripted process is always more reliable than a human one.
dinoriki12@reddit
The risk isn't automation itself. It's people assuming automation understands context when it doesn't.
titlrequired@reddit
*thing
s3xynanigoat@reddit
I'd automate you if you asked me this question in the office.
KrazyGonk404@reddit
I would limit automating onboarding to the simple things (software deployment/removal level stuff), this is the time period that you are able to build a good relationship and manually working on tasks will really help give you a positive image, which can go surprisingly far. That being said, after the onboarding process is done, automate anything you need to reliably repeat.
itishowitisanditbad@reddit
...what?
whythehellnote@reddit
I guess the idea is if you spend an hour with someone fixing their computer they're grateful as you have spent a lot of time with them. If you simply run a script you'd pre-made and it fixes the problem then it wasn't a major issue in the first place.
Its analogous to the "hitting with hammer $1, knowing where to hit $9999" invoice legend. If you spend 5 hours looking like you fix a machine, you've done a valuable. If you simply fix it in 5 seconds, you aren't. It's wrong, but perceptions are often important.
JuicedRacingTwitch@reddit
Is this a troll?
whythehellnote@reddit
For the actual "Doing" stage
First stage - check list. Capture information upfront you need and put in where that goes.
Second stage - for each item, write the commands to run and the expected result. Start to capture exceptions and understand roll back.
Third stage - for each item, automatically run the commands and show the result, and ask for permission to continue to the next command. Continue to capture exceptions and formalise what to roll back. If there's an exception, point to the roll back instructions (which itself is automation and thus follows the same principal)
Fourth stage - Don't ask for permission to continue, but throw out an exception and point to manual roll back instructions
Fifth stage - automate the roll back too
At all times - copious logging for when there's unexpected exception in the rollback
Affectionate-Cat-975@reddit
Structured defined tasks. If you can build the rails to keep it on track with the business rules, then the AI/BPM is great. I had implemented on/off boarding process that went from HRIS through account and ap provisioning for a hospitality company which would annually churn about 5000 works across the seasonal flow.
boli99@reddit
DailonMarkMann@reddit
lol. Everyone senses the third rail.
TW-Twisti@reddit
What does that expression mean ?
patmorgan235@reddit
"third rail" refers to an electrified rail used to supply power to an electric locomotive, the "third rail" is usually something to avoid messing with/talking about.
Now what "third rail" the comment or is referring too, I have absolutely no idea.
TW-Twisti@reddit
Ah, cheers, thanks for the explanation.
Fearless-Feature-830@reddit
Certificates
Prestigious_Rub_9758@reddit
I'm more comfortable with tools that enforce guardrails instead of full autonomy. Whether that's Siit or a heavily locked-down ServiceNow workflow, limits matter.
benderunit9000@reddit
C-Suite
Awkward_Leah@reddit
Access provisioning works until someone wants an exception. That's where automation usually breaks down
WhiskyTequilaFinance@reddit
End to end without human oversight? None.
End to end with stage gates that prompt a human to review before proceeding? Lots of them, but also I'm the one who writes the automations and determines the stage gates. A good portion of what I've written is about automating the menial things I don't want to do, or do rarely enough I forget all the steps.
justaguyonthebus@reddit
All of it. But I enjoy creating automation, so I might be biased.
I'm only not comfortable when the validation isn't automated enough. But I usually start with the validation because 1) I don't trust humans to perfectly validate every time and 2) it helps me know the automation is doing the right thing as I build it.
rfc968@reddit
„Hello, IT“ … … … „Have you tried turning it off and on again?“ … … „Cheers“.
void_admin@reddit
*crickets*