anyone actually running AI automation autonomously in prod?

Posted by Major_Layer_5664@reddit | sysadmin | View on Reddit | 27 comments

anyone actually running AI automation autonomously in prod? I hope not tbh, without some sort of human over sight lol Laid off in November, sole tier 3 at a 5K-user shop before that. Built a lot of automation — SMTP alert parsing into Freshservice, HRIS to Entra sync, scripted offboarding. Zero LLMs in any of it. Since then I’ve been running my own stuff locally. Apps Script, Claude and OpenAI APIs, Workspace admin SDK. Useful, but it’s a sandbox. Can’t tell from the outside how much of the AI-in-IT pitch is real at scale. For people running it in prod: • Anything closing tickets, containing endpoints, or pushing config without a human approving? • Where has it actually saved time? • What did you turn off after trying it? • Charlotte AI, Security Copilot, Rapid7 AI Engine — past the demo phase anywhere? Working theory: it’s a multiplier for the humans, not the autonomous SOC vendors are selling.

Reply to Post

27 Comments

[-]

Kumorigoe@reddit

Sorry, it seems this comment or thread has violated a sub-reddit rule and has been removed by a moderator. **Do Not Conduct Marketing Operations Within This Community.** * It is not acceptable to advertise a product, service, Blog or FOSS Project within this community outside of authorized threads. * It is not acceptable to perform product research or market research within this community without permission. * The [Reddit advertising system](https://www.business.reddit.com/) exists to help you reach out to new or existing customers. * Product Representatives are free to discuss their product in the context of an existing, naturally-occurring discussion. Astroturfing is not permitted. * As always, users must disclose any affiliation with a product. * Content creators should refrain from directing this community to their own content. Your content may be better suited for our companion sub-reddit: /r/SysAdminBlogs ----- *If you wish to appeal this action please don't hesitate to [message the moderation team](https://www.reddit.com/message/compose?to=%2Fr%2Fsysadmin).*

[-]

thortgot@reddit

In the past couple of months Ive rolled out automation that handles ~30% of the workload of my juniors. Its not that hard.

[-]

lectos1977@reddit

So, 30% of zero since the juniors tend to do dick all?

[-]

buds4hugs@reddit

Am junior. A lot of my time is figuring things out. Slowly. Usually cross referencing documents & forum posts, then trying things. Copilot has been saving me some time scripting though, I hate to admit. (Yes I understand, verify, & test all suggestions, & challenge the AI)

[-]

Major_Layer_5664@reddit (OP)

Love to hear it! Everyone starts out as a junior and eventually/hopefully they get to move up. Are you using co-pilot that's built into your machine issued by your IT department? or using its as a shadow IT type of thing?

[-]

buds4hugs@reddit

Yep Copilot is built in & our only approved bot. I was reluctant until a coworker basically said "you're spending too much time searching, ask Copilot." Like, I know Powershell well enough to call out issues with what it suggests, but it definitely helps leading me to the tools for a solution.

[-]

Major_Layer_5664@reddit (OP)

does that co-pilot parse info from your internal org data or are you just asking it random questions? I assume you've got an internal knowledge base with articles on how to triage as well

[-]

buds4hugs@reddit

It doesn't parse from our SOPs or ticketing system. Not sure how much it dips into internal data. I've mainly used it for PowerShell & some reminders on networking & Hyper-V stuff

[-]

Major_Layer_5664@reddit (OP)

shitting on juniors is insane. Some are lazy, but some are also there learn and become better professionals. I had one guy on my team get certified while his team was going through our company reorg. He was promoted to senior due to him showing initiative. I've also had juniors who'd refuse to help themselves and were okay with just getting by. Just depends on the person.

[-]

BldGlch@reddit

Lots of tools have AI and correlation built in, but not making decisions. Aria ops(VMware vcf)uses it and 365 has an AI you can talk to to make changes in the environment but I don’t use it. Ai is evident in entra security as well, it’s just not so overt.

[-]

viral-architect@reddit

I'm not letting a non-deterministic tool directly impact production, but I definitely use it to produce production-ready tools. Some PowerShell scripts for example. I can do so much more now than I ever could before because I barely have to even articulate what I want and GitHub Copilot spits out excellent results. We do heavy change management so we make read-only tools to give us insights about our environment. Think daily healthchecks that do way more than what they used to because we don't have to sink so many hours into making the scripts as verbose as we really want them to be.

[-]

buds4hugs@reddit

I just left another comment about using copilot to save time as a Junior. It really helps guide me to answers & ways of doing things that would've taken me 2-4x the time to figure out the old fashion way. Disclaimer again: I understand, verify, & safely test anything & everything from it.

[-]

hihcadore@reddit

Running autonomously in prod is scary, there are some Claude horror stories out there. I think for reporting you’d be fine. No way it couldn’t crunch some logs or generated Md files. I have a local llm that gauges function edit risk, maps the code base, checks for orphaned code, tests, and can answer a prompt and prepare a report to cut and paste into Claude the next morning

[-]

Major_Layer_5664@reddit (OP)

Yep, I'm running some stuff locally as well on my home setup. It's helped me parse, organization, and put together some great documentation. My next role I'm really trying to get good at this stuff at scale in an enterprise environment. Local LLM's are cool for personal use; It's using them for large scale sensitive tasks that get a little cooky.

[-]

hihcadore@reddit

Well a big benefit to local is data sovereignty and the ability to test locally hosted test beds. No clue if anyone’s doing that.

[-]

Major_Layer_5664@reddit (OP)

i'd pray they are before deployment.

[-]

hihcadore@reddit

I think that’s where AI would really shine to be honest. Deterministic testing and AI that analyzes the results.

[-]

newworldlife@reddit

The automation part usually works fine. It gets sketchy later when people stop double-checking what the system is actually doing.

[-]

Major_Layer_5664@reddit (OP)

exactly, the management and implementation is dam near a contract role atp. The model drift is actually a hazard and it needs to be monitored extensively. I've seen some crazy stuff. I'd never let anything run without constant oversight.

[-]

newworldlife@reddit

Yeah, and the scary part is people slowly start trusting the automation more than the telemetry around it.

[-]

Major_Layer_5664@reddit (OP)

That's why I love AI so much when it comes to producing scripts. Especially if you're using it to manage users or parse data. Once you get good enough at explaining your use case to the model you're in good shape. I'm currently running a dev Okta instance, a Google IAM + Workspace instance, and a AWS instance and i'm able to explain to automate a ton of stuff by just explaining it to Cluade. The only issue is the model drift lol shits scary.

[-]

MrHaxx1@reddit

We use a bot for our Jira tickets. * Routes ticket to correct queue if incorrect * If there's insufficient information in the ticket, it leaves a comment, asking the user to provide more details (including what information would be relevant) * Suggests steps for the service desk person, usually based on internal documentation, including who to reach out to Works really really, as far as I can tell.

[-]

atheenaaar@reddit

Would links to documentation not work better than suggesting steps? I’d be too concerned over interpretation issues and accidentally offering destructive advice.

[-]

Major_Layer_5664@reddit (OP)

yeah, links to KB articles or a internal board that allow the user to self service themselves is key.

[-]

Major_Layer_5664@reddit (OP)

been hearing good things about Atlassian and their automation/ai tbh. You're the second person who've mentioned them. My last role we used another ticketing system and were able to automate certain triage but kept breaking is what my engineers would say when it stopped working properly.

[-]

k6kaysix@reddit

AI is pretty useful at converting a conversational language question into a pretty decent Powershell script in a few seconds rather than me having to spend hours working it out myself Just today I asked it 'how do I create a script to automatically move files from X to Y but only if they match a very specific file naming convention criteria' and I had the script and scheduled task up and running in no time I'm not stupid though and will test the life out of anything it suggests before making it live...it can be amusing when you specifically mention 'create me a Powershell script...' then it comes back with something, you test it and error...go back to ask it why it errored and it responds with 'ah yes those commands don't work in Powershell'!

[-]

AmoebaDue6638@reddit

Honest answer: the biggest time saver has been using LLMs to triage and enrich alerts before they hit a human. Not closing tickets autonomously, just categorizing and drafting responses that a tech reviews before sending. The fully autonomous stuff from vendors is still mostly vaporware at enterprise scale. We route a lot of our API calls through Orthogonal which lets us swap between different enrichment and scraping providers without rewriting integrations every time a vendor changes their pricing or breaks something.