Just audited our environment and the scariest stuff is the IT scripts that have been running for years without being touched
Posted by kmonie360@reddit | sysadmin | View on Reddit | 86 comments
Found hardcoded credentials in 3 separate scripts running in production. PowerShell touching AD, a Python cron job pulling from the production database, Bash written by someone who left ages ago that at least several devs have since modified without a PR or any kind of review.
None of it is in any system our AppSec process touches because these scripts live outside the repos we scan. I don't think anyone explicitly decided they were out of scope, they just accumulated outside the boundary of what the security tooling covers and sat there.
What bothers me is these scripts usually have broader access than the application code because the people who wrote them had admin rights and were just trying to get something done fast. That access was never revisited.
OneSeaworthiness7768@reddit
Just another nothing fluff post from a bought account that’ll be used to market something. Obligatory digital marketing post history
carbonfog@reddit
Thank you! Not sure how others aren't immediately clocking this as AI-gen? Or maybe they are and don't care?
Top_Bookkeeper9136@reddit
Ugh, the frustrating thing about these posts is there is a lot of useful, relatable information...
MonoDede@reddit
This should be pinned.
LokeCanada@reddit
I turned off the account for a developer once as he had left the company and crashed a few production systems. Same thing, developer with elevated access hard coded his account everywhere.
There is a reason why dev teams should have their stuff independently reviewed.
rose_gold_glitter@reddit
I use at least 2 APIs that insist I use my own account for the API key. It's terrible design but I don't control them. I know when I leave things that rely on those APIs - like our actual, physical door controls - will break.
We cannot use a 'service' account with these, 3rd party, systems - we have to use a real user and I don't have control over making accounts, there.
fresh-dork@reddit
can it be a fake user that you then delegate to whomever has that job? seems like a way to deal with things sensibly
Unable-Entrance3110@reddit
My question is, what is the difference between a regular user and a service account user?
Just set up a "real" user account and never issue a badge correlated to that ID and/or don't set up any permissions other than what is needed for the API? IDK, just spitballing here.
We have a "real" (actually virtual) user in our IT team that shows up in all of our documentation and is used to test things as a user.
Uncommented-Code@reddit
God yep, similar issue I am in right now.
I would really love to set up things the proper way but a lack of resources for a long time led to an environement that lacks the proper tools, e.g., a proper secret manager.
I leave in a couple months and am slowly going through all my stuff an pulling people aside to ensure nothing is dependent on my credentials, that they at least know about these systems and scripts, and that documentation is in place and where to find it when something inevitably break anyways.
gumbrilla@reddit
Yeah, my favourite is github, when a senior dev leaves their keys are auto-disable once we remove them from the github organisation.. queue many build processes failing. Doesn't hurt prod, so they just get a shit eating grin from me, and a suggestion they use the service account we maintain.. twice it's happened and brings me no end of joy
kayjaykay87@reddit
As a dev honestly the issue is sometimes infra are so intolerable and slow to deal with you've barely got any choice. Want a new blob storage account to move a database from prod to dev? Better join a meeting with our security team to discuss the implications. Can you ask me the questions by e-mail? Nope, need to have a meeting. Okay, can we do this time? No, they're not available. Okay .. Nevermind, I'll write a script to do a bulk import from prod into dev....
sirdmz@reddit
… meanwhile you copy data across to a dev environment that doesn’t have the security clearance for the production data you copied. another dev thinking it’s just test data copies it to something locally, or on a virtual machine somewhere. somewhere this gets caught, and the company is facing multi-million dollar lawsuits.
There’s a reason that was a meeting, and not a casual email chain. It’s so the data could be defined, verified by someone who understands the regulations, and correctly documented to avoid issues like this.
There are very clear reasons I do not have access to prod, because I am also a cowboy who will do dumb shit.
fresh-dork@reddit
this is much easier to do in email. lay out what specifically you intend to export, admins validate that this is okay, provide access path. all in searchable text and docs
elatllat@reddit
Avoiding email is avoiding accountability.
kayjaykay87@reddit
No, it's a dev database on the same server as the prod database with the same sec-- you know what nevermind. You're infra, of course this needs a meeting with a team of 10 people to do something I can do anyway, and of course the questions can't be asked by e-mail (there could be multi billion dollar lawsuits).
Centimane@reddit
The irony of being exactly the type of dev this post is making fun of...
fresh-dork@reddit
or the type of process that makes this happen in the first palce
TerrificVixen5693@reddit
Security’s job is literally to slow you down so you don’t do shit like this. What’s the rush bro? You like working?
Gullible-Surround486@reddit
and then it turns into script archaeology later, which is always great.
ConsciousIron7371@reddit
…..your only complaint is having to join a meeting? That was the best example you could come up with? Scheduling a meeting?
You know outlook will show you other peoples availability?
literahcola@reddit
My least favorite thing about the job is the endless need to have 30 minute meetings to discuss things.
"This meeting could have been an email" is SO true.
Hauntingblanketban@reddit
seems more of a process issue rather than tech issue as you copying or anything requires approval from PdM / Owner/client..
as long as it is approved, product support/SRE/secops team needs to to designated stuff
on our side it requires only mail ,approval and its implication known to Pdm/owner..no meeting is required
gumbrilla@reddit
Yeah, so DevOps does dev ops, and Security/Operations are super blurred, where I'm from. Good fortune of working for a smaller org. As a developer in a past life, it helps also.
I also hate meetings, hit me up on teams, if it needs a ticket for audit I can just create one with a screen shot of our discussion. Job done.
Shit eating grin is my default though. You always get that.
pdp10@reddit
Why wait until a senior leaves and blame flows freely for a time? Chaos monkey disables devs on Tuesday.
When the dev complains, mumble that something must've set off the security alerts, while you re-enable. Then watch audit logs to see the first thing the dev goes for when they're re-enabled. Bingo.
kmonie360@reddit (OP)
Hardcoded personal credentials are the worst version of this because the blast radius is invisible until something breaks. At least service accounts show up in a directory somewhere.
UserProv_Minotaur@reddit
Same thing once happened with my company’s Service Now integration when we axed a developer.
EquivalentBear6857@reddit
Every org has these, ones you found are the ones you know about.
kmonie360@reddit (OP)
This's very unsettling. Which means ones outside that boundary are still sitting there.
billdietrich1@reddit
Stupid idea: suspend creds for some people for a weekend, see what breaks ?
TheGamingGallifreyan@reddit
There’s some companies where this is actually required. Mostly in finance. They will require employees to take a one week vacation every year or so and their account gets completely locked And everything they were doing gets done by somebody else.
Apparently a lot of money theft has been found this way.
sobrique@reddit
Yeah. A place I used to work (not financial) caught a guy who worked in HR because he took too much leave.
He'd created 'extra' jobs and collected the pay for them. And because he worked in HR could field the queries about 'so who is this guy on my payroll?' when they came in.
"short" holidays were fine - the odds of a query showing up and someone dealing with it before he got back were low.
But then he went away for 4 weeks, and the whole thing unravelled. He'd been getting away with it for quite a long time too.
pdp10@reddit
This guy didn't watch Superman III. The trick is to create billions of fake employees, then steal just a fraction of a cent with each one.
TaxHazyShade@reddit
or The Office. Except for "no-talent, ass clown" Michael Bolton goofed up the code.
ConsciousIron7371@reddit
There are tools that discover processes and scheduled tasks and what identity they run under. From there you can at least document what is running and the possibility of auditing the code they are running becomes possible
OpenGrainAxehandle@reddit
Which is probably exactly what OP is selling.
Cormacolinde@reddit
That’s where good IAM processes are important.
You should do access reviews, audit authentication logs and try to find more unusual login patterns. Just looking at username;ip doublets might help a lot.
And this kind of stuff is why I absolutely hate when someone tells me they’ll use their account for something “just for now” and “change it later”.
I’m always going to”no I’ll configure this properly now even if it takes me 30min. Especially as a consultant I don’t want them to get issues in two years when that admin leaves, and I definitely don’t want to deliver documentation with this kind of stuff in it.
pdp10@reddit
There are two ways to initially get something up and running:
TrustMeImAnOnion@reddit
The Known Unknowns
qftvfu@reddit
In a previous job, the problem of "script management" was a nightmare to solve. Unversioned, no repo, no idea of how critical they are, how many, where, who uses them. Bash, python, perl, etc.
The best idea I could come up with at the time was the idea that all scripts need to use some sort of common library that phones home with some telemetry so you can at least track existence. Then later on go around and gradually increase controls around them.
The complexity of the problem and the effort involved meant it never went anywhere. Hundreds of other risks on the risk register to manage before that ever became a priority...
I'd be interested if there's any improved solution, beyond whitelisting, to manage legacy scripts across enterprise?
PCLOAD_LETTER@reddit
Hardcoded user credentials or hardcoded service account credentials?
Foreign_Impress6535@reddit
Yeah, I've spent quite a lot of time updating "critical" tools to only have the access they need, instead of the Domain Admin that was originally granted. Same for user accounts. Why is everyone in Account Operators? They're not anymore!
Strong-Suggestion-50@reddit
There's a large insurance company in the UK. I won't name names.
About 5 years ago I was brought in to look at their digital platform and make recommendations for improvement. It turns out that their online quote process makes a call into a legacy backend. The code in thet backend was first put live in 1969 and has been migrated multiple times as they moved backend platforms.
Every single time you do an online quote for that insurance company, you are running code that was first put into production before we walked on the moon.
I ran away
pdp10@reddit
Old code isn't bad code. Old code that's actually a screen-scraper from a vendor that's been out of business for 15 years, calling into a legacy IBM system that can't use any form of modern authentication, to access old COBOL code that nobody on staff has ever seen or understood, is bad code.
Vegetable-Ad-1817@reddit
Found a script from the early 2000s once that synced two ADs using ldap calls, impressive it still worked for 23 years with domain upgrades, but it did.
pdp10@reddit
I wrote some scripts in 2000 to query MSAD from the Unix/Linux command line, using LDAP. They worked unchanged until, if I remember correctly, authenticated binds started being required.
By the time that happened, M&A meant that the new parent organization no longer had any MSAD.
elatllat@reddit
I love how Google will merge a personal account with a service account without warning and refuse to unmerge it, if they are going to require gcp migration it's a migration to AWS.
thirsty_zymurgist@reddit
We just audited the same and pulled a bunch of scripts off on-prem devices. Moved them all to azure with managed identities and version control. Wasn't as fun as peeling oranges but it wasn't too bad.
pdp10@reddit
So how does it get deployed? Work backwards from there. Then don't let devs hand-tune production machines to get them to work, or you end up with undocumented glue.
Our styleguide requires certain boilerplate at the top of a script, in
READMEs, in the--version-jsonof binaries, that specifies the canonical location/repo. Best idea I had that morning.When I find one that doesn't live in a repo (scan by hash, scan by name, etc.) then I ask about it in email, put a change-control header on it asking why it's not in a repo yet, and put the path under monitoring/auditing. If you're changing the script anyway, you can also add sneaky self-auditing in the form of
loggerstatements, etc.nemor3@reddit
The part that's hard to fix is that these scripts are invisible to your incident response until they break. Application goes down, you're checking dashboards, alerts, logs - but the cron job that silently stopped authenticating two weeks ago doesn't show up anywhere until someone notices the data isn't there.
At least hardcoded creds you can grep for. The access that was never scoped and never reviewed is the part that's actually hard to find.
OneSeaworthiness7768@reddit
So many fake accounts on the post. Scuttling out like cockroaches.
nemor3@reddit
Exactly. Unscoped access is basically invisible until you go looking for it deliberately, and most teams only go looking after something already broke.
OneSeaworthiness7768@reddit
lol wow, not even trying with the bot replies.
Swimming_Office_1803@reddit
Good for you. Now sell us your audit tool
Glad-Watercress4677@reddit
The hardcoded credentials are the symptom. The fix is secret management that makes hardcoding impossible. HashiCorp Vault, AWS Secrets Manager, even Azure Key Vault all support the PowerShell and Python patterns.
Once secrets are externalized, rotation is automated and the hardcoded credential problem stops reoccurring every time someone writes a new quick script.
OneSeaworthiness7768@reddit
Thanks, ChatGPT
T_Thriller_T@reddit
Yep.
I am in security, I was a dev.
It is very freaking hard to do automation with credentials, not hardcoding them, but also securing how things are called so the password is not in the history, or simply another file and and and.
Even without automation a clear process how to safe secrets and how to safely load them into scripts, envs etc makes probably a big difference because people do not want to make insecure decisions, they simply lack the knowledge and time to build secure alternatives but are pressured to get things done.
Cheomesh@reddit
Yeah you're supposed to do a design review and set up recurring tasks to check these things - ie document for quarterly review or whatever your org calls for...
JuniorCombination774@reddit
There are two areas of focus here, giving admin rights to your devs is the base layer. You could implement giving them on-demand admin access (EPM tools do this) and allowlist apps they normally use. This would be request-based or you can define a policy that is created based on their behaviour.
The second - hardcoded credentials ; If you use a password manager - you can enforce users to utilize APIs to fetch all credentials from the password manager. As for finding hardcoded creds and scripts thats already written, just pray they dont fall in the wrong hands xD
T_Thriller_T@reddit
There are options for finding them, though.
Tons of secret scanners - the ones attackers use, sometimes. Because those were not developed for attacker's, but for the folks securing.
Just means logging onto machines where scripts live and letting them run, which takes time.
But it's very much doable.
MeetJoan@reddit
The hardcoded credentials in scripts with admin access that nobody scans is genuinely the most common unpatched attack surface in production environments - get those into a repo and run Trufflehog against them today before anything else.
How are you handling secrets rotation once you find them - do you have a secrets manager in place already, or is that part of the problem too?
Ok_Score_9685@reddit
1st day in IT?
Tech-Cypher@reddit
First move before anything else: revoke the admin rights on the service accounts those scripts run as and replace with scoped permissions. Credential rotation and secret management can come after. Least privilege on the execution context limits blast radius while you figure out the rest.
kmonie360@reddit (OP)
Challenge is some of these scripts will break when you do that and the person who knew what they needed is long gone.
jimicus@reddit
Your options are to schedule maintenance to square this away when it suits you. Or have the universe schedule maintenance when it pleases itself.
Your call.
Borgquite@reddit
But the disruption caused if you are compromised will be far greater than the disruption caused by doing the above - and you get to choose when it happens.
DiabolicalDong@reddit
You can make use of password managers and privileged account managers to fetch and use credentials in real time during script execution. Hardcoding is extremely risky and one must avoid it at all costs.
If you are looking to go down this path, look for solutions that has API support and CI/CD pipeline integration.
TheSixFelix@reddit
the hardcoded creds in prod scripts thing keeps me up at night too. what gets me is the access sprawl never gets cleaned up because nobody owns it, it just lives in some corner of the infrastructure nobody touches. you find three of these and yeah there's probably a dozen more hiding in cron jobs and scheduled tasks nobody even remembers exist. moving these into actual repos with proper secret management and review cycles should be table stakes but somehow it never happens until something breaks.
Frosty-Fortune4118@reddit
Service accounts that have been running production scripts for years without an access review are almost always over-privileged. Schedule a quarterly review of service account permissions the same way you review user access.
rickAUS@reddit
Stopped counting how many "sevice accounts" i have found over the years which were DA. Honestly scary stuff how lazy people can be with permissions.
hellcat_uk@reddit
My big take is you had a PowerShell script running for years without Microsoft deprecating the functions. Congrats!
Although nothing to say the script was running successfully.
VintageSin@reddit
It was probably just using direct execution rather than something in the . net framework.
Cormacolinde@reddit
For PowerShell, enforce script signing, don’t use timestamping and require resigning every year. That’ll force some awareness of what is going on.
kayjaykay87@reddit
Nothing better to do than resign a bunch of scripts every year and get notified every time a script needs updating eh?
Proud-University593@reddit
Did this same audit two years ago, bash script situation is universal.
What I found was scripts that had been copied from the original author and slightly modified by multiple people over time, so there were four versions of the same script with slightly different credentials hardcoded in each one. Git blame was useless because the scripts were never in git. Audit wasn't even the hard part but tracking down every place those credentials were used across the environment was the nightmare.
thewrinklyninja@reddit
First day in IT?
Special-Cause7458@reddit
The out-of-scope boundary problem is fixable with one policy change: any script with production access gets scanned regardless of where it lives.
Checkmarx SAST runs against Python, PowerShell and Bash and catches hardcoded credentials and insecure patterns in exactly the kind of operational scripts OP describes. The scan doesn't care if the script lives in a repo or a shared drive. Just point it at the right paths.
graph_worlok@reddit
4755 for the win!
M4niac81@reddit
Started at a place once in a senior role, one of the first things I did was change the password on the domain administrator account and it broke so many things it was ridiculous. Almost every single script, scheduled task or other automation was running under that account, took months to sort it all out.
TheRealJachra@reddit
You can copy those scripts and use AI to help you document what each script does. Just remove the password from your copy before adding it to AI.
When you have documentation on it, then you can form a plan on how to change those situations.
Secondly, I would recommend on running at least once a complete discover&audit scan on your environment. It could give you a report on what is in your environment and where to focus on.
Minute-Confusion-249@reddit
Why were admin rights handed out liberally enough that scripts could accumulate with production database access and no governance process was built around it?
kmonie360@reddit (OP)
They were never categorized as something that needed one. Someone with admin rights needed to automate something, automated it, and it stayed. now access outlived the intention.
Jeriath27@reddit
Only 3? I've been updating ours. Over 50 scripts just for exchange. Longest one is 10,000 lines :)
justaguyonthebus@reddit
Ah, so also not rotating credentials
kmonie360@reddit (OP)
Wasn't skipped deliberately, it was never in scope. some of these predate any secret management tooling we have.
shun_tak@reddit
Is it broken?? -> Don't touch it...
SASardonic@reddit
If you only have three I envy the clarity of your environment. All I can say is thank Christ for our IPaaS