Genuinely hate cyber security teams
Posted by talent_de_tigan@reddit | sysadmin | View on Reddit | 229 comments
After working as a platform engineer for almost half a decade, one thing I developed is a strong hatred for cyber sec teams. I'm not sure if it's just me, but in every place I work they are seen by the business as the guardians of the profit realms while in reality they do fvck all.
Most of the security work is done by us, platform engineers/ Sys Admins. You are expected to build with security at the forefront. You have to think of security on so many levels. You are the guy who manages certs, dns, networking, IAM, firewalls, reverse proxies, load balancing, gateways, while also ensuring your app is not leaking memory, does not have unintended ports open, is hosted on the right platform, you're not exposing creds on VCS, your .env is secure and only the right users have access to it, all while understanding the business logic and making sure the hosted app doesn't get ddosed/ hacked. Also when an incident happens you are generally the one on call, so even under attack we are the ones expected to defend against it.
I genuinely imagine a day in a cyber sec life is them itching their arse, digging for gold in their nose then clicking 'export to pdf' on an automatic SAST scan and then charging you 10k for it. Cyber teams in my experience have honestly just been employing 'block everything by default', then you have to profile your app, use procmon just to find out your app was blocked by some firewall from writing out to logs.
They don't work with you to build something up, instead they just throw a bunch of CVEs at you and expect you to fix them, all while charging you an arm and a leg. If they were to be more integrated in the team rather than being in their own little separate enclosure and sitting on Forbes all day drooling over the latest node js supply chain attack, then maybe, MAYBE things would be more smooth for us.
I think of cyber security the same way as I think of the San Andreas ambulance. On the way to save some granny it ran over 10 people. The amount of extra work they create for us is just crazy.
moffetts9001@reddit
I am sure they have specialized skills, but I am also very unimpressed when they hand me a R7 export (that is riddled with false positives) and wag their finger at me.
Deathra9@reddit
The only good cybersecurity expert I worked with was a former sys admin, and a good one at that. The rest only know compliance checklists and don’t understand risk beyond compliance. It’s worse when they aren’t consultants but part of the org. Then you REALLY can’t get anything done.
caribbeanjon@reddit
I have a 20 year career in infrastructure, and recently transitioned into Information Security and I feel where you are coming from, but here’s the other side. Sysadmins that leave everything open, http by default, or best case self-signed https certs. The same password for everything or passwords stored in an excel file. Someone has to look at the mess, and say “No, don’t do that!” and “Do this instead.” InfoSec is more about coordinating and focusing resources to address risks.
1stUserEver@reddit
We are at a juncture where we need a new role. Security admin possibly. Someone that has access to make changes while consulting the sysadmins. There is too much work and no resources to deal with this madness.
burlyginger@reddit
I've worked in a couple places with DevSecOps people and..... There was no dev or ops capabilities. Just Sec.
Sec people are like network peoole, if you find one who can code you've found a diamond in the rough.
nkwell@reddit
When I look for people who would make great pentesters, network people who can code are who I look for.
I hate that AI has suddenly made everyone a "coder".
That's why in interviews, we show them bad code and ask them "what's wrong with this code?"
And if they do manage to figure it out, we ask them "OK, how would you fix it?"
Tends to separate the wheat from the chaff.
burlyginger@reddit
Mos deff.
One of my favourite moments when interviewing for a principal network engineer.
Me: "Managing hundreds of client VPN tunnels is a lot of work. How are you managing changes with that much volume?"
Candidate: "With manual changes."
Me: "It was nice speaking with you...."
NisargJhatakia@reddit
I had to google the answer to this question and the best thought I could think of was transit gateway on aws. I didnt had any other better idea if I were to be asked in an interview.
burlyginger@reddit
Literally any automation would have been acceptable.
smuziq@reddit
That's called an Information System Security Engineer
nbs-of-74@reddit
Security engineering.
Problem is, we're bumping into the Ops teams (ie infrastructure) who think they should be doing all the engineering work instead and CS leadership wanting you to do policy work only. So engineering orientated security people either leave or move back into infrastructure.
ConsciousIron7371@reddit
Coming from a security perspective, no thank you. I do not want to get back into the business of being responsible for systems that support users. Do not wake me up in the middle of the night.
I value greatly the separation between systems support and securing information. I am very happy to discuss changes and risks and how to address them and generally believe whatever timeline the sysadmins come up with.
FearlessAwareness469@reddit
My boss made me that. I was application engineer that had really good security ideas. So now my title is application/security engineer. Basically the implementation arm of infosec. They have good ideas but I get to explain why some thing are not feasible
cmack@reddit
That literally started in 2009 - Security Liaison
Original-Locksmith58@reddit
Except most places I’ve worked (US) the people doing the compliance are also under infrastructure, not security. Security is usually just SOC kids staring at blinking lights or forwarding SIEM alerts to infrastructure well after they’ve already fixed the problem. I also recently transitioned to the Info Sec side of the house as an ISSO, and there’s definitely a lot of work there, but the non-policy/advisory people in my group are useless and they’re like… 80% of the team.
Fallingdamage@reddit
Its bad sysadmins and lazy IT professionals that make cybersecurity consultants so important.
stone500@reddit
Yo where can I get some of those cybersecurity analysts that do the second part? Cause in my org they typically just point out whatever risk vectors Copilot and Bleepingcomputer.com is telling them, but offering virtually no suggestion as to how to mitigate those risks.
telvox@reddit
Have watched the growth, the biggest problem is you are getting out numbered buy people that have only been security. 10 years ago the sec team was server admins and dev ops who transferred to sec. Now its intern who used chat gpt to graduate. We had one who didn't know ping was a computer term not just slang for reaching out to someone. That shift is major and causing this issue.
AGsec@reddit
It's why I unsubbed r/cybersecurity. Way too many people who have masters, CISSP, and a bunch of certs who won't even consider non-security jobs, despite having zero experience. One guy laughed at me when I suggested he look into systems admin/engineering, he said "i didn't study this hard to go into IT" and I got downvoted when I told him cyber security was a subfield of IT.
caribbeanjon@reddit
As someone closer to the end of my career than the beginning I will say there are still a few “diamonds in the rough”. Specifically on my (relatively) small 15 person security team we have 3 guys in the operations team with solid technical chops, but more importantly the curiosity to continue building that knowledge. I hear where you are coming from, but I think you can find talent if you are careful during hiring.
RikiWardOG@reddit
also have to pay for said talent. People want a 1 in a million candidate but pay helpdesk wages for it
vCentered@reddit
The real crisis is the number of frauds in information technology in general.
I hear from my security people every week about vulnerabilities that are waiting on a patch from the vendor. Every week it's like they think they've uncovered something nasty I'm supposed to have fixed. It's the same vuln. I have to explain again. They don't get it. They don't understand.
I also have to explain to my sysadmins that they can't leave "Everyone" or "Domain Users" on filesystem permissions. Or that the firewall rule they created basically amounts to any/any and isn't acceptable.
80% of people don't get it, they don't care, and there's no incentive for them to change because the 20% of us that do are propping the industry up.
nkwell@reddit
30 years in IT, 15 of them in Security. This is 100% reality. Lots of people got into it for the paycheck. The ones who are passionate and do this shit for the love of the game are the folks keeping the trains on time.
RikiWardOG@reddit
and all of the passionate ones are burned out lol
jay-dot-dot@reddit
Im getting there at year 14. Cybersecurity is drowning in frauds andnim tired of dealing with them.
many_dongs@reddit
Also probably being underpaid or having their accomplishments stolen by their ladder climbing ignorant boss
ThoriumOverlord@reddit
100% truth and it's pretty disgusting. I've watched new employees come and go starting from the recruiters saying a tone-deaf "I think they'll make a nice fit for you." all the way to their exit interviews saying "Well, I was thrown to the wolves. The shift lead just told me to shoulder surf after he showed me around the building and that was it." In between those was a metric ton of "Hey man, how do I do this?" without even thinking of Googling or putting in the effort to actually learn something.
I guess that's to be expected when they can't be bothered to stay their asses off YouTube for entire shifts and heir leads can't be bothered to tell them to get to work. /shrug
graph_worlok@reddit
You should see the amount of CVSS 7+ vulnerabilities in a Debian install when you do the vulnerability analysis based on installed packages rather than external / blackbox. And I mean - they literally have no patch from Debian, or plans for one. Uuugh. And I get it. But fuck my metrics….
JoelyMalookey@reddit
What sys admins are you working with? This sounds like consultants more than internal
sroop1@reddit
You haven't worked in a smb manufacturing company I see.
ObiLAN-@reddit
I was hired a few years ago as a sysadmin in the same type of enviroment, after they had been ransomwared.
Im convinced their previous admin might have been smoking crack or somthing. What a mess.
-GenlyAI-@reddit
Lol you're joking. Lazy sysadmins is a full meme. On top of that they are understaffed. When I was a sysadmin there was no way I was working a full day and then reading up on security changes, patches, zero days, etc to implement.
Most places with in house sysadmins and no security team have tons of security flaws. This is why SOCs, governance teams, red/blue team exercises, tabletops, audits, scans, reports. Are so important.
JoelyMalookey@reddit
Private companies can be wild but as soon as you have contracts with any public, financial, or govt company you can’t sustain non security minded admins
-GenlyAI-@reddit
That's true, which is how your org ends up with an MSP for 6 figures a month, compliance controls so tight people can barely work, and shit tier service desk support.
caribbeanjon@reddit
My organization grew quickly due to many acquisitions. There was some outsourcing, but the biggest problem with is getting 5 sysadmins across the globe from 4 different acquired companies to work together at an enterprise standard.
My_Legz@reddit
Oh brother, I wish. The amount of times I have seen that from internal IT is maddening
StateOfAmerica@reddit
In smaller orgs where internal IT may be a one man show and his minions that falls upon one person or a team led by idiots.
Been in many places where quick fixes were left forever. The best one I've seen was WAN -> ANY = ALLOW on the China site because they couldn't get something working.
I'd say MANY smaller org IT departments are completely clueless.
For my current gig I know we're at the very top in security and we're a two man show while some others are ran by either MSPs or their own internal IT. Bottom half don't even have MFA. Which is why the umbrella covering this line of work monthly sends out IT 101 papers of "this is something everyon should have".
We've been called in by these sister companies to help at times and in one place their head of IT hardly knew how to restart a computer.
Soggy-Attempt@reddit
Leave everything open? 🤦♂️
robocop_py@reddit
Me: Don’t expose RDP to the internet okay?
Sysadmin: Then how are we supposed to admin our domain controller remotely?
Me: Through a VPN
Sysadmin: You mean I have to create a whole VPN setup just to admin my stuff from home? Do you know how much work that is?
Me: Sorry 🤷♂️
(Two months later)
Me: Hey, our SIEM is showing RDP login success to our main business SQL server from a Latvian IP. What is up with that?
Sysadmin: WHAT??? Why didn’t you block that?!
Soggy-Attempt@reddit
Thats lacking basic policy and following basic security standards. Not what the OP is talking about.
ersentenza@reddit
And in fact this is why policy exist. Because admins can't be trusted to do the right thing.
Soggy-Attempt@reddit
Then set policy, hold people accountable, and get out of the way so the good people can do their jobs. 🤷♂️
-GenlyAI-@reddit
That's what policy and governance is for. Shitty sysadmins who just want their work to be easier. Then you have to monitor. And make changes when the landscape changes. Then the crybabies will throw a fit again.
knightofargh@reddit
I enjoy the number of LLM prompts with plain text PII, PANs and secrets I see at a bank from devs who don’t believe security is their job. “But it’s just test data” just means you are going to do exactly that in production and you know it. Having one of the DLP people try to get me to turn off prompt monitoring on certain items because it’s a “secure endpoint” was obnoxious. I’m also a customer of my employer, I don’t want my PII being fed into some LLM. I certainly don’t want GDPR subject data being fed into one.
The non-technical risk and security people are the worst though. They don’t even have the basic skillset to know what they are asking, just that the spreadsheet has a red cell.
greensparten@reddit
Preach!
Dreilala@reddit
See, that's the difference between you and the type of security OP complains about.
You have a "do this instead" policy rather than just blocking everything.
wowsomuchempty@reddit
MAKE IT GREEN MAKE IT GREEN MAKE IT GREEN MAKE IT GREEN
Fallingdamage@reddit
You're not wrong.
Though at the end of the day, A sysadmin's application of cybersecurity practices is risk-based. You dont have to plug every hole, but if you dont you need to be able to say why and acknowledge that its there or what other safeguards are in place to accommodate that risk your business has decided to accept.
At the strictly business side of things, its about liability and cost. Its about insurance and the ability to operate. Its bullshit a lot of the time but you HAVE to have the review done anyway.
Ive been through multiple security risk assessments with our insurance company and been the recipient of several pentests. The 10k you spend is a lot cheaper than the amount you pay in the event of a breach or the increase in insurance premiums for your business.
As a Sysadmin, think of it as a peer review of your security posture. If you do your work well, always build with security in mind first and when that calendar invite shows up in your inbox, you wont be stressed. Its all documented and in place right?
I think cybersecurity companies are way overpriced, but what can we do?
OffensivePanda69@reddit
Almost half a decade, huh?
In all seriousness, it seems you have an issue with your org. As a member of those gosh darn cyber sec teams, I can assure you we own the shit we own. Our App teams have not a clue about what ports they have open until we tell them.
Anywho, calm down. Chill. It ain’t that serious.
kombatunit@reddit
Running Nessus is "hard."
JohnnyricoMC@reddit
There are a lot of frauds in cybersecurity who can only run a scantool and generate a report and point to the colored sections in the reports without considering any nuances. I've had to explain concepts like backporting to such people when they came up with a report that just looked at major version numbers.
There are also a lot of frauds in system administration who don't use proper filesystem permissions, allow weak password policies which they never rotate, don't regularly update their systems, don't consider man-in-the-middle vulnerabilities, allow password authentication for root and think they can compensate bad practices with some firewall rules.
To be decent in either field, you must also have some proper awareness of the other field.
ElvinLundCondor@reddit
I see you’re running version x.y.z of Apache. That’s vulnerable to this CVE. You must upgrade. But Redhat backported the fix to that CVE in package -w. Don’t care. The report is red. Ok, remove version number from the headers. The report is green now. Thank you.
junpei@reddit
I just dealt with this with our security team. Reading your comment raised my blood pressure. Managers had to get involved.
Reetpeteet@reddit
That's not a security person, that's some manager who only looks at dashboards.
many_dongs@reddit
By this definition, I don’t think it’s required to be an actual security person to lead a security team
Source: 12 years of experience in infosec
mrsocal12@reddit
"But Tenable says..." 😂
Johnny-Virgil@reddit
Where I worked they’d let you say it was back ported, but then we’d need vendor validation and have to do 20 minutes of paperwork to request a variance that has to be signed off by the CIO, given an end date, tracked, and renewed every X months. I felt like I spent my days filling out security forms.
inucune@reddit
Have a similar problem. Why are the security team not filling out these forms?
graph_worlok@reddit
As per the Debian / Redhat / Ubuntu / etc CVE data for that vulnerability here, it shows that the running version has actually been patched against this vulnerability. show banner they didn’t include show CVE fix data
I’d be ashamed of myself if I bothered a dev or admin with something without doing this first.
Infosec / Servers & Networking for 30 plus years, and as tedious and time consuming as it can be, still enjoy proving the tools are busted.
Because nobody knows quite how busted they are until they drive them each day.
CaptainZhon@reddit
As long as it scares management into thinking- thry don’t give a fvck.
codewario@reddit
In our org if this happens our security team takes the finding back to the vendor and asks why it's flagging if we have a patched version.
I do mostly agree with the sentiment that they tend to push work off onto other teams. And in the case of my org, they make their tools available for download in the most convoluted way possible: I have to REST API into an endpoint to download the payload as a base64-encoded string, convert it back to whatever executable format we expect it to be in, and then execute it.
"Can you guys just put your tools on the fileshare? Or can you upload them to Artifactory?"
"No we have them available at CONVOLUTED_REST_ENDPOINT"
Whatever this method they make their tools available through predates generative AI, so somebody at some point thought this was a good way to make downloads available 😑
Starkoman@reddit
Or unavailable.
codewario@reddit
Right?
It's also very difficult to deserialize over WinRM/PSRemoting without running into OOM issues with some of the larger payloads as well, which causes issues for some of our remote installation scenarios we use for testing and specific environments.
Not a single consumer of these packages likes how they are serving them up but they won't budge.
digitalsleet@reddit
If your cybersec guys are using Rapid7 for vuln scanning, they have a few vulnerability categories for RHEL that are excellent for dealing with this. They are titled "Vulnerabilities deemed not relevant on Red Hat Enterprise Linux [8/9/10]" with 5k-20k CVEs in each category. If they search for the CVE number in the global search, and look at the vulnerability section of the results, that category will be listed if the CVE is in there. It has zero risk score and zero instances. It's a quick self service method that the vuln ID guys can easily prove to any auditors it doesn't matter, while still showing it does on distros that may not have the same backporting. Far faster search with still authoritative-enough results for the vuln management guys.
Sincerely, your much maligned friendly Cybersec spiderman who dislikes the posers on infosec, infra, and dev teams as much as you do.
JohnnyricoMC@reddit
You're spot on.
cyvaquero@reddit
That is a failure on your security people. All we do is send them the RH CVE which documents where it was patched and the artifact showing the system at or above that package.
That said, it took some meetings to get everyone to understand and onboard.
graph_worlok@reddit
Funnily enough, I just dealt with the exact opposite. But I deal with that too. Constantly.
Key-Web5678@reddit
Oh God this is what exactly happened to us in our last pentest.
OpportunityOk567@reddit
DrSnuggs@reddit
Have my angry upvote or downvote. I don't know what's appropriate.
AGsec@reddit
100%. I feel like you see this more often in cyber security roles because of the explosion of cyber security degrees that have flooded the market with people who are do not understand it's a subfield of IT. I've been downvoted on r/cybersecurity for saying that, or suggesting that maybe someone with no experience and a grad degree and CISSP should be looking at engineering and admin roles vs ISSM roles.
Kuipyr@reddit
I have Systems Engineers that spin up systems without applying baselines. It is infinitely easier to baseline a server before you setup the services, doing it after the fact is painful.
Nnyan@reddit
It's funny I was just going to post about my strong hatred towards platform engineers. j/k. But seriously I've worked both sides of this fence and this take sounds less like “security teams are useless” and more like “your org has terrible security integration.” A lot of what you listed is absolutely security work, Platform/SRE/sysadmin teams are the core of security controls because they actually "own" the infrastructure.
But that doesn't mean that cyber teams "do nothing". It just means that you are confusing control ownership with security governance and risk management. A mature security org is not supposed to be manually configuring your NGINX box or rotating certs themselves. This violates the separation of duties (at least in a mature environment). Security defines standards, risk tolerance, compliance mapping, IR process, etc. Platform teams operationalize those controls. This separation exists (or at least should) for a reason. I think you are massively underestimating the amount of 'invisible' work good security teams do.
Your "they just throw CVEs at us" is just a symptom of of immature security leadership and/or immature platform engineering practices. I had to laugh a little at "block everything by default". That is literally standard security engineering practice. Default deny is foundational security arch.
Candid-Molasses-6204@reddit
Hey, hey, I resemble that statement. I also agree with you. There are quite a few CISSPs that can't run a basic Powershell script or administer a firewall. So being the devils advocate (and being a former Network/Sysadmin for 10 years prior). I have personally witnessed numerous events caused by someone forgetting to enable MFA, someone screwing up a firewall rule, someone forgetting about that firewall for out of band access and that becoming a significant security issue. When I say security issue I can neither confirm nor deny I mean bringing in forensic companies and having to bring in Cyber insurance because someone couldn't do the damn needful.
skiitifyoucan@reddit
hate is a strong word but.... i get a lot of low value busy work from these guys.. .which makes it clear to me how much time they must have on their hands
chiefmonkey@reddit
30+ years of Cyber security here. Yes, there are worthless people in some cyber organizations and every other organization. It's a challenge, but in my experience it boils down to poor leadership, poor communication and silo mentalities between teams. What fixes it? No magic bullet, but the key items are: * Build a relationship with cyber (if you know their kids' names, you're probably already there) * Explain in detail what, why and how you do what you do. What is easy, what is hard, and what you currently do to support a secure ecosystem. * Ask cyber to explain to you the same, and what they are on the hook for - what key controls are they concerned about? If they can only talk about certain tools they use and reports they read, you need to go up a level * Commit to a co-accountability plan where both teams understand what is needed, why, when and what priority. Ask them for help. Offer to help them.
I regularly have the tech and business teams present to my whole staff, and vice versa. We get things done because we've eliminated the confusion and trust issues. We help the business hit deadlines and they help us maintain controls and protect the business and customers.
Your mileage may vary, but I've done this at 3 large corporations with great success. Good luck!
sdrawkcabineter@reddit
Is there a digital signature for that binary?
TournamentCarrot0@reddit
Wishing you best of luck in the Mythos era if you’re upset about having to fix CVE’s.
vogelke@reddit
It's 100x worse in the US Air Force. The cybersec guys answer to people who can get them fired but can't find our base on a map, so I don't strictly blame them.
They're caught in the middle and have no recourse other than to run some stupid tool that finds the same stupid false positives every time. Replying to them about the false positives will get you crickets for a response, and then we do it again in 6-12 months.
japanfrog@reddit
Security teams at my work are highly qualified on paper but a bit inept on practice.
They are the ones during posting links to random blogs in CV group chats with highly technical people asking if it applies to us. They seem to do less work than a junior but somehow take all the credit.
many_dongs@reddit
As someone who works on a security team that behaves this way right now, it’s because the VP is a micromanaging ignoramus who insists on doing everything this way, seriously.
ThePr0phet_@reddit
Security guy here!
Good security teams should not add more work or headaches. They should be hands on and help automate a lot of the security stuff so that you don’t have to keep re-configuring stuff.
Maybe you’ve just had bad luck with your teams.
It also sounds like you’re talking about PAID cybersecurity. If so, you are absolutely right and most are dog sh*t. They will just forward you alerts or throw a bunch of vulnerabilities at you without doing a risk-based approach. The decent ones will easily run you $100,000+ per year.
Usually, this is why in house teams are better. They understand your environment and can be more hands on. For example, patching the vulnerabilities themselves, creating templates for infrastructure deployment, automatically removing dangerous things from .env files (aka DLP), etc.
webmaster9919@reddit
I never saw a security guy actually doing security like fixing a bug or securing a configration. They just cannot do it. I followed various pentests from different companies and they always only provide a report with correct this correct that. Noone can actually answer my simplest question how I should solve it. I like to ask it just for fun because they should be the security person but instead its alwas me.
Stryker1-1@reddit
Sounds more like you just work in a place without a proper cyber team.
dvizzle@reddit
We are needed more than ever now because everyone just wants to copy/paste their job into AI without thinking of the consequences.
caldks@reddit
It sounds like you just haven’t had the good fortune of working with actual professionals. A good cybersecurity / infosec team will do all the compliance and documentation work for you, provide early insights to help avoid issues before they even occur, and provide defence in depth so that actual vulns can’t be easily exploited before you patch. The pros are trained to function as a support and an integrated part of the team - not just parroting CVEs.
FeralSquirrels@reddit
Either your experience is limited, or it sounds like you're rather jaded. Arguably you've got a sore arse from the sounds of it - did something get you in trouble or is it the workload?
I guarantee you there's security teams, helpdesk staff and probably general staff who also think "I genuinely hate Sysadmins" as well, but you wouldn't want to be lumped in with them, right?
Exactly. There's great teams out there, both internal or ones we've brought in - it's like picking a mechanic for your car: some are awful, some are great. I won't say "every" mechanic is rubbish because I have a bad experience or two. Or am I being.....unreasonably reasonable?
Us who, you? Two people? If that's your "IT team" that explains a lot, then. Worth noting that there's a lot of work for a Sysadmin to do any given day of the week, CyberSec is also a part of that at times, especially if you haven't got a bigger team.
This is why orgs have dedicated teams - if you're the one being lumped with all the work then it's not a "Damn you, CyberSec!" it should be "Damn you org-I-work-for for not having a larger IT team".
Are management or your boss open to a discourse over the subject? Is it to stay conformant with ISO or something else like insurance? End of the day you gotta' accept it is a big responsibility and you do need to take these things seriously if your org is a target or vulnerable - it's what you sign on for.
Working in the industry, as you do as a Sysadmin ,means you accept every business handles your role differently and what your boss/manager/company as a whole decides your role is responsible for? It'll vary and let's face it: if you don't do it, who will?
May as well blame "the developers" for a problem in a game you enjoy after I dunno....a balance change? they're part of the wider team, but doesn't mean they're responsible for the issue you're having or decision making.
End of the day if there's CVEs, vulnerabilities/risks etc they highlight, it's then going to be a case of accepting the risk, mitigating it or if possible eliminating entirely. It's part of the job.
Steerable-Octopus@reddit
I don't interpret the complaint isn't that the cybersec team is being critical but rather that they're not helpful. My experience is similar in that I got a sense as well that cybersec teams mostly just did analysis and didn't actually engage in engineering any solutions. I didn't have as much friction with them but they certainly felt kinda useless. If the cybersec team's alert happen too often and if alerts aren't actionable then they're just exhausting the limited patience and attention of the team for no reason amd I think that's at least a valid criticism.
talent_de_tigan@reddit (OP)
I can assure you I am not part of some vibe two man IT team. Even within sys admin/ cyber space, there are many many teams. However there is a common theme within the cyber space - the cluelessness of the systems we manage.
I am not sure if they genuinely don't care, don't like, or are not expected to, but here in operations we are actually EXPECTED to understand the business logic of our systems - what applications sre hosted on which servers and what their intended purpose is. Where the cyber team don't have the slightest clue, and write policies that sometimes break apps in production because they genuinely have no clue.
ersentenza@reddit
Would you care to present some actual case of policies breaking applications?
Usually this simply means the application was badly built to begin with. The only other case is usually a new law forcing new requirements in which case you suck it and say 'yes master, will do master' while silently cursing the government.
glockfreak@reddit
Curious about this too as a former sysadmin now in cybersecurity. If we create a policy that will break a production application but still benefit 99% the rest of the assets then there is a review and exception process for that app so it doesn’t break.
tankerkiller125real@reddit
Sounds like bad hiring policies/training policies for cyber in your org then. A good cyber team is a solid mix of people who have been with the business for years and understand it's workings, and cyber security experts brought in for that exact purpose who should be receiving an in depth deep dive of how and why the infrastructure works the way it does when they join.
Has anyone in operations bothered to go to the cyber security group to offer to give them a deep dive on the infrastructure? And I'm not talking about a 1 hour power point high level summary. I'm talking " come follow me around for the next 4 weeks to see how our infrastructure actually works"
Robbbbbbbbb@reddit
That speaks to a failure in alignment and mismanagement between the two teams.
Cyber is not an IT function, it's a business function. They need to understand how your company functions and provide their remediation and mitigation requirements based on that.
It sounds like you should be speaking to leadership to explain this issue so they can properly align the two teams and make sure that the cyber function is enriched with the proper business logic that enables them to make meaningful recommendations to your team.
JoelyMalookey@reddit
They are not butthurt - if you have to chase everyone down to mitigate issues and the only thing they provide is the scanner report you yourself set up - you’d be bothered too. I doubt feel like you’re not getting the sentiment so maybe it doesn’t apply to you.
Th3Sh4d0wKn0ws@reddit
Thank you, as someone in Cybersecurity this gave me a good chuckle.
A lot of what you pointed out can be true. I've seen bad Security teams that don't do anything.
For some reason it stands out more than the Sys Admin teams that can't configure anything with security in mind, can only open tickets with vendors, and refuse to learn anything new.
Dense-Purchase2643@reddit
Charging you? Your company’s security team is external to the company?
Who are these external people that have access higher than devops/sysadmins but do not work in the company?
enigmaunbound@reddit
The feeling can be mutual. It's always a joy when an ops engineer decides their conscience and experience outweighs technical security issues. It's always fun having to argue again and again why yes, patching does need to be expedited. Here are the risks and here are the threats and while there is a single percentage risk of failure that doesn't justify not acting. And, yes I do need system level access during incident response. Escrowed credentials are fine but you can't perform my tasks efficiently. No that spreadsheet of passwords is not appropriate. Just because you have always done it that way doesn't mean it was ever a good idea. Yes you do have to rotate the system service accounts on admin terminations. You never have? Why not try it in test/dev? No test/dev? Huh, your compliance docs say otherwise. Can we all agree to go drunk now?
mn540@reddit
Former CISO at two different midsize companies. I worked with a lot of CISO who sees things as black and white. They talked about risk based decisions, but they want no risks. It absolutely drives me bonkers. Many of these CISOs never had to truly manage a production team. They don’t understand how some of their insane policies put a stop to things.
I worked for one company where it took at least 1 year to get a NDA signed because the CISO blocked everything. I worked for another company where it took MONTHS to purchase anything. Company uses firewall brand x model 100. I want to purchase firewall brand x model 101. Instead of a short security review, we do a complete security assessment even though we did the security assessment in module 100 just 6 months ago. Absolutely INSANE. No wonder so many people dislike the information security team.
ersentenza@reddit
TBF, how many times have we seen brand new model 101 introduce defects that weren't there in model 100? You can't just trust it's the same.
pkvmsp123@reddit
Yeah, I think the important thing is finding some balance. You’re right that some risk has to be accepted, otherwise nothing ever gets done. But at the same time, in your example with the newer firewall model, there probably is some additional risk there too. Especialy on newer hardware/models where firmware updates may not have had a ton of real world testing yet, organizations can end up assumimg more risk than they realize.
I’ve seen both extremes honestly. Security teams that block absolutely everything, and ops teams that push things into production way too quickly with almost no review at all. Somewhere in the middle is probably the right answer.
viking_linuxbrother@reddit
The amount of cyber sec people who can't do anything is astounding to a sysadmin or sre but a good one is invaluable. You just gotta figure out which is which and then treat them accordingly. If all of your cybersecurity folks are just "scanner bunnies" then you are gonna get some weird requests and have to say no to a lot of "fortress mentality" bullshit.
1TRUEKING@reddit
do u also hate the pentesters that literally broke into your infra in like 30 minutes because of bad security posture as well?
beagle_bathouse@reddit
bait
breadstickz@reddit
This thread comes up every week here and it’s largely because sysadmins don’t understand that the role of information security isn’t to be part of IT, but to be part of risk management.
Sysadmins have this expectation that because infosec employees generally come from an IT background and are supposed to have deep technical knowledge that they’re also supposed to be configuring the security controls in an infrastructure team’s systems. This is not the case (or it shouldn’t be).
You own and configure your own systems. Infosec advises the organization on the risk those systems introduce. Depending on the organization’s risk appetite, we recommend remediations. It’s quite frankly absurd to expect someone in infosec to configure an ACL on a router or create a GPO to alter a workstation or server confirmation, etc. Those are functions of a fundamentally different role.
Commonly, we own our own systems that you may or may not even be aware of (though the infrastructure team as a whole generally is). These are often systems like the SIEM, malware sandboxes, pcap collection boxes, etc.
This is all in addition to one of the most common infosec jobs of SOC analyst. It’s pretty obvious what those people do and why they aren’t involved in system configuration. Within the SOC team you also generally have threat hunters, incident responders, and detection engineers.
It’s vitally important to understand that while infosec feels similar to an IT role, it is fundamentally completely different and serves an entirely different purpose in the organization.
SomeCar@reddit
Don't worry, we hate sys admins too.
Sad-Comment-6018@reddit
Why?
sgguitars190@reddit
I’ll take a stab at this. These are things I’ve had to tell/ask some of our sys ads:
No you cannot install a backdoor to remote into your office PC from home. Use the VPN from an authorized device like everyone else.
Why did you put an EDR exclusion on the entire C: drive after using ChatGPT to figure out how to do so.
Why did you purchase an explicitly unauthorized, cloud-based RMM service suite.
No, it is not okay that a group of you share the credentials on a local account to manage a cluster of critical appliances (an account that belonged to a previous admin that has since departed).
Why are you logging into the VPN with your admin credentials
Why are you logging into your desktop with your admin credentials to browse the web
Please don’t send users their plain text passwords over email and not require an immediate password change
For the love of all that is holy, please remove the management interface from being public facing
The above is really just the tip of the iceberg. A common theme here is cutting corners for the sake of convenience. Sometimes you do have to log in to a device with your admin account, but don’t do it on your daily driver every day just because you don’t feel like priv’ing up to perform your actual administrative tasks. Sometimes there are very legitimate reasons for an EDR exclusion. But make it a specific directory, not the entire drive.
There are also some really fantastic sys ads where these things aren’t an issue, and I’m sure it’s frustrating getting a printout of CVEs. But as others have said, we send these to you because you are the subject matter experts administering said systems. We rely on your input if something is not applicable, or if something has been back ported. And if you’re a larger organization, you’re not the only administrator or environment we’re doing this with at the same time.
I get where you all are coming from as well. Stricter security controls make your daily tasks more difficult. I think it’s absolutely a balancing act. The biggest thing to keep in mind for us on the infosec side is when (not if) an adversary breaches the network, what will they be able to do, how will they be able to pivot. Will they have to make a lot of noise to try to move laterally? Or were they handed the first set of keys they needed and blend right in. What I listed above just does their jobs for them.
SomeCar@reddit
Anything that goes wrong with any of their systems, it's automatically the fault of a security tool (EDR, vuln scan, policy change). When a single core of a 32 core system spikes above 3%, we get a call asking to uninstall security tooling. No troubleshooting at all, just point the blame at security.
Thatawesomeguy4@reddit
Excruciatingly true. “We are having firewall issues, cancel all vulnerability scans”
reaper987@reddit
Don't worry, they will fix the firewall issues with Any:Any rule.
StrategicBlenderBall@reddit
Because all you do is bitch and moan when your systems get scanned and you get caught being lazy.
/s, kinda.
Unusual_Extreme_One@reddit
Can someone tell me what value the Cyber security teams are offering your average system administrator? I have never worked with a Cyber Security person who has assisted or provided any value. They seem to be glorified project managers who get paid more because they have Cyber Security in their title. They never know anything about the technology they are asking for remediations on. They never offer any reasonable advice or guidance or assistance.
In my experience most of the time they read a bunch of impossible unreasonable security recommendations and then make those recommendations official policy.
After that they tell sysadmins to enforce this policy blindly.
I create grant access or install the security scanning software on the endpoints. The software runs and creates reports about vulnerabilities in my environment. The cyber security person takes the reports directly from the software and hands them back to me and tells me to fix it all.
They literally hand me the reports. Then they harass me nonstop for ETA’s on when my understaffed team can remediate a false positive or 100% of all endpoints. 90+ compliance isn’t good enough because the system is reporting X number of endpoints are still effected from a last scan date of 7 weeks ago.
I genuinely want to know what they are supposed to be doing that is security related because anyone can hand me a PDF of a report generated by a system and ask me when I am going to work on it. You don’t need any cyber security skills to do that. I have worked for over 20 years as a sys administrator, and I have never interacted with a cyber security team that has done anything useful for the infrastructure team.
Do I have this wrong? What are they supposed to do?
robocop_py@reddit
Many of us older security guys and gals used to be sysadmins. I was a sysadmin and network admin in a previous life. Before that I was in help desk. Here’s a few things to keep in mind:
Security should only have limited keys to the kingdom. No domain admin access, no firewall admin access, nothing that lets us make changes in the environment. We need read access to everything and maybe access to disable accounts and isolate computers. That’s it. We can’t be sysadmins too.
Yes block everything. Default deny. Then allow only those IPs and ports you need for stuff to work. That’s generally accepted security practice and it’s because we’ve learned that keeping the attack surface as small as possible limits attackers’ ability to pivot in the environment.
If I’m giving you a list of CVEs it’s because I don’t understand the architecture of what I’m seeing vulnerabilities in. I don’t know if Log4J is even a thing to be concerned about with how you’re using Java. If the architecture was better documented, with data flow diagrams and system definitions then I could provide better remediation advice.
ddesla2@reddit
This.
I've been sys admin and engi for years. I pivoted to threat and vuln mgmt bc it was of interest to me and I saw an opportunity for a fruitful career there. I build out the program to focus on vulns that are being exploited and relay where I see those vulns and what to do to fix them. Then I validate and let you know if you were successful or not. It's imperative to keep the attack surface small and prevent these easily exploitable vulns from showing up on web facing assets for obv reasons. There is a method to the madness tho, I promise you.
screampuff@reddit
I have been sent CVE's for "Cisco Firewall Management Center" because we have Cisco Catalyst switches.
You don't need the 'application expert' to narrow down whether or not our actual inventory is impacted.
knightofargh@reddit
This post is the actual truth. I know a lot about systems, I cut my teeth doing automation and server engineering. I was there when VMs still sucked and cost more than bare metal.
I can identify a risk. I can tell you how it works but I can’t possibly know your application which you are the theoretical expert in. The basics are the basics, but there are limits to deep knowledge and how much you can possess.
screampuff@reddit
When I was a systems engineer, I'd be sent CVEs for Cisco systems we don't even use, just because we have Cisco switches.
GoogleDrummer@reddit
My security team gave me a ticket regarding a particular .NET CVE saying I needed to remediate my entire Windows fleet. When I searched that CVE and landed on Microsoft's page about it, it said three times if you were running Windows you were fine.
I get what you're saying but based on my experience, and talking with colleagues at other companies, you're a rarity.
bitslammer@reddit
I can't upvote this enough for #3 specifically.
I work in a global org of \~80K employees in 50 countries with just under 3000 apps in our CMDB. The VM team is a team of around 10 people. It's completely unreasonable to expect those 10 people to understand all of those 3000 apps and be able to handhold those app owners in doing the remediation. Even if they were miraculously all knowing they wouldn't have the time.
It's up to the hundreds of system/app owners to be the experts we hired them to be when it comes to fixing stuff. The VM team will go out of their way in pulling more data, helping them setup on-demand scans, showing them the raw findings, etc., but as an app owner it's on you to do the digging.
Tasty-Ad-580@reddit
On the flip side, I know a lot of sysadmins who are very protective of their duties and do not want to delegate some of the responsibilities to sec engineers. We had a compliance audit done recently and were getting flagged for stupid shit like smbv1 shares and unencrypted sql credential transmissions in our environment. We notified them of these issues months ago and are still waiting on them to remediate. I used to be a sysadmin and moved over to secops and can easily take care of it for them if they would give me access, but they are holding on to their responsibilities for their dear lives because they are scared that we will take over their duties.
Fit_Reveal_6304@reddit
Someone just failed their security audit I'm guessing
bmelz@reddit
Probably clicked a simulated phishing test
PoliticalDestruction@reddit
“Half a decade” is a weird way to say 5 years..
bmelz@reddit
Not even half a decade, it was "almost" half a decade.. so like 3-4 years, lol.
kruvii@reddit
Hahahahaha
tayf85@reddit
One time I tried to get QGIS installed on my corporate laptop. Cyber Security team said it doesn’t meet their security requirements. I point to the fact that the NSA use it—I link them to the NSA’s fkn GitHub which is full of QGIS stuff they’ve open sourced. Apparently it meets the security requirements of the NSA but not good enough for our company, lol.
cwk9@reddit
Sounds like your cyber security team lacks practical hands on IT experience. Or they're siloed to the point all they can do is toss CVEs at people. I work embedded in the infrastructure team and not in a separate department. Security is a team effort and gets considered right at the start of new projects.
cryonova@reddit
Lol bro just tell us you don't understand security. This is pure comical gold.
roastedfunction@reddit
Security teams get to set all these ridiculous targets like “remediate all critical CVEs in 7 days” then they claim other team’s work as their own success. It’s fucking infuriating. CISO externalize all the costs & work then claim they hit their own “KRIs” for their fat bonuses. I’m in a regulated industry so none of the other technology execs ever stand up against security for fear of being perceived to be advocating for less security.
Same_Bat_Channel@reddit
Being in security, for every one of me theres 10 of you. Could I fix everything, sure. But that doesn't help the business when ops and apps teams continuly committ things to production without consideration. Then whats the expectation, I just have to come wipe up your mess like I do for my 3 year old? Sure are many of the cyber finding minimal and do they use scare tactics to get things done, sure. But let's be real and honest with ourselves, many of the things are fixed very easily, perhaps you don't think so, but they are. But like my 3 year old, I need to coax and pressure you to eat your vegetables because if you turn out fat and unhealthy whos to blame. who's accountable when audits come in a mess or theres a breach. Certainly not you, when that happens you'll point the finger at the security team.
MashPotatoQuant@reddit
What are you advocating for? Giving people access to your infrastructure to do security work?
thevnom@reddit
I feel like the reason we are so frustrated with them, is because they dont come into the infrastructure or the codebase, and fix the issue themselves, or add themselves to projects to add security while they are being built. This would provide tangible work to everyone, not just scan reports. They would be seen as providers and not constraints
h4ck3r_n4m3@reddit
For every dev that builds with security in mind there's 10 that don't know how authorization works
RobertJCorcoran@reddit
You are not paying somebody to export a pdf. You are paying for that somebody to know what to look for in that pdf.
talent_de_tigan@reddit (OP)
And who do you think is going to have to fix those vulnerabilities?
Robbbbbbbbb@reddit
Why are the vulnerabilities there in the first place if the system was set up correctly and within your company's specs?
talent_de_tigan@reddit (OP)
Why did mice come into your pantry? Your kitchen has walls doesn't it?
Tangential_Diversion@reddit
So your argument is we should live with it because it's pre-existing? I'll be blunt with you: I've exfilled a lot of CC data from CDEs and PHI from healthcare companies due to similar attitudes. Thanks for the job security I guess?
talent_de_tigan@reddit (OP)
Lol no seems a lot of people probably misread this in the same way. You can have as a reactive policy you want. New vulnerabilities will always arise. So asking why you didn't have 0 vulnerabilities is the same as asking why are mice in ur kitchen lol. They will always find their way. If you disagree with that you probably never had a serious tech career
Software is a liability and it will always be
Tangential_Diversion@reddit
I don't disagree that new vulns pop up. However, after a certain amount of "live time", an active vulnerability persisting points to a flaw in your vuln scannint/patching/monitoring process.
OneSeaworthiness7768@reddit
If they’re in the systems you’re responsible for, why wouldn’t it shouldn’t you be the one the fix them?
iSunGod@reddit
lol then why didn't you just run whatever tool was run, generate the report for yourself, then fix the issue yourself before whatever team/service did? You'd be the big hero of the day. Save so much time & money for the company & you'd know exactly what's wrong, how to fix it, and the downstream effects of that change.
I do understand your gripe, though. When I joined my current company there was a massive rift between the security guys and the global infrastructure engineers for exactly the reasons you're whining about now - like they'd literally scream at each other in meetings. Security would run a scan, kick over a report, and say "fix it" with no conversation on priority or further detail.
On one hand the report literally scores priority & has documentation included how to fix it. On the other there are a lot of FP that get kicked over the fence. The tools aren't perfect but it should be a partnership & a discussion to make sure the right things are being addressed in a timely fashion.
RobertJCorcoran@reddit
Yes, but following which direction? I mean, I don’t know where you work, but in my company SecOps is a small sub-team of IT, and any patch is discussed and implemented step by step following a plan created by both teams. So I don’t get your initial post either way, you hate someone because does a different job?
anxiousvater@reddit
Most of the SOC guys have no f**king understanding of technologies themselves. Just run a bunch of tools, look at SIEM logs & assign tickets for respective teams to address.
Few days ago, there was a supply chain attack, I had to check if Palo Alto FWs are doing DNS poisoning of the suspected domain ie., just with an
nslookupbut those bastards didn't even check the command rather just asking me to provide justification why I accessed HTTPs website. Can't even check DNS vs HTTPS.Their CVE justification is also a joke, look for steps to reproduce on CVE & claim as if they are the oned who wrote those steps. Sometimes CVEs have no fix coming from vendor (takes days to weeks), they give a f**k & expect sysadmins to patch those (close to zero day vulns). Little they know that it's not possible to patch something without a fix from vendor. I push it back to them to find a way to identify violators using their EDR tools & flag with me, I never got any response as they have to do that work.
RobertJCorcoran@reddit
You can find stupid people pretending to know in every team. Clearly I am lucky because so far where I work, I haven’t seen a person not understanding the stack we use
BigLeSigh@reddit
And that’s still the sysadmin
FrancescoFortuna@reddit
Blocking everything by default is a good security posture. That is the first step to zero trust.
wezelboy@reddit
We had some auditors come in and they couldn't even figure out how to run their scanning tool on Linux.
CluelessPentester@reddit
That probably explains why there are dozens of security incidents in the news every week.
talent_de_tigan@reddit (OP)
There are indeed many incidents caused by us. But it feels like we have to mansge your shit while keeping systems up and running.
I Imagine a gardener who is doing all the work, design, maintenance. While they mow the lawn, some dude comes up to him and says "you missed a spot there".
BadSafecracker@reddit
That's...literally the point, though. The gardener is paid to do all the landscaping and they're focused on their tasks. The guy comes up and points out a flaw, not because he hates the gardener, because he wants the landscaping to be the best it can be. The gardener messed up and it needs to be fixed; it doesn't matter if they still need to trim the hedges and plant the roses.
I've been a sysadmin/engineer and am now in infosec, so I get your frustration. The infosec guy should be looking at the big picture of the org and environment (not just the technical, but the compliance and assurance, as well). This is not just to ruin your day or make extra work for you.
talent_de_tigan@reddit (OP)
OK, now imagine the person who is pointing out the flaw is also getting paid. In fact, he gets paid substantially more than the gardener.
Do you see my analogy now. Of course its a flaw and it needs to be fixed. But do you think the amount of work the gardener and the person who pointed out the flaw is the same?
Now tell me how that is fair
BadSafecracker@reddit
Yes, it's fair if that's the pointing guy's job. It's his job to know what doesn't meet expectations. It might seem like the guy is just pointing at something, but it's his job to know the issue, find it, and alert the correct people - similar to the analogy of the guy that paints the X on a broken machine. Pointing guy is probably the only one that knows the HOA's rules that grass can't be over 1.5 inches tall and knows the potentialities of that rule is broken.
I will concede that not all InfoSec people are the same and I've known some that had very little IT background.
cmack@reddit
wrong ....making up stuff
PK84@reddit
I have worked with some awful sec teams and some really good ones. The problem i had are sec teams being super lazy. Just giving demands then when you explain it isnt that simple they get all pissed. Also if i ask for a compromise "that's not my job to know". I would just do their job for them. Then I had great sec teams that got into the weeds with me. We would go through PAC files together, architect and be super collaborative.
I got my CISSP and am a director now, so I force my sec teams to understand the engineers side, it only makes them better security admins.
DrStalker@reddit
Here's the trick: paste it into your risk registry, add "this will take an estimated X days of work (including team Y as well) and $Z to remediate." then assign it to whoever is in charge of deciding to allocate resources vs. accepting the risk.
RevWaldo@reddit
Common-Carp@reddit
Ugh. You’re not wrong…
badaz06@reddit
"Oh My God there are standards I have to meet! Pity me! I'm so abused" (SMH)
largos7289@reddit
So you never met the Cyber security guy that made OU group policy changes to lock down USB ports because it was a security risk? Fun times... We had that guy for a long time till the complaints kept rolling in on him. Dude was nuts... So he did that change then all of production just halted because the licenses that worked equipment had usb dongles in them.... FUN FUN!! What was funnier was he did this on a Friday, didn't tell anyone and just left on vacation. LOL when he came back they told him to extend his vacation indefinitely.
argama87@reddit
That is why Change Management is a thing.
frankentriple@reddit
Yet, nearly every day I get firewall request from someone on the dev team to open port 80 on the firewall so that their web app will work. Weird.
We tried this without the cybersecurity team once before, remember? Were you around in the early 00s?
CaptainZhon@reddit
100%. They are there to scare management into thinking the entire environment is a giant honey pot and you are not doing your job to “secure” it.
ThrillzMUHgillz@reddit
Yeah the security team here gets paid assloads to read business letters and forward emails about potential attacks. Just to say “ya better do somethin”
We also get the same letters and do routine patching.
I genuinely don’t know what they do.
Sysadmin, Networking, DBA teams here all prioritize security above all else. And always have.
Security is actually a fairly new team. I think it was pushed by the board. Just assuming we needed one. But they’ve added no value. Just hit the budget and actually cost us some convenience with licensing certain product. Making our jobs ever so more tiring.
egamma@reddit
The reason why we have cyber security teams is because platform teams DIDN'T do all the security things for 30 years. So, we're paying for it now.
Always good to have someone else check your work.
GoogleDrummer@reddit
Meat based event forwarders.
pneRock@reddit
At the first company I worked at, there were two groups of security. The first did a nessus scan and dropped a 600K row csv file on our doorstep and told us to fix it. Not super useful. The other security team pentested our own apps, walked around with a wifi dongle on their laptop to catch a rouge network a vendor setup, and ran the anti-exploit agents across the org. One was policy paper pushers and the other was bad ass. The problem is at a certain point you have to have both for SOC2/ISO compliance: one that takes care of the audit evidence and the other to actually keep things safe. I sure as hell don't want to deal with customer/vendor questionaries on policies, let them.
Reeces_Pieces@reddit
Most of my security work is explaining why whatever x scan tool picked up was a false positive.
tin-naga@reddit
Me too bud. Worst of it is leaderships who buy into their rapid fire unrelated timelines and you have to figure what’s breaking what. Then our system owners turn into grumps.
coolbeaNs92@reddit
I think the problem with InfoSec is that generally speaking, they lack both practical knowledge/understanding of how the real world works, in combination with poor soft skills.
As an infrastructure engineer, I would love nothing more than to CIS L2 everything with no exceptions, patch everything day 0 and resolve every single CVE that exists in our environment. Sadly however, it's just not that easy when you live in complex environments with 100/1000s of systems and thousands of users in an always on environment.
The Security engineers who I've worked with who are actually pragmatic, understand the realities of working in large environments and work with you to actually focus in on the important things, are diamonds in the rough. I've worked with a few of these people and I loved it and learnt a lot from them.
The vast major of Cybersec people are either compliance, or they are just tool/report people who take something, add the bare minimum around it, and then chuck it over the fence for someone else to resolve. And they do it with an almost ant hill esque mentality, wondering why they don't get the responses/engagement they think they should.
Cybersec was infinitely better when it was former Sysadmins and engineers who wanted to pivot into a different path. Now that Cybsec is a straight out the gate pathway, the quality has gone down hill so fast.
FatalSky@reddit
Yeah I was like that when I started 10 years ago. Shit was always a thorn in my side but I learned to work with it. We fixed the communication breakdown about 6 years ago on STIG implementation and vulnerabilities and how they get reported to the team. Some cool custom tools were built out of that that work amazingly. The coders HATED learning hardening practices and pushed back hard (heh). But after their shit version control absolutely fucked a dev cycle, management finally slapped them in the nuts and said do this, this, and this. The best thing I did to help my self was to sit down and do a month program in cryptography. Like full implementation from beginning to end across windows and Linux environments. The next best was logging and forensics.
bmelz@reddit
"almost half a decade"... So what, 3 or 4 years?
budlight2k@reddit
I'm with you.
I often find that they are not technical and don't understand consequences or systems, say things like remove all browsers from all servers or add two factor to PAM or block github and remove monitor and mouse drivers and prevent scripting.
Conercao@reddit
I generally ignore the Cyber Security guys at my workplace. They go on one course and think they are god's gift to security. Half the time they are wrong and the other half they just don't understand what they are doing.
I had an issue where they had blocked a well known chromium based European browser solely on the basis that Microsoft didn't give it a score on their threat scanner thing... as if nothing outside of Microsoft exists. I gave up arguing with them in the end
Quasi26@reddit
Sadly, the need for dedicated cyber security teams (some of which are good some of which are terrible) is born out of decades of terrible sysadmins running everything with root access or just giving admin access to solve a problem or “I’ll just set this up with no restrictions now just to test” and leaving businesses extremely vulnerable. Cyber used to just be a component of infrastructure and development until it became clear that they old not be trusted because whenever they were asked to make a decision between fast and secure they always chose fast.
Knyghtlorde@reddit
Every cyber sec person I have worked with has been fantastic.
graph_worlok@reddit
Theoretical Degree in Information Security?
Zackey_TNT@reddit
Oh so you've never worked with one?
Knyghtlorde@reddit
Been in the industry for 30 years, and worked with quite a few.
wrt-wtf-@reddit
Join the industry this week?
Knyghtlorde@reddit
Been in it for 30 years
LegRepresentative418@reddit
I know the firewall interface has a self-signed certificate. I put it there. I'm the only one that has a login. Get out of my office and take your stupid Nessus scan with you.
tdic89@reddit
I’m afraid I have to disagree, cybersec/InfoSec teams are there to keep us accountable.
Plus, if something does go wrong, you can point to the cyber security policies you follow and protect yourself from liability. It’s not your fault if your organisation got attacked even though you follow your org’s InfoSec policies to the letter. The fault is then with InfoSec.
cmack@reddit
You'd be wrong too honestly. Too many cyber secs forget the A in the CIA triad. Without the A, nothing else matters. Close up shop.
graph_worlok@reddit
All depends, in some circumstances it’s always better for downtime than the other possibilities. I’ve persuaded major dev teams to shut down systems for a few days with no pushback.
Robbbbbbbbb@reddit
You're both right lol.
Your point is pretty common all over the business spectrum. I brought this up with a decision maker the other day when going over MTD. They told me that 100% uptime was a requirement, but when they started to understand what goes into 100% uptime (and how much the "march of nines" costs), the lightbulb went on.
This is also part of the risk process. Like, in a perfect world, the cyber team reporting the vulnerability should trigger a BIA and Risk Assessment. But if the team's threshold for reporting isn't tuned, this would be noise that would just fire off endless work to GRC folks and the vulnerabilities would take forever to reach Ops.
Cyber is a hard program to get right. Very few orgs do, and even then, the disconnect between Ops and Cyber can be rough because... well, cyber is giving them more to do on an already tight schedule.
Sad-Comment-6018@reddit
This. They take the blame
unclescar@reddit
As an ex SysAdmin that moved into Security - I agree.
This entire sector is made up of under-educated Grads who think they're experts.
Jaded SysAdmins who have been off the tools so long they're useless (looking in a mirror here)
and the Compliance folks who've moved over and want to tick boxes with no experience of what that actually involves.
I'm out here fighting the good fight, trying to be a security guy who helps make the fixes and prioritising the work we push to our Admins without just saying "HAH YOU BAD FIX THIS"
smuziq@reddit
There are too many "Cyber Security Professionals" that do not have experience as an Infrastructure Engineer and know absolutely nothing about how to implement secure infrastructure. It's infuriating. There needs to be more Cyber Security Engineers or Information System Security Engineers that can bridge the gap.
graph_worlok@reddit
The fact you are complaining about “charging 10k” and generating PDF’s tells me you are working with an outsourced group rather than in-house. That’s the cause of most of your problems. I’m in infosec. I can, and have, done your job. I’ve also dealt with the exact things you are dealing with from both sides of the fence.
StrategicBlenderBall@reddit
Cybersecurity engineer here, well over a decade into my career. From my experience mostly in the public sector, we have to deal with many problems.
First, the old heads in management that don’t understand the difference between compliance and risk management. They just want ALL the boxes checked. For example, “What do you mean that system is OT, runs in a sandbox and can’t implement enterprise controls?” Boom, non-compliant. No nuance. No willingness to understand.
Then you get the youngins, fresh out of college with a masters in cybersecurity. No practical experience. Can’t even plug in a monitor. They get spun up on running Nessus and STIG scans. “Hey, the water plant is running on Windows 7. They need to upgrade to Windows 11.” No. They don’t, because they literally can’t. Unless you can convince the big guys to spend hundreds of thousands upgrading not only the software running the plant controls, but all the PLCs as well. Good luck buddy.
Last, you get the sysadmins that refuse to take part in any sort of the documentation process. If I come to you asking to help me write up some mitigation statements, or to draft policies that actually apply to your system, don’t cry about not having the time. Take a few hours so I can make both our lives easier. Please.
Jacmac_@reddit
I hear ya. The block everything approach was taken by our company. They had a crazy firewall rules request system that involved Word documents and spreadsheets on a SharePoint site and I'm talking about a company with 50K+ employees.
The monitoring was so tight, that me running powershell queries to the four (!!!) different directories to check if employee infomation was in synch rasied some kind of red flags and they would keep hasseling me like I was a hacker. It used to be that they were worried about the external threat. Now they see the threat everywhere.
That said, the big security push came out of a break in when the company took a less serious approach over a decade ago. After that happened, they started taking things more seriously. I feel like they did a lot of it without much planning and thought and implemented system with onerous managment overhead, but it was above my paygrade to do much more than complain to managment that they were costing themselves money. After all is said and done, it's their money to burn up as they see fit.
huskyvarnish@reddit
It’s been happening for years. About 14 years ago, we had a wireless bridge between offices. Our company hired a “cyber professional” to ran a pen test, and they found a version of SSH to be insecure because the report indeed was red. I told them i had already contacted the manufacturer’s development team and verified it had been recompiled and made safe by their team and the CVE did not apply to their equipment - they had no clue what I was talking about, but it was insecure and needed to be removed. I eventually won, but it was a PITA dealing with them.
galland101@reddit
Some cybersecurity guys are just dipshits who demand to use Macs (“for the command line”, lol) and then proceed to backdoor all your Linux systems with cron jobs that send the logs to their mailbox, all the while talking shit about Microsoft sucks.
StiH@reddit
Nowadays, Cyber sec department isn't IT, they're compliance. They make sure everything that's wired in the company checks all the boxes and that's it. If they understand the background on how things work, that's a bonus, but they're primarily there to use their own tools (for checking those boxes) and report what they find.
I understand where you're comming from and I've had (and sadly still have) the same frustrations as you, but it's just not worth my time and nerves every time I have to explain for the n-th time why when their scanner flags an old kernel on my servers, they scanned a file sitting on the system, not the running version and that's not going to change, because it's a feature of the OS to be able to boot an older version in case something breaks, but that's not a security risk since their CVE doesn't apply to a file just sitting on a disk somewhere...
ArmondDorleac@reddit
“throw a bunch of CVEs at you”
Well, yeah. Why are you building stuff with a bunch of CVEs?
Common-Carp@reddit
I’m over here on a security team having to explain, to network engineers, why QUIC is a nightmare. I have to explain powershell scripts and graph api calls to system admins who only know how to use a GUI. Somehow, I don’t end up hating system administrators… just inept folks who don’t care to learn. Another thing, your description of job duties should be spread across three roles at least. Developer duties, network engineer, IAM engineer, at least. Making sure apps don’t get DDOSed should be a collaborative effort between your network/WAF, and security teams.
I wonder, though… have you ever exploited a CVE? Have you seen how easy it is to pivot through a network and crack a domain controller if the sys admin doesn’t fix the CVEs or just plainly misconfigures things? I have. Have you ever gotten too sift through millions and millions of lines of logging information to find the initial point of entry, identify lateral movement and additional compromise, determine what was exfiltrated… etc? Something tells me your attitude would benefit from a security event without dedicated security team assistance.
SevaraB@reddit
Shitty sysadmins and shitty security “analysts” are both going to ruin your day.
Good security analysts aren’t getting paid to stand in front of the firewall with a flaming sword, fending off all attackers- they’re paid to make you think twice before yeeting something into production that can take the entire company down as soon as Little Bobby Tables comes visiting. The name of the game is “compensating controls,” and things go a LOT more smoothly with security if you already have a few planned in your back pocket.
Chumphy@reddit
I see you met the security guy where I work. Gets paid the same as me, but just gets paid to send phishing emails and use ai to write policy. Also doesn’t have an on call rotation or anything. Outside of work he’s not the type of guy you’d want as your security analyst either. Yeah, I’m not a fan of security people either.
Maro1947@reddit
in my experience,.Cyber teams are the new grifters who have little real world experience
Convince me this isn't the case
dabbydaberson@reddit
It's just like any other team. 1 out of 10 actually know how to do the job. The others just lean on the one that understands.
hardingd@reddit
Our cyber team is in house and works with us. We have a lot of legacy debt but are fixing it … slowly. There’s a lot to do but they help us prioritize.
OneSeaworthiness7768@reddit
I genuinely hate people who hate everything because they work for a crappy company and think their experience is universal so they have a weird chip on their shoulder.
naughtyobama@reddit
I can't fault you for lived experience. And there are real issues with cyber security teams:
Depending on their expertise, their objectives are different from yours and that causes friction. You're operational. Services have to be up and not causing downtime for the business.
If you're GRC (Compliance), your goal is passing audits and minimizing risk from regulatory requirements. Sone of these requirements are nonsensical in how they are audited by 3rd parties but GRC still needs to get the company through them. So, these teams don't hire for technical expertise and harmony with sys admins. They hire for skill in navigating those standards. This creates a TON of friction, even with other sister cyber security teams.
If you're a SOC, you're either a lean internal team with managed services, poor logging, etc. If you're an internal shop, you're probably logging poorly and full of junior kids overwhelmed with tickets. In a handful of cases, logging is sufficient to be competent, staff is reasonably competent. They still need to constantly bug you because a lot of time early stage intrusions look exactly like regular admin activity. There's a million events to triage and attackers are always changing their targets and tactics.
If you're a security admin/engineering team, you probably collaborate the most with sys admins because the admins have to reach out to you to implement IAM, firewalls rules, etc.They think of security first but they also have a mandate to do no harm.
For every competent sys admins, you have the other ones who want and have God powers on their regular account they check emails with, do t want to use PAWs, jump servers, onerous MFA, etc.
So, a lot times, how painful the experience is depends on how big the security budget is, whether you're working with kids vs experienced and knowledgeable adults, whether security functions is internal or outsourced.
iSunGod@reddit
Dude my company hired an sec analyst that was the dumbest MFer on the planet. He and an Inf engineer randomly started meeting to go over BloodHound findings from a pentest. They were meeting for monthly & trying to fix the stuff in the BH gui but after the first "fix" they spent the next 6 MONTHS trying to figure out why the gui never updated to reflect the fix.
The inf guy finally invited me to their meeting & explained what they did. The fix was technically correct but they didn't know they had to run sharphound again to refresh the data. The deflated, defeated, looks on their faces was priceless. That kid did shit like that all the time & wondered why people stopped engaging with him. People like him is the reason posts like this exist.
yaricks@reddit
The amount of tickets I get from our SOC or som the the security people that just contains a copy-paste from some random tool with no actual research or more in depth, but still flagged as a P2 incident is driving me mad. 99% of the tickets are irrelevant or so minor that it will be picked up in the next patch cycle, and it’s causing notification fatigue.
d4ganGera@reddit
Took the words out of my mouth!!
TheAgreeableCow@reddit
It sounds like you just hate inept people. These appear across all roles.
I spent 20 years in infrastructure and operations and the last 6 in cyber (now a CISO). I have seen security fakes and floaters. But most often I have worked with IT teams that just don't care so much about security. They need so much handholding to get the basics done. I'd kill for a decent engineer that had capability and initiative.
1_________________11@reddit
Security peoples job is to make sure you are doing yours right. Of course you hate them.
Fairlife_WholeMilk@reddit
Sounds like maybe you're bad at securing things on setup and expect someone else to fix the problems for you?
Kill3rT0fu@reddit
To be security it should be a prerequisite to be a sys admin for at least 7 or 8 years. You need to understand what you’re asking sys admins to do before you say “block everything”
code_monkey_wrench@reddit
I've had a similar experience as a developer. They are the masters of busywork and security theater.
ersentenza@reddit
I transitioned into cybersecurity after decades in system and network administration. Just recently I had engineers swear on their mothers that they absolutely HAD to keep the systems with years old obsolete and vulnerable components because the client runs their systems that way and we must be aligned and blah blah blah.
Me: that client is under NIS2 which explicitly prohibits running obsolete systems they can be royally fucked for that, how is this possible? Guess what it wasn't, the client actually upgraded their systems but they never even bothered to check!!!!!!!!!!
awetsasquatch@reddit
There's two kinds of cyber teams - technical, and policy. They ideally work in tandem but often don't in a lot of places. The policy group is far closer to Compliance than technical cyber security. As an example: If you work in a regulated industry like healthcare, they've likely forgotten more about the nuances of HIPPA than you'd ever learn, but that's their job.
sgt_Berbatov@reddit
Those who can do, those who can't go in to that bit of cyber security where all they do is box tick.
Gaming_Wisconsinbly@reddit
"Bro if you want to lock things down so much just go back to pen and paper"
-Cthaeh@reddit
My MSP has a large SOC. I know there are people over there doing things, but from our perspective its fuck all. Its like they just forward every alert to the client and the service desk. The lower teirs know almost nothing it seems.
Asleep_Spray274@reddit
Sounds like there is an easier job available for more money. You are knocking your pan in for less money. Who is the fool exactly?
CaseClosedEmail@reddit
Yeah, I disagree. You really need someone to check if your application and infrastructure is secure.
Bahurs1@reddit
Like the dude said. It's just a guy that clicks export to pdf and the job then is relayed back to me which is what I said was needed to be done, but we didn't have the budget. So this new hire/external audit somehow unlocked that budget? That's what this frustration is about.
Granded there are orgs that are swiss cheese and these audits are probably their best shot to get their shit together.
cyvaquero@reddit
They serve a role. Should Nessus exporters and CVE copy pasters be earning that extra bump in pay for something that requires no actual understanding of the subject matter? No.
However, as someone who has been in IT for 30 years now (including a five on DoD unclassified networks when it was the wild west). Things needed to be reigned in. Dev and SysAdmins have different motivations and therefore different priorities, a third party needs to put security foremost.
InfraScaler@reddit
If they were not after your ass your platform would be a Gruyere cheese.
Ill-Barracuda9031@reddit
Not nearly harsh enough.
1stUserEver@reddit
They are the pointer outers. We are the doers. “Hey look, this is bad”. Ok do something or offer solutions.
JoelyMalookey@reddit
I always joke they are our natural enemies lol we get the reports that we set up the system to generate and that we can grab ourselves
Pict@reddit
Yeah it’s a far easier job than yours, tbh.
angry_cucumber@reddit
depends on the realm, it's much harder when you are at a federal level and actually dealing with nation level training and funding.
sitting_not_sat@reddit
Yeah, based on my experience I agree. I have come across a few security folk that have moved over from other areas and are more self-aware and work with you, but in my experience they are in the minority.
Ente_0@reddit
While this whole comment may sound a little bit butthurt, on a fundamental level I wholeheartedly agree