security team handed us 600 vulns to fix. half werent even reachable from internet

Posted by relived_greats12@reddit | sysadmin | View on Reddit | 115 comments

Infosec ran their quarterly scan and dropped 600 vulnerabilities on us. 200 marked critical. Leadership wants remediation timeline by Friday. Spent two days triaging with DevOps. Most criticals were libraries we import but never actually call. Internal APIs behind VPN flagged as publicly exposed. Staging environments with test data treated same as production. Best one was a critical vuln in a Lambda that runs once a month. Scanner sees vulnerable package but has no idea if the code even executes or if anyone hits that endpoint. Asked security how to prioritize. They said fix criticals first. Cool, but which ones are actually exploitable versus sitting in unused code? Scanner can't tell the difference. Devs are ignoring security findings now because we've cried wolf too many times. Then we miss actual issues buried under the noise. We spent half our sprint on vulnerabilities that didn't matter. Meanwhile an actual exploit attempt last month took 8 hours to detect because buried under 300 false positives. Security team is mad we're not moving fast enough. We're mad they can't tell us what actually needs fixing versus theoretical risk. Has anyone solved this or do security and engineering just hate each other everywhere?

Reply to Post

Reply

115 Comments

[-]

Retro_Relics@reddit

why tf is your sec team not actually showing results from attempts to exploit? are they not actually attempting to pen test and just scanning? is it a side duty of another department and as a result they also dont have staffing to actually help you prioritize?

Reply

[-]

MrSanford@reddit

People that use the terms "Pentest" and "Vulnerability Assessment" interchangeably piss me off to no end.

Reply

[-]

TopHat84@reddit

Yes there is a valid distinction between the two terms...but there's no need to get your undies in twist because someone conflates the first steps of a pentest with a vulnerability scan. Since there is a small amount of overlap between the two. Don't perpetuate the "angry antisocial sys admin" stereotype. That mindset was dated, even in 2000s.

Reply

[-]

MrSanford@reddit

I think some outrage over a company being told they’re getting a pen test and in reality it’s a vulnerability assessment is valid. I don’t know what field of IT you work in but I pray it’s not cybersecurity.

Reply

[-]

TopHat84@reddit

I just said there is some overlap between the two, and you gotta go full angry mode Redditor. But after reviewing your profile, it seems clear you're just sad and lonely. No wonder you post in joe Rogan subs, probably another red pill "alpha male".

Reply

[-]

Retro_Relics@reddit

ok, fair enough, that one is on me. but the underlying sentiment that infosec should be helping them triage by how likely a vuln is to actually be exploited vs existing in theory is still sound.

Reply

[-]

bitslammer@reddit

That's not really practical in many orgs. I'm in an org of 80K employees. We have \~120K systems that are scanned every 3 days. Everything is highly automated. The team that runs the Tenable platform is only 8 people. The teams responsible for patching are \~400. There's no way that team of 8 can provide assistance across the 120K systems that are home to 3800 applications. We do have a well setup process for scoring and handling false positives and exceptions, but analysis and patching falls to the SMEs who are supposed to be experts in their systems. If you are the admin/owner of something like SAP, Informatica, OracleBI or any of those other apps, then you need to be able to take those scan results and research your vendor's site to determine if they are valid and how to remediate. The Tenable team probably hasn't even heard of most of the apps we use.

Reply

[-]

ddesla2@reddit

To an extent. I get where you're coming from for sure. The issue is, seasoned TVM program and staff should have a handle on where the most risk may lay. I say "may" here bc it's all quite subjective... External vs internal vs edge asset? Crown jewel? Etc. I get that they can't know every app used, esp in house built stuff, but they can certainly have a risk register and a basic understanding of what is most critical, what houses it, what all it touches (db's, VM hosts and the like), and triage from there. As for analysis and patching, TVM needs a solid foundational knowledge of how all that works to effectively do their job. I've personally done patching for giant global fortune 5 companies. I have analyzed more vulns than I can even count. Now, when I see them again years later, I can tell engineering that hey, this one is bc xyz was Uninstalled but it sometimes leaves this dll behind that is getting dinged by the scanner. Or even just recognizing that a whole slew of superseded crap is resolved by simply updating to the latest and greatest for the software or whatever. Or this log4shell still exists bc it's using the old version in a backup made of the app in this user's desktop folder...and so on. The point is,

Reply

[-]

bitslammer@reddit

The volume is simply too big for any manual prioritizing to happen. We have that automated using the Tenable -> ServiceNow integration. Every asset has a criticality score based on things like if it's in a DMZ, if it handles sensitive data, if it serves a business critical process etc. We also use the Tenable VPR score and other external sources to look at things like if exploit code exists for a given vuln and how how difficult an exploit is when possible. While nothing is perfect this is working well and all parties are OK with the current amount of effort. The largest frustrations come from vendors being unresponsive to inquiries or providing poor responses.

Reply

[-]

ddesla2@reddit

Nice. Sounds rounded out. I also suggest taking into account EPSS and if it's in CISA's KEV, but I believe VPR already does. If time is taken to add in asset criticality (ACR) and having that data enrichment from SNOW, you are in pretty good shape there. As for vendors and 3rd parties, gotta bake in SLAs and adherence in the contract to force their hand. Hate that.

Reply

[-]

Retro_Relics@reddit

Is your 8 man team repeatedly turning up 200+ "critical" issues and providing no help in triaging or even in accepting that a lot may be false positives and working with the team to remove false positives from the "critical' list tho?

Reply

[-]

bitslammer@reddit

No, those 8 staff have their hands full just running the Tenable and ServiceNow parts of the process even with the automation they are using. They do not regularly assist in any triaging. The volume we have is too large for any of the involved teams to do much manually, including prioritization. It's just not practical to expect one of those 8 people who may be in Canada or the UK to be able to contact some vendor for some obscure application in Singapore for help in assessing a vulnerability and potential impact of patching on their app. Time zones and language barriers alone are pretty steep. We have to rely that our system owners/admins are experts enough in their applications to determine if something is a false positive or not with their vendor when needed. As for prioritization that too is automated due to volume. Every asset has a criticality score that comes from the CMDB based on things like if it's in a DMZ, if it handles sensitive data, if it serves a business critical process etc. We also use the Tenable VPR score and other external sources to look at things like if exploit code exists for a given vuln and how how difficult an exploit is when possible.

Reply

[-]

merlyndavis@reddit

Why are you importing libraries that aren’t used? Clear out dead weight in your code. Vulnerabilities that aren’t publicly accessible can be deprioritized, not ignored. Also, should never only run scanner internally. An external run will help validate priority.

Reply

[-]

Humpaaa@reddit

Correct. >Cool, but which ones are actually exploitable versus sitting in unused code? That's your responsibility to assess, not the security teams. If you are responsible for an application, you are responsible to know how it works, and therefore be able to see if there a valid attack paths. That should help you to prioritize. Responsibility always is with the asset owner, not with the security team.

Reply

[-]

domestic_omnom@reddit

How is that not the security teams responsibility to know how something is exploitable?

Reply

[-]

vCentered@reddit

People confuse security with white hat hacker. That is generally not the case. Most security professionals that I have met really only know how to export reports from tools that tell them what the problems are believed to be. In my case they also are often not familiar with how the network is laid out or what the segments are. Or they forward reports from pentesters who really are likely only following a script of instructions for executing tools from Kali and really have no understanding of your network or the results of their testing. We've had pentesters insist we drop them into our segmented management network where the management plane for things like vSphere live, then submit those interfaces as "publicly exposed" on "Executive Summaries" because they were "able to reach them". In reality the only reason they were able to discover those interfaces is because they and our security team insisted we drop them into that network.

Reply

[-]

jstuart-tech@reddit

So the security team is supposed to understand your weird and wonderful codebase? You are the people who understand what it actually does and how it works. The security team is flagging that hey XYZ has a critical vuln, it's up to you to remediate. Or do you want the security team disabling shit that you've just spent 3 weeks working on because they don't understand what your trying to do?

Reply

[-]

domestic_omnom@reddit

I would expect the security experts to be more useful than an automated report. Or at least be able to explain the threat.

Reply

[-]

Acceptable_Potato949@reddit

They don't know what's in your head or your code. That's for you to decipher **to them**. Their responsibility lies in keeping you up to date on what could go wrong. You're the one to make sure nothing does go wrong. There's no automated report. Security teams have to be more than just vigilant and if you did what they're doing, you wouldn't have time left to do any code anymore. Respect it and STFU.

Reply

[-]

domestic_omnom@reddit

Im sysad, not a coder. Every report I've ever received from our third party sec team has been automated. Any question about the vulnerability was met with shoulder shrugs or "consult with your IT."

Reply

[-]

Acceptable_Potato949@reddit

I'm a systems architect and I can't even fathom what your organization looks like based on that description. It's only natural for the security team to tell you to consult with your IT, especially since they're not an in-house team. What the hell do you even expect them to say?

Reply

[-]

domestic_omnom@reddit

I expected an answer to my question. Not given the run around and gaffed off completely. Like jfc dude, why is professionalism and industry expertise too much to ask when it comes to info sec.

Reply

[-]

Acceptable_Potato949@reddit

It just honestly sounds like your organization needs to do better. If the 3rd party isn't living up to your standards, then consider looking for another provider. It also sounds like hierarchy and structure in your organization is not where it needs to be. This has nothing to do with professionalism.

Reply

[-]

jstuart-tech@reddit

While you might think your the only team in the entire org, you (probably) don't have a dedicated security team assigned to only you. Understading what you do is up to you, You 100% have the ability to tell the security team that XYZ isn't vulnerable due to ABC and they should understand that the risk has been mitigated by ABC. Depending on where the security team (and your own) maturity level's lay, then an automated tool that pulls stuff out could be applicible. Put it this way, before the security team told you about these vulnerable libraries etc etc, did you know about them? If not, they have helped you

Reply

[-]

magictiger@reddit

Exploiting a vulnerability and knowing that a potential vulnerability exists are two different things that can require different amounts of time. If a vulnerability is listed for a package and there’s no publicly available proof of concept (PoC), do we really have the time to spend to learn the vulnerability and build an exploit from scratch? I know that I and my team do not. Most companies can’t even afford to hire enough exploit developers to pull off a program like that. You’re basically tacking on 0s to the budget just so you can dunk on the devs. Much easier to go “Vulnerability in this package version in this function. Update the package or get your VP to sign off.” Then we give them the info so they make the call which to do.

Reply

[-]

reegz@reddit

I’ll be really quick and general, say there is a vulnerable function in Java. Your application is only vulnerable if it uses that function. The update changes how the function is invoked. If you update that library it could break the app. The app owner would triage whether that function needed updated in code, whether it’s even applicable etc. From there you’d either roll out the patch, or update code and then roll out the patch.

Reply

[-]

Humpaaa@reddit

A good security team should be able to provide that data as part of the scan (since most scanners include tools to prioritize via KEVs or Attack paths). However, responsibility for the security of each application is with the asset owner of that application. Security does only provide the data. In most orgs, security is a government function, not a technical role.

Reply

[-]

RobbieRigel@reddit

It depends on how much we get from Threat intel. Sometimes we know, the vulnerability down to the specific function call other times is vague on purpose like the recent PHP vulnerability.

Reply

[-]

themastermatt@reddit

Then they are nothing more than auditors.

Reply

[-]

Humpaaa@reddit

They are a governance function in most orgs, correct.

Reply

[-]

themastermatt@reddit

As long as thats all they claim to be. Fine. All their "findings" need to come with a big disclaimer "we dont actually know what this means, all we can do is forward emails and check boxes".

Reply

[-]

jaydizzleforshizzle@reddit

Feel like this is starting to become the understood norm, security folks very rarely tend to be technical, they are governance and management functions, which leads to a lot of PM like security folks who are just looking to check boxes they are being told they need to check, regardless of its actual technical nature.

Reply

[-]

Rhyobit@reddit

I don't think that's entirely fair - you need to appreciate that security folks have to cover \*everything\*. I'm a systems engineer from my background, but I moved into security architecture a year ago. Give me a linux system and I can run through what's wrong with it and tell you how to fix it. Give code it'll take a little longer but I'll get there too. Give me a windows ecosystem, and I struggle. Security folks can't be experts in \*everything\* the length and breadth of our full technology stack is too broad. Under circumstances where security aren't an expert, they will revert to governance to ensure that holes get filled to the best of everybody's ability.

Reply

[-]

jaydizzleforshizzle@reddit

I agree with your sentiment and in theory this is a solid mindset, but it’s slowly just not become the reality in enterprise. Every sysadmin repeats “you can’t secure what you don’t understand”, but we keep hiring ex military folks that don’t understand technology and just apply a physical security mindset and it’s painful to deal with sometimes.

Reply

[-]

somerandomguy101@reddit

A security team reporting security issues is no different than the help desk team reporting user issues.

Reply

[-]

Pup5432@reddit

I wish our infosec team worked as auditors. They expect us to write policy for the audit as well as remediate it. It’s more or less how US congress works now and that’s just sad. We should absolutely not be the ones making policy as no sysadmin should be, too much risk of juicing the numbers to make scans come back cleaner than they should.

Reply

[-]

Diplomatic_Gunboats@reddit

You have correctly identified their job.

Reply

[-]

jaydizzleforshizzle@reddit

Yes

Reply

[-]

Inquisitor_ForHire@reddit

Yep. Auditors and advisors. They are not the fixers.

Reply

[-]

themastermatt@reddit

Those who cant do, forward emails then disappear to the next InfoSecDefHaxCon

Reply

[-]

missed_sla@reddit

If you want us to be familiar with your code and make changes, then you can get familiar with the security frameworks and best practices and regulatory requirements that we are making sure you follow. If that sounds like too much work, shut your face hole and get back to your keyboard.

Reply

[-]

themastermatt@reddit

Oh but i am. Youve not the slightest what my scope is.

Reply

[-]

merlyndavis@reddit

If you want folks who have no idea what your code does poking around and changing things, more power to you. I’d rather not be in that position, or in the position of a security guy changing code and getting blamed for every future bug for the rest of that program’s life.

Reply

[-]

themastermatt@reddit

Youre right! They should not be merging code. But they should be able to discern an internet exposed system from one that isnt. They should know how to manage their application and/or risk register to prevent having to relitigate the same things every time Tenable runs a scan. They should be able to offer guidance and partnership to better understand their environments.

Reply

[-]

merlyndavis@reddit

Just because a system isn’t external doesn’t mean it isn’t a vulnerability. Insider attacks are a thing. That’s why I said fix external systems first, then internal. Leaving any vulnerabilities open is begging for trouble. Because one day, random newbie in accounting is going to click on the wrong link and open up a whole new attack vector.

Reply

[-]

themastermatt@reddit

Why does it always have to be a 1 or a 0? Any time any security guy gets the first hint of hesitation - its always straight to this strawman of not fixing it at all. You bring your scan results, ill bring diagrams and SMEs - and yeah if there is something fixable itll get done along with the 100 ops tasks. But if it is an accepted risk, write it down somewhere so we dont have to do this again.

Reply

[-]

Humpaaa@reddit

Yep, most IT responsibles are fully in favor of a model like this for exactly that reason. Security and asset owners should cooperate. No asset owner would want other people to do actual changes in the asset they manage.

Reply

[-]

DeliveryStandard4824@reddit

>Vulnerabilities that aren’t publicly accessible can be deprioritized, not ignored This is a key point right here. Just because they're not punishing available doesn't mean a threat actors can't get into the network through an endpoint and potentially exploit it. Risk evaluation is required to determine priority. How many endpoints could reach the target server or environment? The larger the number the higher the priority to fix that. Maybe firewall rules could be used to reduce the risk as a quick fix to reduce the priority of the Dev based fix. Lots to think about and review to produce the best roadmap.

Reply

[-]

Khue@reddit

> Vulnerabilities that aren’t publicly accessible can be deprioritized, not ignored. Sec team lead here with tons of experience from the Sysadmin side. The right way to go about this is to do the recon first by starting with the criticals. There are always going to be vulnerabilities identified within accompanying libraries and what the OP is looking for is something called "reachability". Criticals and highs that are *reachable* are the priority. Anything that's not reachable isn't a current priority. My suggestion to OP is that you need to do the following: 1. Request a viable time frame for recon. A lot of the findings will be redundant especially if you reuse a lot of the same code. In the recon you need to identify and prioritize 1) Critical-Reachable, 2) High-Reachable and then everything else. Realistically you should have a policy/standard that dictates the prioritization, but I doubht you guys have that right now 2. Then with your recon from step 1, apply a secondary prioritization based on risk assessment of the system. If the risk assessment of the system is critical/high and the vulnerabilitie identified is critical-reachable, these are obviously your first targets for remediation. If you have a system that's risk assessment is low, with a vuln that's critical-reachable, that's not as important. Risk assessment on the system is going to be based off business priority like if there is revenue impact to it, it would probably be a critical risk. If it's just a development system that's not exposed ot the outside world, this would be a low risk. 3. Once you have the above worked out, then you assign time frames to do updates. You slap all this together in an action plan with number of hours of work required and present it to management with a reasonable time expectation. You need to get the business appetite for the scope of work presented to you by security. Effectively, if you have a well maintained vulnerability management program, you should be doing these at some regular interval so the scope of work is relatively manageable but if this is the first time you are doing this exercise... it's going to be overwhelming. The reason why you need to do this level of work is because management most likely thinks that this will just be a few button presses and you're done, but that's clearly not the case. It will take hours out of your day to accomplish this and you need to get a greenlight on the work because if you proceed to remediate this stuff, something else has to get sidelined that you would normally be doing instead.

Reply

[-]

aenae@reddit

If it is anything like the container scanners i have seen before, it is because of system libraries that are included in the container, but not used by the app. If you for example have apt in the container, you also have a lot of python and perl libs that are needed for apt, but not used by the java app you are running in that container. If one of those libs have a vulnerability, it would be very hard to exploit during normal operations, but a scanner would pick it up and alert you because they cant know if your app calls it

Reply

[-]

TheCTRL@reddit

Very hard to exploit means it could be exploited somehow in the near future

Reply

[-]

nethack47@reddit

We got curl flagged in all the containers. Think it was CVE-2024-7264 It is a good example of things that aren't very urgent to patch when the container doesn't use curl. The containers flagged are build containers that doesn't expose anything to the outside world. Being potentially vulnerable can be a bit of a quest for perfection. If you have enough resources you can try to reach it. Most have to triage.

Reply

[-]

1z1z2x2x3c3c4v4v@reddit

Winner! Winner! Chicken Dinner!

Reply

[-]

Turbulent-Pea-8826@reddit

This is my job every week. Security sends us a vulnerability scan. Half of them are because they contain individual updates that are covered in cumulative updates. So I have to argue this. Then they don’t update the spreadsheet. Instead of running the automated scan again some dipshit just keeps sending me the same spreadsheet. So I spend more time hunting down false positives than doing actual work. But I get paid so oh well.

Reply

[-]

ddesla2@reddit

That's the worst. See if they have "show superseded updates" or similar setting enabled in whatever scan tool they use. Disable that shit immediately. Not only will it reduce the numbers dramatically, it weeds out all the cumulative noise. I always group vulns by single actions so folks can see exactly how many are fixed by doing the core os updates or updating xyz to the latest version and update level. It also reduces the amount of actions, chg reqs, etc for patching teams. Sorry you have to work with such shitty security folks.

Reply

[-]

Turbulent-Pea-8826@reddit

Oh no, no, no. The security people would never let us see what settings they have set. That would "violate the principle of least privilege." Not that I think anyone over there could explain what that actually means. Nor would they ever take advice from us on how to run their scans...or anything. There are a bunch of contractors over there who have no idea what is going on. It's fine though, keeps me busy. The 'perks' of working for a large organization.

Reply

[-]

somerandomguy101@reddit

Security Team here. Why is your team handing stuff quarterly? Our Scanning tools are built into the CI/CD Pipeline, so the devs know when new vulnerabilities pop up as they pop up. Ideally your dev team should be set up with regular scans, and have a program where vulnerability finding over (critical/high/medium) are resolved prior to release. As for triage, keep in mind that your security team may have limited visibility. They may see that code is using a vulnerable library, but not how often, if at all the vulnerable function from that library is actually called.

Reply

[-]

whtbrd@reddit

Still counts. I get it that this is overwhelming, but it all still counts. You have a vulnerable library that the scanner can see and your code never calls it? CAN it be called? Just because your application doesn't call it doesn't mean it is non-exploitable. If you don't need it, cut it. The habit of devs calling ALL the libraries just in case they might possibly need something is a terrible security practice. You end up with exactly this situation with vulnerable and expired libraries, that linger and expose the company to breaches. "Not from the internet!" You say? OK, so the only bad guys are from the internet, and they have never found a way to get an application to call a vulnerable library on an asset that is 1 or 2 hops in? Your DMZ is perfectly hardened and the apps on those servers cannot be misused? All your network security and credentials and account security and personnel security are all perfectly implemented to ensure there are no gaps that can be exploited by anyone inside or out? The devs who call all these unnecessary libraries are 100% sure to only have secure data transmission and limited administrative access to those who need it. No overly permissive firewall rules, no-one at your org who shouldn't have administrative network connectivity to these assets who might be on a Hotspot with malware on their machine. You have IPS between your internal workstations and your servers? This is 1 piece of the puzzle. Your piece. It works hand in hand with all these other pieces. You cannot expect that your security, with all these holes, is the only piece with problems. You have to patch up as many holes as possible. Other teams should be working on theirs. You have to start somewhere. Start with the highest criticality score in any DMZ. Put library removals on the devs workshops. Your lambda package that runs once a month? Disable it with a ticket to enable it at runtime until it can be corrected. Devs are ignoring security findings? Good grief. That's a mgmt issue. Get their leadership in the meetings to discuss the timeline. Leadership wants a remediation timeline by Friday? That's just a timeline, not a full remediation. But you're going to need buy-in from all the stakeholders. Devs, sys admins, security, everyone's mgmt and the leadership who wants the timeline by Friday. Have a kickoff meeting today and a working session every day until Friday analyzing the approach and breaking things down into low hanging fruit vs critical, and the people needed to be involved for each one, and the timelines for each. By Friday you should be able to have a workable plan, but mgmt MUST be involved and the leadership who is demanding the timeline NEEDS to be at the kickoff meeting to show they're supporting/driving the effort.

Reply

[-]

VA_Network_Nerd@reddit

Sorry, it seems this comment or thread has violated a sub-reddit rule and has been removed by a moderator. **Inappropriate use of, or expectation of the Community.** * Avoid low-quality posts. Make an effort to enrich the community where you can- provide details, context, opinions, etc. in your posts. * Moronic Monday & Thickheaded Thursday are available for simple questions, or other requests that don't need their own full thread. Utilize them as much as possible. ----- *If you wish to appeal this action please don't hesitate to [message the moderation team](https://www.reddit.com/message/compose?to=%2Fr%2Fsysadmin).*

Reply

[-]

thegreatcerebral@reddit

OP I hear what you are saying but you are making a lot of wrong assumptions. All the vulnerabilities are legit. It doesn't matter if you use/don't use something or if something is behind a VPN because the problem is that if someone got to system A, if they can see you have venerable package X on a system that they can then move laterally to then that is what they will do.

Reply

[-]

rrmcco04@reddit

So there were 600 new vulnerabilities added this quarter or did the scans change? I would look to get some advice on prioritizing from the security team showing a risk matrix (maybe they did?) looking at severity, ease of exploit, and application risk factors (type of data, exposure, criticality to the business, financial impacts). This is the way to manage and solve them. If they don't have one or you don't have the means to assess the application, this is a thing to work on next quarter (sounds like you will be busy this one). I would also suggest that you add some additional scans in the interim for code push or something. Remember, the security team is there to help you, not just dump work. They are showing the risk today, thank them for providing the data, get to work on classifying them and work with management to create reasonable timelines.

Reply

[-]

lightmatter501@reddit

Most attacks involve exploit chains. 300 vulnerabilities reachable from the public internet is still a lot.

Reply

[-]

Tx_Drewdad@reddit

Several issues, here. 1) Unreasonable timelines 2) No risk analysis. (High CVSS = critical, sure, but what's the RISK? Criticality plus exposure = risk) 3) Sudden large dumps of vulnerabilities. "Oh yay! We can finally scan so here's EVERYTHING we should have been watching for the last five years" is ridiculous. Security team should ALREADY have had a vulnerability program, even if they weren't scanning. 4) Sysadmins should be responsible for OS-level patching and infrastructure tools. Code libraries should be the responsibility of the app teams.

Reply

[-]

calladc@reddit

You're not concerned about lateral movement for these libraries you've deployed to your environment?

Reply

[-]

PlannedObsolescence_@reddit

This is clearly written by an LLM, OP is a karma-farming bot look at their profile.

Reply

[-]

laz10@reddit

Sounds like your security team knows how to run a scanner and not much else

Reply

[-]

Ihaveasmallwang@reddit

It’s not their job to remediate. It’s OP’s job and OP seems to want to blame others rather than doing it.

Reply

[-]

CAMx264x@reddit

Reminds me when I had to do STIGs for the DoD/VA. The first week we got hit with 3000 vulnerabilities as one of their tools didn’t look at the .d directories and produced almost all of those vulnerabilities as false positives. Leidos required a manual screenshot be taken for each vulnerability and a small writeup on why it was a false positive.

Reply

[-]

syberghost@reddit

If half are reachable from the internet, all are reachable from the internet. Your bedroom isn't reachable from the front porch until you open the door.

Reply

[-]

Nonaveragemonkey@reddit

To be realistic, if a single one is reachable from the net.. generally everything is reachable.

Reply

[-]

johnfkngzoidberg@reddit

You have sloppy code that loads libraries and doesn’t actually call it. Dude. Security is done in layers, because eventually something will get exploited even if it’s not easily exploitable now. The environment shifts, the code changes, and people forget about something that’s not run often. It’s still dangerous and your team gets brownie points for bragging about the 600 vulns you patched. Patch the criticals, then highs, and so forth like the security team says. “Devs are ignoring security findings now because we've cried wolf too many times. Then we miss actual issues buried under the noise.” Oh you know security better than the team of professionals that do it for a living? Patch things just like the infosec team asked you to.

Reply

[-]

Small_Golf_8330@reddit

This is always a point of contention everywhere I’ve worked. Those lines are really hard to define sometimes. My opinion is that there is no one size fits all for every organization. The best is when you have security staff who have solid knowledge of the product, and administrators/dev who care about security. Often times, people hyper focus on what they consider their “role” which leads to finger pointing and not solutions. When everyone starts looking at the whole product as their responsibility and they work together to solve problems it’s the best. That magic balance rarely happens though unfortunately.

Reply

[-]

EuphoricAbigail@reddit

Is your security team technical or compliance focused? Let me preface this by saying I'm an engineer so an infosec person may have their own valid analysis. In my experience, there are really two very different types of security teams. There are teams that actually diagnose problems. They work to understand what is going on and work with you to resolve real issues. They will dig into alerts, figure out root causes, and collaborate with other teams to fix things. The other type are focused on certifications and audits (ISO 27001, SOC 2, etc). They might run their own scans or hire a third party, but often they don’t really understand the results. They highlight an issue and consider their job done, leaving the rest to other teams. Technical security teams have been a dream for me to work with as a sysadmin, the compliance variety not so much. Its really a company cultural thing, is the core aim of the security function to protect the company and secure against threats or is it to tick the boxes on the paperwork that makes clients happy.

Reply

[-]

Klop152@reddit

Having been both a sys admin/engineer and now a security engineer the last few years, boils my blood seeing compliance focused security teams delivering these results with little to no context or guidance. Sometimes not even understanding what they’re asking to be fixed.

Reply

[-]

klipz77@reddit

Spot on. In my experience the “compliance” edition of these sorts could easily be replaced by an automated Nessus scanner (which is probably what they are running anyway) and are worth close to zero.

Reply

[-]

broknbottle@reddit

This. The ones I deal with on regular are the compliance turds that focus on pushing half baked policies and delegating scanning to some other tool and expecting everyone else to do all the actual work.

Reply

[-]

bageloid@reddit

Unused code is only unused until it isn't. If the libraries aren't being used, just remove them from the builds. Quick and easy way to knock out the majority of the criticals. >Scanner sees vulnerable package but has no idea if the code even executes or if anyone hits that endpoint. If it doesn't execute and if no one hits that endpoint, then it should be removed. >Staging environments with test data treated same as production. Unless you have complete segregation between staging and prod, it's still an big issue.

Reply

[-]

danekan@reddit

Same with batch jobs that just 'quickly run and don't receive input' .. well what happens when the code that gets loaded to batch is malicious? It will run as long as it needs and do whatever it wants, not what your original job wanted.

Reply

[-]

bitslammer@reddit

IMO it sounds like your org has a somewhat ad-hoc process for Vulnerability Management since this seems to have hit you as somewhat of a surprise. If that's the case then that's the core issue. In my org we have a very well documented and formal process for vulnerability management including patching/remediation. That was developed with members from the corporate risk management team, compliance team, infosec team, and all of the affected IT operations teams. Those responsibilities are included in everyone's core job descriptions and are considered as such so this work isn't dumped on people out of the blue as a drop everything fiasco. There are well understood SLAs for how remediation needs to be done as well as clear processes for granting extensions to the SLAs as well as granting exceptions.

Reply

[-]

reegz@reddit

If you’re doing it right there should be very few “drop everything” types of vuls.

Reply

[-]

bitslammer@reddit

Agreed. Our last crazy one was the Log4J fiasco and after the dust settled it as clear that wasn't quite as bad as advertised for us, but in that first 12hrs when things weren't all that clear nobody wanted to be the one holding the bag for saying not to patch and we figured it was good practice to see how well we were able to deal with one of those situations.

Reply

[-]

chaosmonkey@reddit

That's what we had at my last org as well, however it took us about 5 years to get there. I put a lot of effort into shaping our end of things to meet the needs of security as well as user impacts, lots of iterations improving on each other. Then they replaced the vulnerability management people in the security department and the new guys try making up new goals/SLA out of nowhere, it was easy for my boss to tell them to pound sand because our stuff was so well established, and none of the changes were from any leadership level, just two new young guns looking to run roughshod over a few other departments. It's been a minute since I was there, but did catch up with my old boss the other day, apparently those 2 vuln mgmt guys are still doing the same thing with the same results.

Reply

[-]

bitslammer@reddit

>Then they replaced the vulnerability management people in the security department and the new guys try making up new goals/SLA out of nowhere, Yet again another horrible organizational issue. All of the stakeholders have a voice in the SLA process, but the largest factor in that comes from the various regulatory requirements we face operations in a little over 50 countries. I've worked for both a major MSSP and Tenable and was really shocked at how poorly a lot of orgs, even big well known names, treated Vulnerability Management. It often seemed like some afterthought or add on duty. I never understood how patching wasn't seen as important as having well functioning AV/EDR or backups.

Reply

[-]

disbound@reddit

It’s your job to figure out what needs prioritizing. From my experience security teams are just button pushers who just repeat whatever security tool they’re using.

Reply

[-]

danfirst@reddit

A lot of us security folks were also sysadmins or devs longer than most of the people in this sub. We also can't know the inner workings of every team's custom app all at the same time either and work with those teams to try to figure it out.

Reply

[-]

brixton_@reddit

It's such a shame that's been your experience. Thoughtful, prioritized vulnerability reporting by security teams should be the norm.

Reply

[-]

New-fone_Who-Dis@reddit

>Thoughtful, prioritized vulnerability reporting by security teams should be the norm. Their tools pull in the "score" vulnerabilities have been given/assigned....that generally determines prioritisation. However, just like OP has said, there are cases where it isnt being used at the moment, such as libraries imported. However, security aren't devs, they dont attend your stand ups, they dont perform code reviews on your work. Have a piece of code or script somewhere that imports a library, doesnt use it, and is giving off about security not knowing what they are talking about? Its not their job to audit your code, just fix it. If you don't feel like fixing it, then own the risk on the risk register that says why you aren't fixing it.

Reply

[-]

Noobmode@reddit

Damn you must work with some shit people. Prioritization is one of the top things you need to do.

Reply

[-]

Own-Trainer-6996@reddit

Usually comes from management thst doesn’t listen. “Security said this needs to be fixed ASAP!!!”

Reply

[-]

Noobmode@reddit

Cool if you are a security professional, spoiler I am, you push back and use asset criticality, exposure, etc to set prioritization and let the teams patching know. Your job is to help them make data driven decisions, not be a yes man.

Reply

[-]

Zerafiall@reddit

Yep. On the security side, most of our priorities are from management FUD. It’s rare you see a team that has the freedom to run security sanely and from best practices.

Reply

[-]

GoogleDrummer@reddit

Meat based alert forwarders.

Reply

[-]

HopingillWin@reddit

Sometimes it's the tools, I've seen for example a very high profile tool complaining about default IPv6 settings on a Linux box where the IPv6 stack was disabled via kernel params at boot time. The button mashers don't understand that the report is nonsensical.

Reply

[-]

Own-Trainer-6996@reddit

Button pushers is a good one.

Reply

[-]

jobohomeskillet@reddit

I like computer toucher too

Reply

[-]

Ihaveasmallwang@reddit

Why should you need to be told by security how to prioritize? Presumably, you as a sysadmin know these systems better than they do. You shouldn’t need your hand held. Devs ignoring security findings should never be a thing. Don’t you have some sort of system to scan code before it’s released? Nothing should go to production if it has that many vulnerabilities, ever. That is a culture that shouldn’t exist at any company. Sure, they may get annoyed that they have to fix their broken code, but that’s literally part of the job description. Escalate that to higher management if it continues to be a thing. Why shouldn’t staging environments be treated the same? They are still systems on your network. Here’s how you could prioritize it. Is it critical and on an external facing system? Do this first. Is it critical on an internal system? Do this next. Is it high on an external system? Do this. Is it high on an internal system? Do this. This really isn’t rocket science. That attitude of “it might not even be executed” is a horrible attitude to have. If it’s not executed, get rid of it. There’s no reason to have it around. At my company, security and engineering do not hate each other because we actually work together instead of engineering trying to blame security or act like they are incompetent. In the end, you’re all working towards the same goal.

Reply

[-]

danekan@reddit

Not being reachable from the Internet isn't all that matters, where do these backend services get their input from?? Were those paths public or from customers?

Reply

[-]

Longjumping_Gap_9325@reddit

"Most criticals were libraries we import but never actually call" As others said, then trim that code out. It's not used until some exploit finds a way to use it. Why have potential extra code even loaded if you're not using it? Trim the fat and make a more efficient application. "half werent even reachable from internet" I despise this line of thinking (unless literally you're saying it's not accessible as it's air gapped or otherwise already isolated in other way). East-West is your concern. A SQL server may not be accessible from the internet, but a flaw in a Web App on a front end web server could provide that foot in for the traffic to now be East-West, and boom.. your data is now being exfiltrated. Thinking something needs to be directly exposed to the internet to be exploited is overlooking very real potential security concerns. It'd be like thinking a bank vault doesn't need a secure and solid door because it's not reachable from the exterior of the building, but all it takes it picking the front door lock or breaking the door to then have that "East-West" access to the vault.

Reply

[-]

Heavy_Dirt_3453@reddit

It's also then open to exploitation if someones account is compromised or something else in the environment is popped. Thinking that something not being directly exposed to the Internet being safe just on that alone is asking to be hit.

Reply

[-]

delti90@reddit

I'm in infosec and run vuln management and AppSec. Prioritization of these should have been done by your security team. It's also possible that these need remediation due to compliance if your company is under any sort of regulations (nis2 is the current huge pain in the ass). Even if they aren't reachable many compliance frameworks require no crits or highs, full stop.

Reply

[-]

jxd1234@reddit

Has your security team recently started doing vulnerability reports or this this happen every month? If this is a new thing from the security team then yeah you're going to get a lot that have been flagged. They should be able to help prioritise them though. Clean up unused libraries in your code, automate patching servers/endpoints and use something like dependabot for updating dependencies. This will get rid of a huge amount of vulns Although you've got many that aren't exposed to the internet, you've got a lot that are and the ones behind your VPN still need to be addressed. If your VPN gets compromised or your network is compromised in another way you're making lateral movement much easier. You need to have a defence in depth approach to security.

Reply

[-]

CluelessPentester@reddit

Your whole org sounds like shit. Why doesn't security prioritize? It's literally their job to make a risk assessment. Why do you even have hundreds of unused, vulnerable libraries? Also, just because something is not reachable externally doesn't mean you can just ignore it. All it takes is just one successfully phising mail to go from "This isn't reachable from external" to "Oopsie, our whole domain just got encrypted." Your company needs to fix its processes and needs clear policies for risk assessment, remediation strategies, and first and foremost responsibilities and accountability.

Reply

[-]

Either-Cheesecake-81@reddit

Last week my security team send me an urgent alert to fix a script that had a password in plain text IMMEDIATELY. They also demanded an immediate plan about how we were going to go through the rest of our scripts to ensure no other scripts contained passwords in clear text. It was their server and their script. They just said, “our bad.” I asked for a plan about what they were gonna do to go through the rest of their scripts to make sure no other scripts contained passwords in clear text. They told me to fu(k off. I really don’t have patience for “security teams” anymore.

Reply

[-]

blavelmumplings@reddit

Just because a vuln isn't exploitable from the Internet doesn't mean it's not a vulnerability. Risk = likelihood x impact See what's most risky for your environment and deprioritize the rest. See if you can fix all critical and high by Friday and the less critical ones after these. That's what we'd do.

Reply

[-]

LBishop28@reddit

As someone currently in security and someone whose background is as a sysadmin, it doesn’t matter. Those vulnerabilities become accessible in the event that a bad actor gains access into your network.

Reply

[-]

feldrim@reddit

It look like your organization does not have a policy or procedure to handle these at all. Then, make it work for you. Build an SLA for patching accounting for for risk assessment. Something simple like this: - Critical: 5 business days for remediation plan, 15 calendar days for mitigation - High : 5 business days for remediation plan, 30 calendar days for mitigation - Medium: 15 business days for remediation plan, 60 calendar days for mitigation - Low: 15 business days for remediation plan, 90 calendar days for mitigation - Informational and others: Best effort Update the numbers however you want. And then, have your own assessment. The CVSS may be high but it depends if it is publicly accessible, internal access limited via segregation, or the impact is minimal in case of a successful attack because asset is not business critical, etc. Use your assessed severity levels instead of the CVSS score-based severity. You can accept some of the risks if you have justification. You can mark some as false positives. For instance some scans rely on version numbers, and RHEL backports fixes without increasing the version number. Finally, provide the leadership your plan. This is a thorough plan that leadership would accept. You have numbers and a proper assessment. Due to the high number of findings, notify the leadership that it is hard to reach the SLAs you created this time, but you can ensure that you can meet them on following quarters.

Reply

[-]

sw4gyJ0hnson@reddit

I’m an IT-Sec Analyst that Manages vulnerabilities. Our infra-Team wont fix any vulnerability if I don’t spoon feed them every Information and remediation method available. And even then they breach every possible Deadline

Reply

[-]

Mountain-eagle-xray@reddit

Look up the vuln by CVE. Does it have an assigned KEV? This is priority 1. What is the vector? Local or remote? Remote is priority. Is it privileged escalation? Priority.

Reply

[-]

sociablezealot@reddit

“quarterly scan”. Jesus. No one is waiting months to exploit you. Automate your patching, use distroless images. This should happen daily.

Reply

[-]

UnkleRinkus@reddit

Jesus fuck. You are in a minor level of hell, where you are being cursed by people that can barely read the holy script they want to chant at you. I'd create a matrix of: \-- visibility to malicious parties, particularly the path (as you note), \-- the amount time of availability of that exposure (your lambda), and \--the sensitivity of resources that the vulnerability can obviously access. Don't overthink this, you're just creating a sort key to prioritise the items Present that back to security team and management as your order to address these. Security will predictably say they all must be addressed, just ask them to justify any change to the --order-- you work the items. If Management squawks, ask them what other work they want to delay to increase velocity on the list. Then just work the list as time is available, and don't let the idiocy drive your life.

Reply

[-]

TickleMyBurger@reddit

Priority is any externally exposed KEV (known exploitable vulnerability), after that its internal KEVs, after that is critical cve without kev on a best effort basis. Your security team should’ve giving you the kev list - this is part of every vulnerability scanner worth a dime. If you have deprecated libraries that aren’t resident in memory from being called, clean up your backyard - that’s on you. There’s some tools out there that can scan for memory resident vulns/obsolescence but it’s not perfect (to your point about the monthly execution example). That will take time and start paying attention to the technical debt you’re adding by being lazy on new release.

Reply

[-]

Tcrownclown@reddit

If the security team is internal, not a 3rd party SOC etc.. your team sucks

Reply

[-]

TuxAndrew@reddit

We worked pretty hand in hand with our Security team in resolving issues, a lot of the questions you're asking the System Admin team should know the answers to. Every vulnerability should have a QID associated with it along with instructions on how to remedy it, if your security team isn't providing that information than you should be hounding them for that information since it's a basic functionality of every scanner.

Reply

[-]

maziarczykk@reddit

Got to love these security bureaucrats. They own nothing, help nothing, only create noise... \> Infosec ran their quarterly scan and dropped 600 vulnerabilities on us. 200 marked critical. Leadership wants remediation timeline by Friday. Tell em you can mitigate one every week, maybe two or ask for entire team to help.

Reply

[-]

enforce1@reddit

Do what you can, and then wait for next month. Thats what I do, make measurable progress. Also half of my errors are usually unpatched insecure ciphers so one good push clears out a pile of them.

Reply