Ransomware PTSD
Posted by Sultans-Of-IT@reddit | sysadmin | View on Reddit | 155 comments
One of my clients got ransomware a few months back. It ended up being ok as we were able to recover from a backup.
The problem is that now, every time something "weird" happens, I immediately get an adrenaline spike and think the worst. This can't be healthy. Has anyone else ever dealt with this?
no_regerts_bob@reddit
Did you do a full incident investigation and audit? Make changes to address the way this happened so it can't happen again? I usually feel better about a client's network after an event because they typically improve their systems and processes.
Sultans-Of-IT@reddit (OP)
That client had a full audit with a leading Cyber Security investigation firm in the US. With their help, we figured out exactly what happened and how to prevent it. I will tell you they exploited Veeam and VPN. I'm not even worried about that client as they paid to implement everything under the sun to protect themselves.
I'm not even paranoid of their office anymore. It's smaller clients who refuse to pay for things I've grown to feel are a requirement to stay safe now. Not to mention, that was one of our few clients who actually has cyber insurance.
no_regerts_bob@reddit
One of the best things the MSP I'm working with now did was to fire bad clients. Don't want to spend for required security and maintenance? Bye. Don't want to pay for cyber insurance? Bye.
At first they did lose some revenue, but they picked up new work (from better clients) very quickly and a couple years later everything is better. Techs are happier, clients are happier, revenue is better.
Sultans-Of-IT@reddit (OP)
I wish we could do that. I really do.
wazza_the_rockdog@reddit
Maybe approach it a different way, their risk is higher or their recovery is harder because of choices they're making against your recommendations, so if they get ransomwared then any recovery efforts are out of contract and paid at your hourly rates if the plan is an AYCE deal, or at higher rates if it's already hourly. If they don't have the recommended protection for ransomware then any ransomware related jobs for them are treated as normal jobs not emergency jobs, and all recovery efforts will be done in normal business hours.
A lot of companies (and people) likely take these risks because they expect they will get bailed out if worst comes to worst, they quite likely think that the worst that will happen is it will be more effort for your team to resolve, so they don't care to make your job easier.
meesterdg@reddit
Understand that while you're allowed to feel empathy for them, if they get compromised it's not on you. Your job is to take care of their systems, and in the event of a compromise respond to it. You can take steps to make compromise less likely but you can't guarantee it won't happen so that means you might as well reorient your perspective to focus on network hardening and high quality and efficient incident response planning.
itishowitisanditbad@reddit
Yeah, its business.
I've always been surprised when MSPs appear to cling to bad business.
Nah, they tight with money? Fuck that shit, go find a better business to support.
I did small business support for some years and would somewhat often go to a client that i'd refer to a MSP OR had just been dumped by their MSP... because they refuse to spend any money and just think cutting a single cheque a month means 'all issues are taken care of' when it doesn't.
"Sometimes the printer needs ink and paper and rollers and shit dude, its not one-and-done in IT" - Daily phrase for a while. I found its the easiest thing to explain that sometimes IT things need more and sometimes less.
wazza_the_rockdog@reddit
Don't care about someone elses business more than they care about it. If they won't pay for necessary protections despite being advised of them and the potential impact of not having them, they're the ones who have decided that the impact is acceptable.
I don't know if you own/run the MSP or you're an employee, but if you're an employee then also remember that you have an additional level of not caring in between as well, because any impact to the MSPs business also won't be on you. It may not even impact the MSP very much, even if absolute worst case scenario happens and the client goes bust, doesn't pay bills etc - as long as they're not a whale client then the truth of the matter is their failure shouldn't significantly impact the MSP.
Windows95GOAT@reddit
Been there. It's really gave me a lot of stress for what felt like counting down to the day i would get the ticket.
I will say, for as much crap i give the MS cloud. It being somewhat resistant to ransomware makes me sleep at night.
uptimefordays@reddit
You can’t worry about clients who refuse risk mitigating efforts.
zakabog@reddit
Make them sign a waiver, document that they are going against your recommended solution and you are not responsible for their data loss in case of an incident. They'll be angry when something happens, just remind them that they refused to implement proper backups and security procedures to prevent such an attack from happening.
TheWino@reddit
Yes. It’s been 3 years and I’m super paranoid. It’s not healthy.
reard3n@reddit
Same. But 5-6 years. Never goes away.
ceantuco@reddit
10 years here lol
Fallingdamage@reddit
Experienced a few working at an MSP in 2009-2014. Ever since then anytime I see a sudden spike in network sessions I get anxious.
ceantuco@reddit
hahah same here... or when I see servers taking long to reboot too.
ensposito@reddit
Anyone think that it will go away once we retire?
wazza_the_rockdog@reddit
Nope, you'll farm goats in your retirement and worry if one of them takes too long to come when you call them!
ceantuco@reddit
I hope so lol
OutsideLookin@reddit
I’m at five. Same.
moldyjellybean@reddit
It happens. I was prepared when we got new servers, I just tested the restore of all vms via offline backup. Then I got an idea of how much time, in which order to restore the vms for best results etc.
We had SAN snapshots, veeam backups to local storage, cloud, tape backups. I had probably 4 different options and tested each so I slept easy knowing I could restore, I did, and it was a great thing being the hero when everything was down and I brought it up.
TheWino@reddit
Yes me too. We didn’t need payout and now we have more monies for security. Still paranoid though. 😂
guardianz@reddit
I worked at an MSP and for awhile there was at least one hotel a week that got ransomwared. Some had back ups some didn’t. Was never fun but it just became like any other issue after like the 40th one. Then I got a new job and it happened to us and I was probably the most relaxed about it or at least the least panicked compared to my coworkers. I still worry about it but I’ve recovered from it so many times now.
Isord@reddit
I often find big problems less stressful I think. I know in many cases the big problems seem so intractable that you'll get kudos for solving it at all, vs with little stuff you are expected to nail it perfectly.
All the better if it's so big that it is out of my pay grade. Like when I arrived at a shared data center that was on fire. I texted my coworker a picture and said "no longer our problem."
Igot1forya@reddit
The epitome of "this is fine" in the burning house.
guardianz@reddit
It’s 2024. The house is fully on fire.
Sultans-Of-IT@reddit (OP)
CBT, literally you were exposed to it enough you learned to overcome it lol
post4u@reddit
3.5 for us. Still paranoid for sure. I think about it often. We're a fairly large organization and ours was pretty major. We survived, but it was the worst week or two of my 25 year career in IT by far and the legal gifts keep on giving months or years after. Hope it's a once in a career experience.
GoodVibrations77@reddit
have you considered seeking professional help ?
TheWino@reddit
Scotch and tequila. Not really I think I’ll just need another career.
Ok_Series_4580@reddit
Ditto.
Brilliant-Advisor958@reddit
I've haven't had an incident and I'm still super paranoid. I know it's just a matter of time.
Sultans-Of-IT@reddit (OP)
Well glad im not the only one LOL
thewalletisempty@reddit
Yeah we ended up pushing Datto unit's as I was super stressed. Even after a good outcome.
dmuppet@reddit
It's the reason I refuse to silo myself into cyber security. Barely any praise or recognition when everything is fine but the second you get breached from a zero day or most commonly a user getting phished you are the first one thrown under the bus.
Healthy-Poetry6415@reddit
I deal with them all the time.
I have no remorse for the people anymore. Its always outdated shit they quit paying attention to despite being told 100x they should get this issue fixed.
If 500 bucks is a struggle for your business. Enjoy your 500k ransom and the complete smearing of your company image.
I am sick of propping up trash with my blood sweat and tears. Especially the ones that do nothing after the fact to strengthen the environment
xtheory@reddit
My company got hit HARD by NotPetya in 2017. The only thing that saved this multi-national company was a DC in Ghana that happened to be offline during the attack thanks to a poor internet connection. It's gonna take time, but this is your chance to not let a good crisis go to waste and implement best practices and a local or remotely monitored SOC that will automate alerts for any IOCs. While your company had backups, there's always a chance that the threat actor has persistence on the network. I had a colleague that found out the malware had hidden a copy of itself on a IP camera that has open FTP and Telnet access with plenty of storage to keep a payload. Nessus/OpenVAS your network to find vulnerabilities, get patch management under control, and have a 3rd party pentester come in and check your resilience.
Your peace of mind will return and nerves will calm, but expect a heightened level of paranoia for awhile and avoid developing any drinking problems.
GeneMoody-Action1@reddit
Man I did this in a capture the flag exercise once, hid in a printer, drove them mad how I was getting control back.
When you are thinking "what do I reload clean?", along with servers and workstations, think phones, printers, switches, industrial control systems (Even if you think you do not have them, think climate control, lighting control, etc), cameras, APs, televisions, Anything that could have been connected to by any means. Overkill, you betcha, because you want to know that the last piece of bacteria is exterminated so infection is no longer possible.
And passive tap your egress for IDS, you can fool a lot of things, but packets are packets, if they are communicating off prem, it has to be through that wire. Even if you think you have zero budget, go grab SecurityOnion, a throwing star LAN tap, or a small managed switch if your internet exceeds 100M. Disable IP, config via serial, one cannot hack what one cannot get to.
And I have 100% seen people that do not check backups get nailed by someone slowly corrupting backups...
shifty1016@reddit
I know the feeling, but I don't get it so strongly.
However, even if I JUST tested our backups 12 minutes ago, I always get the sweats when someone asks for a file to be restored, or better yet, a whole server. No matter how good I know Datto is, no matter how often I do manual tests, I always get nervous.
CPAtech@reddit
I had PTSD from Hafnium. It finally faded, but we also had some significant power related issues after an overnight storm and I woke up one morning at 5:00 AM to hundreds of server alarms and temperature alerts. Had to race to the office and deal with the fallout as our UPS software failed to do its thing.
I now have a moment of panic every morning when I wake up as I reach for my phone to see what happened overnight.
jordanl171@reddit
I'm with you on the power thing. Not fun. I've been there. Smaller scale.
CPAtech@reddit
Power and cooling is the bane of my existence.
jordanl171@reddit
power gods messing with me... this morning one of my main UPSes failed battery test, bad battery. new one ordered, .. and we've had ZERO bad weather in the last month plus... except for tonight, some wind. good stuff. I'll be fine.
Sultans-Of-IT@reddit (OP)
Crap, that's not good, man. This is too stressful of an industry. I've always wanted to work with computers since I was 14. My 14 year old self would have said screw that, I'm not a security guard.
CPAtech@reddit
If I had known at the beginning of my career that everything would end up so security focused with nation states attacking businesses on a regular basis I probably would have gone a different direction too.
JacksGallbladder@reddit
Its super common for employers to provide mental health resources after a cyber attack. You're not the only one.
I highly encourage you to check in with a professional. It helped me.
Waving-Kodiak@reddit
Yeah I had it too.
First 36 hrs of non-stop anxeity-filled work (just ate some nuts and drank water) and then I went home and slept to wake up to that one domain controller took a dump.
Fortunately it seems that the DC issue was unrelated and I could manually remove the broken one and spin up a new one.
But since then when I we got an alert or indication on something severe I could feel cold as all blood rushed to my stomach.
Lando_uk@reddit
I used to get this all the time, it's a miserable feeling. Now we have a proper secops team and have invested in expensive protections, it's certainly better as its no longer my problem.
Frankaintmyfriend@reddit
It's been 8 years for me and I still panic everytime someone says "I can't connect to the printer" That was the first email we got when the ransomeware started. It was the worst 72 hours of my entire career. I will never get over it.
garcher00@reddit
It took me 3 years to accept that the attack was not my fault.
Caranesus@reddit
I got such experience about 5 years ago, and we started looking for immutable storage so our backups could not be encrypted. Luckily, Veeam created the hardened repository, so we deployed the one from Starwind and can now sleep better: https://www.starwindsoftware.com/blog/starwind-vsan-as-hardened-repository-for-veeam-backup-and-replication/
Sultans-Of-IT@reddit (OP)
We've implemented Immutable everywhere that wanted to pay for it.
Caranesus@reddit
Oh yeah, after a ransomware attack, you want to implement immutability wherever it's possible.
HJForsythe@reddit
Yeah ours happened in 2017 and it changed me for the rest of my life.
Sultans-Of-IT@reddit (OP)
Im sorry :(
HJForsythe@reddit
Its alright. Now I just want to know who it was but the Internet/krebs never figured it out.
TehZiiM@reddit
Does it really matter to know if it’s a group from Russia, china, India or North Korea?
HJForsythe@reddit
Its just like closure. you know.
vlycop@reddit
I had a "cops virus" on my nom laptop as a kid, don't know if this is the right English name but you know the one who replaced explorer.exe with something saying they are the FBI or whatever.
It's been what, 18y? I still have some nightmare where i get this virus... And I've been off of windows for 10y now
chefkoch_@reddit
Just use crowdstrike, to give you another thing to worry about.
ScotchyRocks@reddit
The silver lining: A LOT of people got to do a dry run on how to recover from something just about as catastrophic as a ransomware attack.
They had the outages, downtime, and lack of access. All while having the benefit of many others around the globe troubleshooting the same issue at the same time. Less worry about data exfiltration WHILE it was happening. And the actors responsible were trying to help fix it rather than extort the clients.
As awful as it was to go through; I really hope many did some self reflection after the fact, and shored up their short comings regarding recovery procedures.
Windows95GOAT@reddit
lmao
BigChubs1@reddit
Funny enough, cortex xdr just released pretty much a beta testing setting for the agent if you have a testing group. I kinda giggled at this because they learned from someone else mistaken. Which I had a testing group anyways and delayed everyone else. So made the switch easy.
Sultans-Of-IT@reddit (OP)
Do you think Crowdstrike is something I should be worrying about?
thephotonx@reddit
Think they're referring to the incident earlier this year.
But in all seriousness, managed CS is the only thing that got me sleeping again at night.
Sultans-Of-IT@reddit (OP)
Oh, I actually forgot about that. We have CS complete and knowing they are managing it makes me feel 100 times better.
GeeGeeMachine@reddit
Nice try, CrowdStrike
Sultans-Of-IT@reddit (OP)
Walking around to a bunch of computers to fix that issue was actually kind of a fun day for me. Its nice when you get to blame a vendor and then the rest of the world is also down. They just looked at it like, oh well if airplanes are grounded over this our IT staff isn't to blame. After finding out how simple the resolution was who cares. After that stock fell you best bet they implemented something to fix that. Not to mention webroot F'd our pcs up more times than CS did.
autogyrophilia@reddit
After it finally happened to us, I not only recovered the encrypted volumes by hand (workstations unaffected thank god), but got permission to replicate our whole environment taking advantage of ZFS on a cyclic way (it takes a complete snapshot and sends it cyclically) the result being a few minutes of data loss.
I even showcased how I could online the whole environment from total loss of data in less than one hour.
Not that you would want to do that because It is pretty important to figure out how you got infected before.
Sultans-Of-IT@reddit (OP)
I always hear horror stories of ZFS getting toasted from a simple power outage. Always scared me to put it into production.
autogyrophilia@reddit
That's kinda impossible to do. Where do you hear said stories
ProfessionalBee4758@reddit
get a SOC. it helps you to get some rest.
DesperateGenius@reddit
Yes, OP get RocketCyber or a similar solution. Can help get some rest by providing continuous monitoring and threat detection
Kwuahh@reddit
Are you a Kaseya shill? Your whole post history is peddling Kaseya products.
ElasticSkyx01@reddit
Not if they tell the client hours after an event and expect the client to do an investigation, etc. The Mob should be in the IT security business. It's all a fraud.
ProfessionalBee4758@reddit
a SOC should be able to donot by themselves
networkn@reddit
I really feel this. I am sorry you are finding it tough, I did too. You have my sympathy and understanding. You are not alone. Understand you can't control everything, educate yourself reasonably, take reasonable,basic steps to secure your environments and esp your backups, and understand that if customers are protected by insurance, they will have a path for you to follow. I bought myself a LOT of reassurance by having a response plan, deploying an XDR solution that has the job of isolating suspicious behavior 24/7, having decent antispam product, and insisting on MFA. Any client who didn't want to pay for those things were sacked, I lost 1 client, who was taken 4 months later in a totally preventable way. We now also require 365 accounts to be monitored and that costs less than $3 a mailbox so even breaches there are locked down within 15 minutes. I sleep well now. Anxiety is caused by feeling you won't cope or can't cope due to a lack of resources. Putting your mind at ease, comes by stopping those issues at source. Know it will probably happen again but with those measures in place, the damage will likely be limited and help will be close to follow.
I am a stranger, but I'd love to help you if I can. If you think a chat would help, to know you aren't alone, you are most welcome to DM and we can have a chat or two and see what we can do together to provide you some reassurance. I am not selling anything, just happy to share some experience and some advice if youd like it, or just to hear your concerns.
You aren't alone. We have got your back.
InitiativeAgile1875@reddit
Join us over at r/shittysysadmin you'll feel better.
I once had several hundred clients get ransomware and made global news.
(It wasn't my fault lmao)
Sultans-Of-IT@reddit (OP)
Kaseya? lol
InitiativeAgile1875@reddit
I'm pleading the 5th lol, if I give away too many details people might figure out who I am.
Even thinking about it gives me flashbacks. The CTO almost ended himself on the roof of our building, some of the helpdesk quit working in tech altogether.
Fucked me up for a minute but after the dust settled our competition was offering me a job so, fuck it we ball.
gistexan@reddit
It happened to my company in 2015. Had backups, but the impatient partners wanted things to happen faster, the back up recover took a long time. We had a 5 day window to pay the ransom. We had to jump through hoops to by bitcoins, they were cheap back then. Paid ransom and got the key. The real kick in the balls was after we paid the ransom, the FBI raided the Exchange we used. We lost 6 bitcoin, confiscated by the FBI, we could have gotten it back but you would have to go to court and file a suit to prove you weren't laundering Money. The nightmares never stopped until I left. I'm a florist now.
BoltActionRifleman@reddit
Wow that sounds like quite a shit show! I love the happy ending of being a florist though, that sounds so relaxing compared to the daily IT grind.
Sultans-Of-IT@reddit (OP)
I like flowers. Good for you!
Savings_Art5944@reddit
Yep. Managing On-Prem exchange servers and felt dread every day because they could be perfect and still a user inside the network breaks things. Or MS update would do the same..
BoltActionRifleman@reddit
I walked into a situation about 5 years ago where the Exchange 2013 server was way behind on updates/CUs because it was “too risky” because it might not come back up. I spend a couple of weeks trudging through updates and fixes, even enlisted an MSP through some of it. Eventually got it running like a champ and shortly thereafter installed a fresh 2019. Now migrating to 365. I can see why even though 365 has some issues, people are overwhelmingly happy with it.
Sultans-Of-IT@reddit (OP)
Every customer that was still on had prem exchange. I was forced to go to exchange online.
Commercial_Lynx2455@reddit
October 20, 2020. They got into EVERYTHING. Production servers, backups. Fortunately we did have offline copies of our VMs in our DR and we had some dirty copies of our SQL DBs so we had something to work with. Took about 2 weeks to get pretty much everything running. I did have some PTSD for about a year. Running on the Crowdstrike complete does give you a lot of peace of mind. I tell everyone at work that I have one more ransomware recovery in me. I’ll stick around for the recovery, but that will essentially be my 2 weeks notice. We have put a ton of security in place and changed a lot of processes so hopefully we’re pretty well covered.
Ark161@reddit
That is the fun part about having an infosec background, you know where all the skeletons are. Though I digress. Best thing to do is education campaigns, backups, and prep for when it happens again, because Johnny sonofabitch in finance is going to inevitably cause the company to get popped again. Do permission audits, service accounts, limit mobility between role access where you can.
Sultans-Of-IT@reddit (OP)
When billion dollar companies still get ransomed with teams of infosec people, it doesn't give me much hope.
Haloid1177@reddit
I’ve been through two of them, first one was hell on earth, second one we had taken appropriate actions, created a playbook, felt we were in a good place, and I was able to basically be an incident response manager for a day. If you still feel uneasy, my advice is think about what parts of the environment make you uneasy, and work on them. It’s always scary, but if you have everything you can have in place, it is what it is.
Virtual-Valuable4504@reddit
I was hacked really badly once. My best advice is to just know that many, many weird things happen with computers. You don't have time to investigate. You should prioritize investigating how someone was able to intrude into your network to install ransomware. Then you harden against that. You should also focus on quick solutions on how to harden all of the devices you are responsible for.
ez_doge_lol@reddit
Such is life, I get that when I hear water in heating systems and drains and fish tanks, all from my property management days... One year, on my birthday, 250 gallons of water flooded down 3 stories because of a rando busted pipe and the only person home lived in the basement level lol
Rantsu@reddit
It never went away. I gave up on IT 5 years ago. My last IT job, I was control over the whole network. Everything was as locked down and backed up thrice with offline copies. They never got it, but previous clients in previous jobs did. The specter of ransomware/crypto viruses in addition to a previous position that crept into a 24/7 nightmare broke me. I ran away to the hills.
-SPOF@reddit
It’s a constant battle between the sword and the shield - protection improves, yet ransomware attackers continually find new loopholes.
mixduptransistor@reddit
Yes, I got hit on day 5 of a job as a solo guy at a company and that was, oh, 9 years ago now and to this day with immutable backups and a small team working with me, I still get in knots when a critical outage happens assuming immediately it's ransomware
-SPOF@reddit
We experienced a ransomware attack two years ago. Since then, management has also implemented immutable backups, virtual tapes via Starwind VTL for extra protection, and replication to the cloud. Additionally, we adopted a zero-trust approach to our backup infrastructure. So far so good...
Sultans-Of-IT@reddit (OP)
On day 5? Thats not even enough time to understand the whole IT infrastructure yet. Oh god.
mixduptransistor@reddit
it was enough time to understand the exchange backups didn't work
-SPOF@reddit
We experienced a ransomware attack two years ago. Since then, management has implemented immutable backups, virtual tapes via Starwind VTL for extra protection, and replication to the cloud. Additionally, we adopted a zero-trust approach to our backup infrastructure. So far so good...
jws1300@reddit
Probably only goes away if you leave IT and go mix paint at Lowes.
SokkaHaikuBot@reddit
^Sokka-Haiku ^by ^jws1300:
Probably only
Goes away if you leave IT
And go mix paint at Lowes.
^Remember ^that ^one ^time ^Sokka ^accidentally ^used ^an ^extra ^syllable ^in ^that ^Haiku ^Battle ^in ^Ba ^Sing ^Se? ^That ^was ^a ^Sokka ^Haiku ^and ^you ^just ^made ^one.
Break2FixIT@reddit
Been less than a year, I just started as the sole sysadmin, found a whole bunch of issues and reported all of them to upper management. Got hit with the Big R and it got the poorly set up and poorly secured backups.. even got the azure backups..
Found out when upper management asked the MSP that setup the backups why immutability was not turned on (I asked for access but couldn't get it) to find out that veeam didn't support azure blob immutability until 12.1..
AdventurousTime@reddit
Law enforcement and sys admins who deal with cyber crime a) aren’t paid enough to deal with this and b) aren’t given many resources to deal with PTSD.
Sysadmins are victimized, have to deal with the fall out and usually have to lead or assist with remediation efforts.
ImLookingatU@reddit
You shouldn't stress. Ive helped 5 clients with this. At the end of the day the answer is backups server and storage must not be joined to the domain and make sure have offsite copies.
Also, send the recommendations to the clients in email. If they decline, make sure to keep a copy so if they get it, it's not your fault.
capetownboy@reddit
I was introduced to cryptoLocker in 2010 mostly with small business clients of mine and it was a baptism of fire and I developed a base set of controls that have served me well. Over and above the basics like endpoint protection and proper intrusion protection plus phshing spam filtering and admin rights restrictions although that's not always possible. Lock down powershell so that no device in your network can use it without an action that you control like adding a user like an admin to a security group, and use basic SRPs with path restrictions. Most malware will need to execute within a certain path and More often than not tmuse Powershell Automatically install executables DLLs etc.
andecase@reddit
The one year Anniversary of our ransomware is on the 27th.
Every call is extra terrifying right now.
peacefinder@reddit
Yep. I was around for the early days of Cryptolocker, and managed to catch it in the act a few days after learning it existed.
For a couple years there snap overreaction was a rational response. It took me a while to get over that.
PlasmaStones@reddit
I get PST from the money gram logo.....
cbass377@reddit
Use that anxiety to fuel your preparations for the next one. Try to convince yourself it is no big deal. It happened once, it will happen again, it is a matter of time.
Prepare an over night bag, work up an outage shift calendar, review your app locker whitelists/blacklists, improve your security posture, improve the response tools, improve your detectiont tools, test your backups (record your times to add some gamification). Also get your docs in order so you can bring in temp staff to help with the recovery. You need to be able to give them an account, a checklist, maybe a radio if it is multiple floors, and let them hit the ground running.
Last time you got screwed, this time you will be ready.
Do this, and when the adrenaline spike lands, you can tell yourself, "I am prepared for this, We will bang it out and be home in time for pancakes."
It is the uncertainty that is causing the anxiety, preparation will take that away. Then it just becomes an extra long day at the office.
Hope this helps.
Sultans-Of-IT@reddit (OP)
This is kind of what I've been doing to the extent its possible.
joefleisch@reddit
I have had PTSD since Exchange ProxyLogin. No damage to our systems. No mailboxes on Exchange on-premise. Just a shell dropped with no connections. Nuked server and rebuilt.
I watched the firewall and EDR logs around the clock for weeks and deployed many more sensors and security rules.
Sultans-Of-IT@reddit (OP)
Is this the one that triggered the mass exodus to exchange online?
ColXanders@reddit
Same.
levan86@reddit
Can vouch, we get into a major ransom incident couple of years ago.. close to 2020, I personally never recovered from it, always overthink and paranoid every time i saw a ticket about servers went offline on weird times.
It changed our company and team to be better and more security focused now.
blurrario@reddit
yep. we were hit 5 years ago and i am sure it shaved months (years?) off my life.
Haelios_505@reddit
I was working for an msp during the first outbreaks of ransomware around 10 years ago. Very scary fucking times as everyone was scrambling to get something in place to detect when file encryptions were taking place and how to shut them down to prevent it propagating.
Itsquantium@reddit
Thats why we should switch to pen and paper. Can’t encrypt hand written files! And you have redundancy if you copy the paper with a copy machine. Easy
Haelios_505@reddit
We were seriously contemplating locking internet access down to a single pc with no drive access for one client that kept getting hit.
Screwbie1997@reddit
That paranoia ended up being a good thing for the company I worked for after we rebuilt the entire company. Employees were always skeptical of odd looking emails, attachments, etc. It was awesome, they got damn good at figuring out phishing emails, even the ones I made to test them.
StepKnock@reddit
Hey there, I completely understand how stressful and overwhelming that situation must have been, and it’s not uncommon to feel heightened anxiety after dealing with something as serious as ransomware. It’s great to hear your client was able to recover, but the mental toll it takes is real.
I work with businesses on improving their data resilience and cybersecurity strategies to help prevent incidents like this and reduce that “what if” anxiety. If you’d like, I’d be happy to chat about some proactive measures and solutions that can give you and your clients peace of mind.
doblephaeton@reddit
I had a full blown panic attack one day at work when someone asked me why the internet was slow.
this was a full on physical attack on my body from someone asking a question. I was shaking, jittery, my fight or flight senses were pinging so much
The IT director asked if I was ok, and I burst into tears.
he took my laptop and work phone off me, and told me to go home, call the GP and get a mental health plan to help me. and not to come back for at least 2 weeks, more if my GP suggested more.
I kept him up to date on what was happenign,, and he was the kindest person during that time..
I had been stressed and burnt out for maybe a year before that after dealing with all kinds of incidents, projects etc, and I needed to find my way back, and find ways of managing stress.
You might be reliving the trauma,, seek help before it consumes you.
anonymousITCoward@reddit
it faded pretty quickly for me... you shouldn't be jumping at everything, there are specific things that happen during a ransomewear episode. the more of those you see piling up, the more aware you should be. Treat no internet like no internet, not ransomewear... if you do you'll never see the forest through the trees.
Sultans-Of-IT@reddit (OP)
I agree with everything you said, and this is how I calm myself down. However, it feels like my body does it without my brain telling it to, like an almost instinctual reaction.
patmorgan235@reddit
Have you taken any vacation since the incident? Maybe you just need a break, even if it's a couple of days.
Sultans-Of-IT@reddit (OP)
I haven't had a day off in 8 years.
patmorgan235@reddit
Fix that. Thanksgiving is next week, take Thursday and Friday off and spend it with your family. No one at the office is going to remember that you valiantly keep everything running during the holidays, but your family will remember that you weren't there.
Bartghamilton@reddit
I haven’t been hit but still a bit on edge about it. Even knowing I’m doing everything right still know there’s always a new way in, zero day’s, etc. Especially anxious around holidays as I know some people who have been hit then in the past. I’ve actually started shutting down anything not in use over long holidays just to reduce the risk surface area wherever possible.
Sultans-Of-IT@reddit (OP)
This is the problem: I wake up every day and see a new zero-day, and I have to evaluate whether this affects me. It's exhausting.
basicallybasshead@reddit
That's actually a scary thing!
stonedcity_13@reddit
RYUK PTSD here! You're not alone!
Investing in a 24/7 SOC has made me sleep better though
Bovie2k@reddit
Someone once told me don’t see zebra’s when you hear hoofs. For example if you hear hoofs in the us it’s likely a cow or a horse not a zebra. I know that doesn’t fix the adrenaline spike but the thought process can help you on your way to recovery. Also speak with a counselor if you need to. Work on addressing it now and don’t let it continue to fester.
hihcadore@reddit
Solo guy at a company. We got hit and I just assumed it was a DNS issue. Nope all of our VMs were encrypted but no endpoints.
We had backups but I only trusted the app backend databases we used.
I rebuilt everything else from scratch. Luckily our corporate endpoints were cloud only and our identities were sync’d so people could still work just find, it just took a three days or so to get the domain and databases back up and running. Then another 2 months to build in all the redundancies.
My after the fact analysis is defender for servers and azure arc are really really nice for the price. It makes life so much easier. I also applied all the CIS benchmarks and our secure score is in the high 80s.
Trip_Owen@reddit
I haven’t even been hit my ransomware yet but I think I have third hand PTSD. I’m terribly afraid of it happening and knowing that I’d have to deal with it, it does scare me so I can’t blame you.
hihcadore@reddit
It’s a trip when it does. You’ll probably just assume it’s DNS.
TechInTheSouth@reddit
Every time the phone rings...
I am part of a small msp, recovered from 3 ransomware attacks over the last 5 years or so. I am sure it is not healthy.
zakabog@reddit
Yeah, 7 years ago, I have some BTC leftover from paying for our data to get unencrypted. My CTO at the time had so much neglected technical debt because he wouldn't let us do our jobs without him being involved so it felt nice to see it actually bite us in the ass. After that we were allowed to implement a proper backup solution with nightly diff backups and weekly snapshots. Having a few thousand dollars now in BTC from the remaining "change" is just a nice bonus.
Sultans-Of-IT@reddit (OP)
LOL what a bonus
killy666@reddit
Yup. Has someone that had to deal with a lot of ransomwares, it gets better, but my heart still skips a beat when i get something wrong in my monitoring console.
Burning_Eddie@reddit
OP. I didn't fit the carafe ony coffee maker properly about 8 years ago and coffee dropped on the counter and the floor before I realized it.
To this day I double and triple check.
No big deal.
truslack@reddit
I'm still in the (network and emotional) recovery stage, since an attack 2 months ago. Nightmares and all sorts of things, flash backs to the morning I discovered it, kicking myself for taking work emails off my phone, which would have alerted me as things started to happen. Glad I'm not the only person who feels the same!
jake04-20@reddit
If you could go back in time, what monitoring and alerting do you think you would have benefit most from?
Sultans-Of-IT@reddit (OP)
Seems to be a lot of us.
Parity99@reddit
Been through 2 x ransomware attacks, 1 minor, 1 major.
To manage the ptsd, create a playbook to follow then stick to it. This can put some proper structure around you response and help.
Appropriate_Cover529@reddit
20 months and major PTSD at my new role since my last company folded due to loss of reputation from LockBit 3.0 attack.
My last Head of IT had 8 years of technical debt. Our Forti VPN was compromised, Veeam exploited, and the creds cracked for an old service account with DA. Kaspersky did nothing...
XDR, SIEM, Immutable Backups, SSO all things that shouldn't be taken for granted.
theKtechex@reddit
Almost 3 years for my company and yeah, still haunts me to this day, specially since we have so many old people that are click happy....
amberoze@reddit
This is why all of my hats are lined with tin foil.
Sultans-Of-IT@reddit (OP)
I need to line mine with some benzodiazepine that soaks in through the skin.
amberoze@reddit
Nah, I just let my doc prescribe that and take it orally.
chum-guzzling-shark@reddit
my phone hasnt rang for 20 years without giving me brief ptsd
YouCanDoItHot@reddit
I've been through three ransomware attacks. The PCI breach I went through is what gave me PTSD.
E-werd@reddit
It's happened to me a couple times. Users don't have admin and they're trained to let me know immediately when they see something. All I've had to do was nuke the machine from orbit (disconnect from network and reimage) and restore file-level backups. Easy peasy.
I've even recently had a few instances where people fell for phishing attacks and they told me so fast that they never even sent out emails before I was able to react.
I love my users. There are many like them, but these ones are mine.
dronly1u@reddit
I get this feeling every time a server isn't reachable / when users tell me that files / services (that should be available), aren't.
I hate it and hate that that's my body's natural reaction... But glad to hear that I'm not the only one!
trixster87@reddit
I had to deal with 3 clients back to back to back .... can't and won't say more than my jack o lantern this year was just *.Play on one side and *.blacksuit on the other side.
Sultans-Of-IT@reddit (OP)
You must really be on high alert.
ceantuco@reddit
yup. 10 years and I am still paranoid about it.