No audit log enabled. Someone deletes files. What do you do?
Posted by Spiritual_Mine1974@reddit | sysadmin | View on Reddit | 129 comments
So, thanks windows for disabling audit log for file events as default. Because we missed enabling logs for file audits in the file server we are unable to detect who deleted the 180 GB folder.
In this scenario what would you do to find the user?
note: We had daily backups so we got them back.
Crumby_Bread@reddit
Any reason you want to start a witch hunt over what was likely an accident?
Spiritual_Mine1974@reddit (OP)
Nope, It was not an accident. I found the one
theoreoman@reddit
Good news is that your backups work.
Any of those files linked to office M365? Or other accounts? Are there any other services that are linked to those files that failed? How many people have access to the files?
Depending on What's being logged you might be able to narrow down who did it through inference
Is management looking for blood?
Honestly I would take this as a win and learning experience that you were able to recover r
Spiritual_Mine1974@reddit (OP)
8 person in sight. Timestamp detected. No services connected. Currently checking Layer 2 networking for delete events. At least narrowing for ip. We have DHCP lists daily. Fortigate upfront.
No one was looking for blood but because one of the users who has access it detected the files are missing, they talked about it in the company. So now they are looking someone to blame.
statikuz@reddit
?
slickrickjr@reddit
This line belongs in CSI Cyber đ
Solkre@reddit
Theyâre backtracing the change. Two people typing on one keyboard so it runs faster. Hopefully the perp wonât pull his network cable out of the wall.
statikuz@reddit
of course this exists
GitHub - bagaffey/Visual-Basic-GUI-IP-Tracker: This is a GUI Interface created with Visual Basic that can be used to track the killer's IP address. ¡ GitHub
Spiritual_Mine1974@reddit (OP)
We know the time of deletion. So we can check network logs to detect who deleted. At least we are trying
Humble-Plankton2217@reddit
It's off by default because it has the potential to create a high volume of logs, especially on busy file servers (2-3GB worth per day).
arslearsle@reddit
You could write a little powershell script that monitors folder.
Its built into windows. Has nothing to do with event log.
How it works is you config a subscription to one or more folders, and define what you want to watch.
Create/delete/read/write for example.
Output to logfile or whatever you see fit.
You want a async solution ie. can handle more than one request at a time.
Google powershell filesystemwatcher async and you find some good templates
My experience is that you already know who these professors/doctors and CEOs causing the problems. Yeah, its always the fanciest titles causing problems đ
Random-D@reddit
restore backup and call it a day
tristand666@reddit
And enable auditing for next time.
Spiritual_Mine1974@reddit (OP)
Yeah, just enabled for everywhere
Indiesol@reddit
There is a reason it's not enabled by default. Keep an eye on both server performance and disk space, because both will be impacted.
Spiritual_Mine1974@reddit (OP)
Damn⌠let me buy an extra 7.8 TB disk. I needed a reason
badaz06@reddit
C'mon. You can but a 5 TB drive off amazon and attach it via USB for under $200. Just get 2 of them and your set! It's that easy!! lol
m0rp@reddit
Security log max size on non domain controllers can be set to 4 GB. If you havenât changed the log size. You probably should. Also, make sure itâs set to overwrite events.
Youâll also want to inspect how many hours of data the security log can hold with your changes. Because if you arenât forwarding the events or storing it in another way. Itâs very possible the log doesnât even hold 30-60 mins of events.
SomniumMundus@reddit
Hey, itâs me. Your long lost cousin. Let me get one too please.
Spiritual_Mine1974@reddit (OP)
Oh damn, I selected 1 but bought 2!
MetalSufficient9522@reddit
If it's a problem, you need to buy some kind of 3rd party software to analyze the logs, because it's difficult to parse them manually when you need to.
DeifniteProfessional@reddit
Christ you've just reminded me about PHP slowlogs I enabled on a server and never set logrotate. Fuck me that's going to be 100 gigabytes of data by now
Disgruntled_Smitty@reddit
Watch you disk space...
The_Long_Blank_Stare@reddit
The real advice, right here. Logging is great, but costly.
rotfl54@reddit
What do you use for analyzing audit logs file and folder deletions? The last time that we activated windows NTFS delete audits on a large share we got a gigantic security event log, because it seems that Windows is tracking cut and paste actions as deletions and creations.
It was a huge mess to find files that were really deleted and who deleted the files.
Spiritual_Mine1974@reddit (OP)
Maybe I can do a program for viewing the audit logs? And maybe add some little bit of analysis graphs? We sill see what happens after this
Brandhor@reddit
I send the windows audit logs to a graylog server but honestly they are really hard to read, sysmon seems to generate better logs but I haven't tried it yet
tristand666@reddit
DEVO.
Impossible_IT@reddit
https://i.redd.it/tduwc3pjheyg1.gif
rotfl54@reddit
Thanks will take a look at it
poweradmincom@reddit
PA File Sight makes it easy, and does not rely on Windows audit logs.
butthurtpants@reddit
Random-D forgot it wasn't r/shittysysadmin for a minute there.
Spiritual_Mine1974@reddit (OP)
We will call it to HR. I call it a day but being unable to detect the person because of forgetting setting up the settings annoys me
llDemonll@reddit
Why? Chances are person doesnât know they even deleted them and it was an accident.
MetalSufficient9522@reddit
Even better, you will just find them dragged into a nearby folder, and now you have two unrelated sets of files to cause confusion later.
Spiritual_Mine1974@reddit (OP)
To delete them, you need to Control+A then press delete then confirm the deletion. 3 step process. And becuase its payday, the payday files are deleted.
Jhamin1@reddit
They could have right clicked at a higher level in the tree (assuming they can get there) and hit the wrong command from the menus. As for the confirmation? Most people just auto-click confirmations.
Its a real Hanlon's razor situation.
The thing we always ran into was people with sticky mouse clickers accidently dragging & dropping folders in a share into an unrelated nearby folder. The data was all there, but you were unlikely to find it without crawling through the whole file server.
simpleglitch@reddit
Second thing I do when I get a restore request for an entire directory is run a search for some of the file names. (After starting the restore)
When an entire folder has gone missing, it has almost always been someone accidently dragging it somewhere.
RainStormLou@reddit
I have been saved by so many confirmation windows, and I start all of my training on a particular product we use by showing every trainee where someone once accidentally dragged a line item down about three pixels, and disabled access for this critical legacy bullshit application to a massive group of users.
that being said, this particular product had the user interface coded like 25 years ago, and you can't just drag that line item back. it requires a full export, reconfigure, delete and reimport which is 45 minutes that I haven't had in the past few years
Spiritual_Mine1974@reddit (OP)
Thats why we dont give mouse as IT
RetPala@reddit
Are you sure someone didn't just drag the folder into an adjacent folder by holding left-click for .002 seconds?
Spiritual_Mine1974@reddit (OP)
Yep. Even the disk size is different
WizardsOfXanthus@reddit
Wait! That's the ONLY way you can delete those files?
Raumarik@reddit
It's more to flag something happened not necessarily to use it as a finger pointing exercise on an individual. There may be other things going on within that team etc which IT have no knowledge of, it's worth flagging if only to have it recorded that this happened and when.
Fantastic-Shirt6037@reddit
So your plan is to just let them keep doing what they just did? Lmao?
roiki11@reddit
Backups? What backups?
Fantastic-Shirt6037@reddit
Stop sign? What stop sign?
Spiritual_Mine1974@reddit (OP)
Plane? What plane?
persiusone@reddit
Enabling audit logging should be a global policy and not rely on device defaults.
DopamineSavant@reddit
What I do is not care beyond getting the file back.
Turbojelly@reddit
Folder not deleted, just fat fingered into another folder.
Spiritual_Mine1974@reddit (OP)
Nope, you are just eliminated.
Turbojelly@reddit
The number of times I have gone through the "THE IMPORTANT FOLDER HAS BEEN DELEED!! FIX NOW EMERGANCY!!1!1" Where the complainer just moved the folder into another folder greatly outweigh the times the folder was actuslly deleted. It is worth a check.
Spiritual_Mine1974@reddit (OP)
Disk size changed. We already looked at the folders dude. 180 GB data shifts. You are ELIMINATED
Curious201@reddit
if audit logs were not enabled, i would avoid pretending you can prove who did it after the fact. first priority is recovery and containment: restore the missing files from backup, check previous versions/recycle bin if available, preserve whatever indirect evidence still exists, and look at timestamps, open file handles, recent access patterns, endpoint logs, and any backup/sync logs that may show deletion events. after that, fix the process so you are not in the same position next time: enable auditing on the share, centralize the logs somewhere users cannot tamper with, remove delete rights where people only need modify/write, use separate archive or approval for sensitive folders, and make sure backups are immutable or at least protected from normal user access. the management answer is also important: âwe cannot identify the actor because logging was not enabled, here is what we recovered, here is the control gap, and here is the change needed.â
Ohmystory@reddit
Use a tool like tripwire âŚ
gumbrilla@reddit
So, I would not bother. I would focus on the error on audit logs missing, as long as you are driving structural improvements, and learning from incidents, I am content. Don't have to be perfect, just improving.
Losing files and folders, the number of times when I worked in publishing, we found they'd been dragged inadvertently into another folder by mistake was amusing to say the least. Accidental mass deletes with someone panicking.. just nice to be able to tell them not to panic.
Getting audit logging improved, and not much business impact, great success.
theHonkiforium@reddit
We implemented a tool that hooks Windows' for Move command, so if a user accidentally drags a folder into another it stops and asks them to confirm. So far it seems to have help cut down on those.
odobIDDQD@reddit
Yeah, if your backups have just jumped by 180GB then itâs your classic drag and drop.
To answer the question, thereâs only so much time Iâm going to spend digging. Restore the data, enable auditing for next time.
rotfl54@reddit
A few years back we tracked down the root cause of inadvertently moved files: it was a sticky tape on the bottom of the mouse causing the mouse pointer jumping around for a few microseconds...
Spiritual_Mine1974@reddit (OP)
I love fallbacks. Learned from it, yes. It just stings because itâs not a normal user activity and we have missed important thing while building massive things with the best possible security.
I can say that we have never thought of it
GX_EN@reddit
Glad you had solid backups.
One other thing beyond what others noted - I would use this opportunity to check permissions throughout the folders as well.
When I worked for a small MSP, we were doing some work with a customer and they were convinced that they had a virus because random files were going missing on some subfolders. As I dug in a bit with them on a zoom call, it was random subfolders within a department hierarchy. When I asked them to show me the permission structure - the entire company had modify on everything under the shares in their file server..
I told them to lock it down more granular and get back to me if they still had an issue after that.
They never got back to me.
Spiritual_Mine1974@reddit (OP)
Well⌠the problem is right there. The permissions are in good status actually. Normally 35 worker access there but 8 is able to delete. The 8 is the shareholders and department heads.
GX_EN@reddit
Well, that narrows it down. :)
HWKII@reddit
The most dangerous gameâŚ
Parity99@reddit
Have a quick look at permissions, who was logged on etc. Not wasting time on fruitless pursuits.
brandon364@reddit
Look at permissions. Send all of them an email that someone deleted the shit. Did someone complain about the missing files? How did they know the files were missing. Are they Colonel Mustard in the library with the candlestick?
RaNdomMSPPro@reddit
File auditing is off by default because it's a potential performance hit. You have to plan resources around auditing files.
Disastrous_Meal_4982@reddit
GPO to make sure this doesnât happen on another server?
post4u@reddit
For what it's worth, this has been the default behavior on Windows servers since the beginning of time. Restore, turn it on for next time, and move on.
netsysllc@reddit
Buy undelete pro and have it journal all changes to another drive, super quick restore and tells you who did it. But auditing is disabled by default because of performance impact. If all auditing was on the servers would crawl.
Spiritual_Mine1974@reddit (OP)
Weâd normally use AD/DC but because they lowered IT budget and never gave back we had to continue with Home licences. Which became a problem fast. Now we are not using dc on computers.
Audit logs for there are becoming mandatory. Wewere thinking it was enabled directly in the default
AcornAnomaly@reddit
Yeesh.
Isn't that a license violation? Using Home licenses for commercial purposes?
Spiritual_Mine1974@reddit (OP)
What commercial purpose? Is there even a license ?
AcornAnomaly@reddit
Nevermind, I was mixing it up with the details of the Office licenses.
(The personal use Office license isn't allowed to be used for commercial purposes, i.e. in a business.)
Spiritual_Mine1974@reddit (OP)
The thing is the seller sells it like this and they are contracted with MicrosoftâŚ
GullibleCrazy488@reddit
Prob got dragged and dropped into another folder or drive.
poweradmincom@reddit
That was my guess too - have seen it over and over.
Spiritual_Mine1974@reddit (OP)
I wished but nope
GullibleCrazy488@reddit
awwh, shucks
F0rkbombz@reddit
Thereâs a reason that audit logging for this (more specifically âobject accessâ) is disabled by default: itâs absolutely ridiculously noisy. You have to spend time configuring that or itâs going to produce an unholy amount of worthless crap in your logs.
Spiritual_Mine1974@reddit (OP)
Any recommendations on logs then?
F0rkbombz@reddit
Either spend the time configuring it, or get an EDR solution that records these kind of things. Maybe you can do something w/ Sysmon, but honestly Iâd go with EDR first.
Spiritual_Mine1974@reddit (OP)
I will try, thanks!
roiki11@reddit
Files? What files?
PowerSamurai@reddit
Bastard operator from hell reference? Nice.
Spiritual_Mine1974@reddit (OP)
Files?
Bright_Arm8782@reddit
Nothing. You have the files restored. Someone made a mistake, it happens.
I would go hunting for which admin configured the server folder and share permissions so that a user could delete from it in that fashion. That is an error, that needs to be corrected.
Spiritual_Mine1974@reddit (OP)
Well, my boss and me. No one else. And for the permission, the current ones who has access had the true permissions. They were the department heads and they wanted it to be like that
Bright_Arm8782@reddit
Good hunting, I hope you find whoever and fine them doughnuts for the whole office.
Spiritual_Mine1974@reddit (OP)
I am going to troll them. Wrote a script that will close the pc randomly. We will see how it goes
Spiritual_Mine1974@reddit (OP)
And as IT i will say no problem found when they come
mkallon8@reddit
Let it slide cause if you raise it then it will fire back at you for not enabling the logs and users can always make human errors.
Spiritual_Mine1974@reddit (OP)
Yeah, no way. The company owner hides from me. So no one is saying anything to me. But yeah, not raising anything best way⌠until we found the guilty one
RetPala@reddit
Captain Ahab over here
neoh4x0r@reddit
Yeah, don't tell people you are investigating an incident so that the culprit won't have a heads-up to hide.
sccmjd@reddit
Isn't there a performance hit for enabling logging on a fileshare? Everything takes a little longer for everyone since it's logging the changes.
Spiritual_Mine1974@reddit (OP)
I really dont know and dont care. Just I donât want it to slide again
justaguyonthebus@reddit
I usually find those files in other folders. Most commonly a slow double click that's accidentally a move so it just disappears.
When deleting that much, they are prompted and explorer has to do the calculations. They get plenty of opportunities to cancel. So whomever deleted it knows who they are
Spiritual_Mine1974@reddit (OP)
Yeah, created meeting face to face. We will see
Fuzzy_Dude@reddit
Are you sure putting someone on the spotlight this hard might not blow back on you in the future?
Spiritual_Mine1974@reddit (OP)
Nope, cause im already leaving 2 months later. And still, who is the one that will try to do it on me?
Ermmahhhgerrrd@reddit
And you checked other shares to see if it accidentally got moved? That used to happen all the time when I was an IT manager. VSS is your friend tho, you don't have it turned on?
Spiritual_Mine1974@reddit (OP)
Yeah checked it. Even volume sizes are different
Ermmahhhgerrrd@reddit
Dang. Now it's really the time to turn on Volume Shadow Copies - mine were set to 3x a day and it only records the changes (it puts it back together if you restore it tho). Why 3 times? Because PhDs.
M4niac81@reddit
We restore the data and carry on. Most of the time it's accidental when data gets deleted and even if it isn't, proving intent is an impossible task, I've better things to do with my day. We have shadow copies on our file servers so restoring deleted data is trivial.Â
Spiritual_Mine1974@reddit (OP)
I have more things to work on but having fun detecting this person. It gets me
ProgressBartender@reddit
You can set up auditing on the server and the enable it on the volume, have it save the audit log as either an xml or an event viewer file. In the NTFS permissions window, you have an auditing tab where you finish setting to the NTFS auditing.
Once all that is done the NetApp will have an auditing volume with the file you can look at to find audit events like permissions changes, access file, delete file/folder, move file/folder; just like a windows file server.
Just have to be careful to set the retention on the files so they drop off and donât fill up the audit volume you created.
Setup CIFS vServer auditing
https://docs.netapp.com/us-en/ontap/nas-audit/enable-disable-auditing-svms-task.html
Set up NTFS auditing
https://docs.netapp.com/us-en/ontap/nas-audit/configur-policies-ntfs-security-concept.html
Dizzy_Bridge_794@reddit
Restore and enable logging
Spiritual_Mine1974@reddit (OP)
Would AD/DC on computers hwlpwd for logging these events?
Garix@reddit
What about sign in or authorization logs to the smb share or domain controllers at the same time? Iâve also seen people drag it onto their own desktop somehow. Depends on how you map the drives for them but itâs possible.
Spiritual_Mine1974@reddit (OP)
Because there are 300+ worker who has access to the file server, it getâs really bad really quick
HDClown@reddit
I have EDR deployed everywhere and it logs file operation events, so I'd be able to find it in that data.
Idenwen@reddit
If you have a time of deletion you maybe can track activity and logins on user devices to see who was active at that time with what, but that depends on user account amount and all over system noise.
Spiritual_Mine1974@reddit (OP)
Yeah was looking for that but because the file server endpoint is assigned as letter to the computer, the logs are not useful in login or logout.
But thanks for the advice
Churn@reddit
Ask everyone what should be done to the person once they are found. The employee that responds with the softest penalty is your culprit, focus on them.
Spiritual_Mine1974@reddit (OP)
You are giving yourself to the HR on the Gold Plate, no thank you.
neoh4x0r@reddit
This won't work....they know you don't know, and will suggest the maximum penalty.
ohyeahwell@reddit
Iâd do a tree and make sure they were deleted and not moved. Our older users struggle with mouse and frequently drag and drop folders while trying to double click.
Used to have audit logs set up to capture these type of events when we were onprem.
WiskeyUniformTango@reddit
Microsoft defender for endpoint logs all events on devices. What are you using for AV?
Spiritual_Mine1974@reddit (OP)
ESET on these users. One of them has Windows Defender only
Sys2Soc@reddit
see the shadow copies of the files and narrow down the deletion window by comparing the difference in the snapshots, now go the SMB protocol logs in windows event viewer check who are all connected and disconnected between the timeframe. also check the firewall logs for sessions with IP details deleting 180 GB folder will create massive outbound traffic from the file server to that specific client IP
Spiritual_Mine1974@reddit (OP)
Will try to simulate this in vm server. Itâs a copy of the server. If it works, I am going to kiss your bald head
Naznac@reddit
if you know at what time they were deleted you can always check the security logs to see who authenticated to the server around that time. Sysmon is a lifesaver when properly configured to trace wierd happenings.
Reeheeheeloy@reddit
Unless you have logs somewhere from something else... not a lot to do.
That's why it's important to have layered systems so you're covered in the event that one is inaccessible/compromised/misconfigured/microshafted.
Personally, I like Threatlocker for that use-case... it's not even one of their main features, but it does do a good job of it.
Icolan@reddit
I would restore the folder from backups, and enable audit logging on the server. I would also review audit settings across my environment to ensure I am collecting the events I need.
You really do not have a viable way to determine who deleted it.
MeetJoan@reddit
Worth checking: File Server Resource Manager logs, your backup software's change logs (Veeam/Datto usually record what changed between snapshots), and any EDâŚ
Worth checking: File Server Resource Manager logs, your backup software's change logs (Veeam/Datto usually record what changed between snapshots), and any EDR you have running (CrowdStrike, Defender for Endpoint, SentinelOne all log file operations with the user account that performed them).
USN journal might still have it depending on volume size - "
fsutil usn readjournal"is worth a try.For next time: enable object access auditing via GPO plus a SACL on the share root.
neoh4x0r@reddit
Without auditing enabled there's no way for you to track this.
This might sounds like an opportunity to enable audit logs, but also to audit your current security ACLs to better secure critical data by limiting this type of action to a small group of users.
daschande@reddit
Shoot the hostage. Take them out of the equation.