User missing a week's worth of changes to an Excel file [LONG]
Posted by jwckauman@reddit | sysadmin | View on Reddit | 53 comments
TL;DR - User said a week's worth of changes were missing from his Excel file (which was stored in his OneDrive Documents folder). Turned out he had attached the file to an email, and then started working off the attached copy which is stored deep in C:\Users\
--------------------------------------------------------------------------------------------------------------------
User came to IT in a panic as the spreadsheet that was due today was missing a week's worth of changes. The file with the missing changes was currently located in his OneDrive Documents folder so I checked versions first and noticed it had not been saved since a week ago. Prior to a week ago, the file had been saved numerous times (already had 39 versions) so something caused that to come to a grinding halt. Here's where I looked next:
- C:\Users\
\AppData\Local\Microsoft\Office\UnsavedFiles - C:\Users\
\AppData\Local\Temp - SharePoint Enterprise Search (on-prem SharePoint sites)
- SharePoint Search (SharePoint Online)
- File Server Shares (by filename).
- User's laptop event logs - for instances of the filename.
- SolarWinds Security Event Manager (SEM) - security/application/system events that included that filename.
- eDiscovery of Exchange Mailbox for file as an attachment
I even searched the entire C: drive for the file by its name.
I also ran an audit in Microsoft Purview using different search criteria on the off chance that he had done one of the following:
- Saved a copy of the file elsewhere (searched for its filename across all OneDrive, Sites and Mailboxes)
- Saved the file with a different name (searched for keywords unique to file across his OneDrive/Mailbox)
- Emailed a copy of the file to someone else (searched across all mailboxes and message trace logs for filename)
The file did not show up anywhere, and but I did find a couple clues:
- Laptop "Microsoft Office" event logs had a reference to the file being opened from Outlook
- Purview audit logs showed Outlook.exe was the last program to touch the file (not Excel.exe).
So I considered the possibility that he had attached the email so he could send it to somebody, but before he sent it, he decided to make additional changes, so he re-opened the file from the email attachment draft, NOT the file's actual location in OneDrive. I looked up where Outlook stores attachments when they are first opened, and found this path:
C:\Users\
I looked in that folder and found 186 files from over the past year, with the most current one being a file with the exact same filename as the one that he was missing a week's worth of data from. The date on the file was today's date though, but hoping against hope, I copied the file over to my C: drive, renamed it, and emailed it to him asking if this file had the missing data. I got back a quick and resounding YES!
I'm still not sure how this happened. I tried emailing myself a file that was originally in my OneDrive and then went back to sent item, opened the attached file (which opened in the above 'content.outlook' folder), made a few changes and then clicked 'Save'. Office had me save it back my OneDrive. I can't get any changes to save to that temp folder. I wonder if I disconnected the network connection if it would let me.
Anyone run into a situation like that? I feel like I need to start a "all the places you can lose a file" document/guide. I also want to write a "places you should not be saving your work" document, which would include "Desktop". I didn't even ask about USB thumb drives or 3rd-party email systems. One lesson learned is that our powerful M365/Azure auditing doesn't cover files that are saved locally. And my Advanced Audit policies that are applied to our File servers, also don't cover the C: drive of a user's laptop. Feels like we might want to introduce C:\Users to the 'Advanced Audit' policy so we could have found the user's activity in that folder.
FireLucid@reddit
I solved this a few times at my last job about 15 years ago, haven't seen it again since. I guess it's still an issue....
Unable-Entrance3110@reddit
I had the president of our company approach me at a happy hour with a story of data loss... He opened an Excel file and made changes, then close the file and open it again... bam, all changes gone!
I was thinking, this must be some kind of user error so we went to his office and he showed me. Sure enough, make changes, click save, close the file, open it again and..... all changes lost.
Turns out, it was user error though. The file was a CSV, not an Excel document. The change he was making was adding a new tab with data within it. He had turned off data loss warnings when working with CSV files...
Add in the fact that Windows doesn't show file extensions by default and you have a perfect environment for data loss.
Took me way too long to figure it out and I missed finishing my happy hour drink! :)
FireLucid@reddit
This can be controlled by GPO or via a remediation script via Intune (it's a registry setting). Something I always turn on.
djholland7@reddit
This is one thing that frustrates me.
How much are the users of these systems and tools required to know how to use them?
I’d expect to lose my job if I couldn’t use the tools needed. But to help people like this is beyond frustrating. And how much time you spent on it is absurd.
Sure_Acadia_8808@reddit
It's not the user's fault if the tools are gaslighting them into low digital literacy. My Linux users gain skills. Windows and Mac users are visibly losing them. The platforms are actively hostile to learning. That's on purpose, to keep customers from having agency. Customers with low agency try to throw money at their problems!
SirLoremIpsum@reddit
I have wow'd a few people back when I was on helpdesk by retrieving things from this location.
Certainly wouldn't have been my first guess (or 18th), unless someone specifically mentioned "i got emailed this".
Good work!
fatbergsghost@reddit
I only know that place because it kept getting corrupted and people couldn't open things from outlook.
Sure_Acadia_8808@reddit
Classic Outlook!
This is why my customers don't use MS Office. Who even has time for these constant breakdowns? I use stuff that doesn't just "stop working" at random cause "corruption." For some reaosn, all these other programs can work for 30 years without losing data and corrupting themselves every week. Plus sometimes we get DRM issues too. So it broke itself? On purpose? Why would we pay for that?
Fender_Stratoblaster@reddit
I was an operator at treatment plants but the 'computer guy'. At one, about every three months the boss would call out over the intercom "Fender Stratoblaster to my office! Fender Stratoblaster to my office!" And every time it was the same thing; couldn't print to his Epson printer which of course just took, once again, rebooting the PC.
At a later job I was still the 'computer guy' and I had set up a folder structure for our division on a shared drive. About every six months this boss would call me as some folder had been deleted and it's important and why would someone do this?!?! And every time it was the same thing; someone fat fingered a double-click and drug a folder into another one.
People opening and working on a file from a temp location via email etc. and then not finding it was pretty standard.
nmrk@reddit
I had a user’s PC die because she never emptied the trash for three years. She thought the files were deleted instantly, nobody ever told her to empty the trash.
Wendals87@reddit
How did a full recycle bin make the pc die?
pdp10@reddit
Fills writable storage?
itishowitisanditbad@reddit
How does a computer die from that?
Sure_Acadia_8808@reddit
TL;DR is the whole PC wouldn't brick, but the hard drive could physically become unusable, and/or the filesystem could corrupt itself to the point the data was unrecoverable.
Physically, hard drives keep a limited number of sectors in reserve, and swap them into use when bad sectors are discovered. If the drive is starting to lose sectors, it'll swap some fresh ones in, until it runs out of spares. Filling the writable space means you're going to touch all the sectors on that disk, and if anybare bad you're going to discover them much faster. Once you've spent your sector spares on preserving junk data in the Recycle Bin, sections of the OS itself might start landing on bad sectors. That's when you get I/O errors that prevent boot.
Windows' filesystems would have been FAT32 or NTFS. These both are legacy filesystem technologies with poor resiliency and almost no capacity for error correction. They both have severe fragmentation issues. NTFS has journaling, but FAT32 doesn't .What little maintenance they can do (mostly just the NTFS defragmentation API) is completely dependent on having enough writable free disk space to work with. Windows disks should always keep at least 25% of the drive unused for this reason.
Fill up a Windows drive past 75% usage, and it's going to crash one way or another.
sakatan@reddit
I'm just here to cheer your troubleshooting
30yearCurse@reddit
2nd that, it was some great t/s. Had one guy say he lost 2 days of work, we told him well not in recovery, not in backups not in shadow copy. Your screwed. Re-do and save it a couple of times. For some reason while it was open autosave never kicked in.
Jezbod@reddit
This is why I tell users that when they open a new Office file, click save and give it is name, this activates autosave.
We also have the GPO settings to save temp file / draft copies if the app closes without saving.
doubleknocktwice@reddit
I will have to look into that.
Jezbod@reddit
It has saved so much director time...strangely, none of the workers have the problem.
30yearCurse@reddit
sage advice, One that I need to follow when I have 9 book1-8.xlsx and some saved with that default name... lol.
HistoryHot705@reddit
If only everyone in IT was trying like this guy! Big ups!!
BoltActionRifleman@reddit
Yeah I would’ve given up once I didn’t find a modified file in the folder or the snapshot/backups. I can’t bring myself to care about some of this stuff anymore. Too many years of dealing with careless users I guess.
thtguyonreddit14@reddit
Seconded! You really went above and beyond the call of duty with this save.
Troubleshooter salute to you!
19610taw3@reddit
Unfortunately I've been in that same situation before too. Right after we switched to onedrive at my previous org, what was supposed to make things easier for end users (well according to management who pushed onedrive onto us) made it more difficult and people couldn't keep their files sorted out. Now instead of having it on a mapped drive or stored locally, they could have it saved in their personal one drive, corporate one drive (sharepoint), temp files, working out of an email on a live document.
It became a mess. I can't say we ever lost any files - we were always able to get the most current version of the file (like you troubleshot and got to) but it created a ton of work for probably 12-18 months. Eventually, most people got the hang of operating cloud files and we stopped having as many issues.
There was still problems with the onedrive client crashing frequently and causing issues ... but that was self induced.
Majonez_@reddit
Been there done that It is pretty hard to explain but I always tell my duds to check if "Autosave" is ticked in top left corner.
My solution was to download voidtools everything, search for *.xls, sort by modified date and ask user if anything looks familiar.
autogyrophilia@reddit
Hey, not dissing on 3rd party software, but you can do the same with this line of powershell
Get-ChildItem -Path $env:USERPROFILE -Filter "*.xlsx" -Recurse | Sort-Object LastWriteTime
You can use -Descending if writing to a text file, otherwise the ascending way is better because you can scroll up for files.
Robertsipad@reddit
I worked in a department where they had a really convoluted folder structure for projects which became overgrown. Voidtools everything made me a lot more productive at finding files that a coworker “remembered working on something similar at some point”
Mackswift@reddit
Never used voidtools, but I've seen this exact same situation a few times over the years. User somehow winds up working on the attachment file from an email but doesn't truly save any changes. But it winds up in the above mentioned directory.
It basically comes down to end user not paying attention to what they're doing plus lack of training with the apps and tools they use.
Chaucer85@reddit
God I love voidtools' Everything.
zero44@reddit
Some people's workflows are just genuinely terrifying.
I_T_Gamer@reddit
I will not bend over backwards for people who do not save their work. Earlier in my career I prided myself on not losing data. I still pride myself on not losing data, but if you don't save and leave your machine on overnight and we lose power I am not helping you restore that work, well probably not(C lvl can be a different story).
SAVE YOUR WORK!!! Often times no amount of coaching on my part will help these types, only the reality of losing hours of your life will teach this lesson.
RedditAccnt_2@reddit
Currently working with a user trying to figure out why they had a similar issue... this post gives me more solutions to try.
Thank you OP for sharing this!!!
Robertsipad@reddit
Did it have the original file name or a random string?
auriem@reddit
Yes, most of the users I interact with would benefit from digital literacy and common sense training.
brink668@reddit
Yes ran into this on a trading desk 10 years ago. I distinctly remember because I was always assigned to help the person after another issue I solved with his quad monitors
gadget850@reddit
I have done that to myself and had to figure it out.
robertscoff@reddit
That’s insane. Why would a copy of a file ingested into the email system continue to be editable?
theoreoman@reddit
I once unplugged a drive coppied it and searched for a deleted file. Some exec had a file with a bunch of info on it that would cost the company millions if it wasn't recovered. All the nice forensic tools that exist today didn't back then. It felt like a ctf challenge
coak3333@reddit
The worst is when the user says the file is gone, but they don't know what the name of the file is!
With CRMs, the amount of files I've recovered from C:\Temp... is uncountable.
TheFumingatzor@reddit
Everything is great for things like that, because you could just filter after .xmlx and sort last date modified, and I'm pretty sure it'll list the files that were last modified including the file you've been looking for.
InitiativeAgile1875@reddit
TLDR.
Restore from backup.
Next.
TheTipsyTurkeys@reddit
Had this happen with a lawyer who was working on a document for the supreme court. Same deal finding it in the app data folder for Outlook. Shit happens!
Affectionate-Cat-975@reddit
I’ve also seen data forking when user A clicks Sync on SPOnLibrary and user B renames the library. User A retains their copy but get no notice of the file name change so their data no longer syncs and they don’t see the online updates
Ark161@reddit
This is exactly why I stay away from user generated content. I have shadow volumes on, I have backups running in multiple ways, but at some point, the user has to be accountable for their shortcomings.
Beefcrustycurtains@reddit
I'm guessing the user did not actually save it and it acted like a unsaved doc.
davidbrit2@reddit
Dumb question, but did you try the "recent files" list in Excel? That's the first place I go when I can't figure out where the hell I just saved a file 10 seconds ago.
GullibleCrazy488@reddit
Or the file recovery option in Excel.
Ok-Double-7982@reddit
Wrote exactly that, then came to see if any other like-minded souls wrote the same. High five to you!
Ok-Double-7982@reddit
The way I would look is open the Excel application, then look at the Recent files saved locations and work backwards from there.
buffs1876@reddit
My wife is an accountant. I just shared the very short version of this. Also, don’t work off of attachments.
Xmuzlab@reddit
User lying for slacking and now blaming tech as usual
mercurygreen@reddit
The important word in all of that was "Outlook" and that has been a problem since I started dealing with it in with Office 2003. (And I know it's been a problem since it was first written.)
tofu_muffintop@reddit
Didn't read any of this but the title and came here to just say how much I appreciate you putting long in the title as not to waste my time much appreciated