The email was never delivered

Posted by CaptainZhon@reddit | sysadmin | View on Reddit | 467 comments

Last week a department put in an incident saying an email was not delivered and because of that it cost the company a substantial amount of money and they are saying the email system messed up. The email SME did his research- it was Exchange- no external email. Someone internally email an attachment to a DL and none of those people in the DL claimed they received it, but mail flow and Exchange Reporter said they did. We were able to show who sent the message and when it was received by all the people in the DL which are internal as well. Fast forward to today and that department is sticking to it’s guns that the company system messed up and the email just “disappeared” Really?! FFS are users this dumb? Not one but FOUR in the DL? What more do you want I thought? I was told by my leadership to find hard proof the message was there. So I did. I gave my adm account access to the mailbox and logged into each one and found the message. The person causing the most stink- I found the message in their recover deleted items and it had been deleted four minutes after it was received. Now the department is blaming us because the file doesn’t have the right content- sorry that is not even remotely the IT department’s responsibility- please find another way to spin it how you f**ked up and don’t blame IT!

Reply to Post

Reply

467 Comments

[-]

LingonberryOne3877@reddit

When i was working as an IT-Consultant i helped a company to set up a Sharepoint page for their work and they insisted that EVERY employee should have admin rights on the whole site. Myself and other people involved with the setting up told them that it is a VERY bad idea but they insisted anyway and got what they wanted. Fast forward a few weeks and one employee basicly crashed the whole site when he wanted to change the looks of the site. I help them out and tell them what user did what and why we should revoke admin rights. The company puts the blamegame on us cause we didnt revoke the admin rights from the start... A few weeks after that we ended the relationship we had with the, due to them not beeing a customer we want to be involved with due to several reasons.  Its always someone elses fault, right?

Reply

[-]

CKtravel@reddit

>company to set up a Sharepoint page for their work and they insisted that EVERY employee should have admin rights on the whole site. I hope you had this in writing too... >got what they wanted. Let me guess: they didn't have backups either, right?

Reply

[-]

LingonberryOne3877@reddit

Yeah, everything was discussed via our ticketing system so we were in the clear there...  They did have backups but the Sharepoint were critical to their work so even a few hours of downtime cost them alot.

Reply

[-]

CKtravel@reddit

Nice! Hopefully they've learned their lesson well too.

Reply

[-]

DrAculaAlucardMD@reddit

It's always fine to fire a customer.

Reply

[-]

NN8G@reddit

Multiple adults are caught lying about the basic fact of having received an email? And any further lip from them is being tolerated?

Reply

[-]

Disorderly_Chaos@reddit

I would look them dead in the eye and say, just like I said to my 8 year old, tell me the truth here and now, because as soon as I start digging… you’re not going to like what I find. I know you took the fucking cookie Jacob!

Reply

[-]

gangaskan@reddit

jacob is a cookie slut.

Reply

[-]

PoopTimeThoughts@reddit

Same tbh

Reply

[-]

gangaskan@reddit

im pleading the fifth.

Reply

[-]

2shyapair@reddit

I'm drinking the fifth, while eating the cookie!

Reply

[-]

ZenAdm1n@reddit

This is a thing and if the person doing the gaslighting is politically protected the best you can do is mark the ticket as an unreproducible error and move on. "All of my failures are shortcomings and sabotage of other departments. All my accomplishments are my own."

Reply

[-]

Tymanthius@reddit

> mark the ticket as an unreproducible error and move on. No. You append logs and findings and document it as it is. It's not IT's job to punish people. It's IT's job to fix errors. There were no errors, and here's the doc's saying so. Close ticket.

Reply

[-]

ZenAdm1n@reddit

Sure append that log and document the troubleshooting. We aren't punishing IT here. We're deflecting. The user states the problem. The logs didn't identify a problem. The troubleshooting couldn't reproduce the error. At what point did I suggest punishing anyone?

Reply

[-]

Tymanthius@reddit

It wasn't an error, and your comment read to me like IT should be trying to discipline. So don't log it as an error. Log it as 'user rcv'd email and deleted it'.

Reply

[-]

ZenAdm1n@reddit

We can't say that the user deleted it because we don't screen record or log keystrokes. You say the "email appeared to be deleted" or "the system logged a deletion". And I'm mainly talking about how to avoid a political conflict with entrenched management. These are the people you treat with kid gloves no matter how wrong they may be. Deflect without reversing the accusation.

Reply

[-]

Tymanthius@reddit

> We can't say that the user deleted it because we don't screen record or log keystrokes. You say the "email appeared to be deleted" or "the system logged a deletion". Wonderful correction!

Reply

[-]

GreatRyujin@reddit

My thoughts exactly, this is definitively a HR issue. This person (very likely) knowingly deleted a unwanted mail and then lied about it.

Reply

[-]

neddie_nardle@reddit

>This person (very likely) knowingly deleted a unwanted mail FOUR people knowing deleted it!

Reply

[-]

inphosys@reddit

One person deleted the email 4 minutes after delivery, then coerced 3 others to delete it sometime later to help them bolster their lie. Hey OP, grant the admin account access to the other 3 mailboxes that are members of the DL and see when they deleted it. File your grievance with HR regarding the destruction of corporate information and then sit back and enjoy while individual # 1 twists in the wind and the other 3 flip on them because they don't want to be associated with an HR inquiry.

Reply

[-]

mitharas@reddit

I am most certainly on OPs side here. BUT! Simply giving yourself access to users mailboxes is a big no no. In germany at least this would be forbidden by privacy laws. Exceptions can be made, but it requires a good reason, someone from HR and someone from the "Betriebsrat" (something like a union steward) present. This has the added benefit of having someone more unbiased there.

Reply

[-]

DrAculaAlucardMD@reddit

It's fine in the US. We notify our director so he is in the loop as a CYA. Plus it's logged to show who was in what mailbox, and further logs on any actions at that time. Logs are they way. I have spoken.

Reply

[-]

_araqiel@reddit

As other people have mentioned, outside the EU, such protections are uncommon. I actually am on the side of “it’s company property you have no privacy”. That being said, I ALWAYS get executive approval in writing before granting myself access to a user’s mailbox. CYA.

Reply

[-]

angrydeuce@reddit

Here in the US corporate hosted email is the property of the business, and the end users have no right to privacy with emails within those corporate email accounts. Which to be fair is appropriate in my opinion. Company email should not be used for personal business. It is, like all the time, and I deal with it regularly when someone retires and we turn off their email and all of a sudden we find out they had all their personal banking tied to it, but it's pretty cut and dry that as admins we have the right to examine any end users mailbox at any time. I get the privacy concerns but at the same time, if the company is paying the bill for hosting it, why *should* there be an expectation of privacy? Also it is pretty clearly stated in onboarding documentation that employees emails are monitored.

Reply

[-]

craze4ble@reddit

> I get the privacy concerns but at the same time, if the company is paying the bill for hosting it, why should there be an expectation of privacy? That's not how it works in a lot of places. In all three EU countries I've worked in IT so far, anything and everything that was required for you to do your job had to be provided (equipment, training etc), and even on company provided hardware and platforms you have a basic expectation of privacy. Unless you give explicit permission, an admin signing into your account and looking through your emails would be a _massive_ violation. Regardless of what they find, that would be an easy lawsuit - you drop an email to your worker's council rep, and wait for your payday.

Reply

[-]

angrydeuce@reddit

Wow, well I guess that works for the EU. I know personally I've had my share of employee theft situations where email evidence was part of the discovery that led to the dismissal for cause. Curious how you guys do investigations like that where informing the party involved would basically tip their hand and lead to destruction of evidence? Maybe just not something that comes up because people get paid a living wage there? Lol. I could see that. Anyways I guess I partially see how it would be considered an invasion of privacy, I've just had to deal with my share of things like the theft, or more commonly, internal harassment of some sort, that required email access be granted to a manager. I have a few organizations where there are even mail flow rules in the background that send a CC of all about emails from specific position emails to a shared mailbox account specifically so management can review those emails regularly. Honestly I can't see how that wouldn't cause problems for catching people doing shit they shouldn't be doing, but like I said, maybe that shit isn't as prevalent here since people there generally don't get dicked around on wages and benefits like they do here in the land of the "free".

Reply

[-]

craze4ble@reddit

In criminal cases the evidence can obviously be gathered, but you need to have a bunch of legal CYA stuff completed first. It doesn't seem overly complicated, but you'd have legal and HR supervising that. However I can only talk based on our procedure docs, I've yet to actually encounter the situation. But what's for sure is that internal disagreements like that of OP absolutely cannot be settled like that, unless it's brought in front of the authorities for some reason.

Reply

[-]

angrydeuce@reddit

Interesting! I've had quite a few of those, myself, must be a US thing lol. Most of the people I support are in various trades and stuff like billing materials to a job and then taking it home aren't common, necessarily, but have happened enough that it really boggles my mind how people continue to think they'll get away with it. The harassment is more common. People really send some stupid ass things to each other with their corporate email. To be clear, though, I'm not the one digging (I hear about the stories second hand from their coworkers, it's rarely completely in the dark), when a C level says "give me so and so's email" we don't ask questions, we give them the email. I don't *want* to know, it ain't my business lol

Reply

[-]

jobfolio_gandalf@reddit

>union steward What Americans often call a "union representative" (or just 'rep').

Reply

[-]

mybloodismaplesyrup@reddit

Germany's laws make no sense then, because all mail sent to and from a business email account that is owned by that company is, or should be private property. If you use your work provided email to send personal messages then that is your own fault and you are violating your own privacy.

Reply

[-]

stimj@reddit

Definitely not the law in the US. The mailbox is company property, along with everything in it. And in this case there's literally no other way to prove it got delivered.

Reply

[-]

disstopic@reddit

Exchange has track and trace capability, it is possible to prove delivery and identify the location of an email without needing to view its content. Everything can be audited to an evidentiary standard. Not what happened here, but it can be done. Although, quite useless if the party the information is being presented to doesn't understand or trust the information Exchange provides.

Reply

[-]

TheBestMePlausible@reddit

Which they used to determine it had been delivered, but that wasn’t good enough. No one can search my personal email without a warrant or whatever. But company email is the company’s, pure and simple. They can do whatever they want with it, and I’m not even mad about it. Don’t put personal shit in your company’s email and you’ll be fine.

Reply

[-]

Dusk_Star@reddit

Unfortunately, the government can search your private email without a warrant if you use any sort of remote service like Gmail. Because at that point it's not your private email.

Reply

[-]

TheBestMePlausible@reddit

My understanding is google won’t give up access to your personal email without a legitimate court ordered warrant. If you have some kind of source that they don’t I’d like to see it.

Reply

[-]

TristanPhilapoe@reddit

> My understanding is google won’t give up access to your personal email without a legitimate court ordered warrant. That's what the tell you. But their "user agreement" isn't worth the electrons it's stored on. The reality is that ISPs, the major email vendors (Microsoft and Google), the phone carriers, etc will share your data in a heartbeat because it isn't YOUR data on their systems. It's theirs. And what are you to them? I'll let you Google the times all of the above have been caught just handing data to the gummit without question.

Reply

[-]

TheBestMePlausible@reddit

I’ve googled it before, and I just did again, and I don’t see anything about google handing anything over without a warrant and/or subpoena. So as far as I can tell, via my own websearch, what you are telling me is BS.

Reply

[-]

Dusk_Star@reddit

>or subpoena Guess what has lower restrictions than warrants?

Reply

[-]

TheBestMePlausible@reddit

Responding to a court ordered subpoena is absolutely not the same thing as just handing data to the government without question. What, are they supposed to just ignore a legitimate court order?

Reply

[-]

Kreiger81@reddit

Ok, then how would you have tracked and traced the deleted email that OP found? Cause I would have done the same, given myself adm access to the mailbox and gone hunting. I probably would have checked backups first to see if it was there tho.

Reply

[-]

SolidKnight@reddit

In the M365 world, you can just do a content search which is the better way to find something very specific without seeing everything in the inbox. There is also the threat explorer, audit logs of user actions (accessed, responded to even if not sent, deleted, moved, et cetera), MDE, threat explorer, and of course you can look in your backups. Emails create a big wake in the M365 world and you have a ton of options to view what happened to that email. This is why I find it amusing when people try to claim they didn't get anything. "After investigating the incident we have determined there was no failure of the infrastructure or client application. Mail was successfully composed at X time by Y person then successfully delivered into Z inboxes at ZZ time. Person A successfully accessed email at B time using their company computer and successfully deleted email at C time. The item currently resides in their deleted items folder."

Reply

[-]

Nu-Hir@reddit

It's been a while since I've had to do a message trace, but the trace should give you the message ID. From there you can do a dump of their email box which will provide you the message IDs of all of the messages in their mailbox. I've never had to trace deleted items, so I couldn't tell you how that's done.

Reply

[-]

hideogumpa@reddit

> the trace should give you the message ID. From there you can do a dump... That's how we *prove* things, but stubborn non-technical users that don't trust IT need convincing. Screenshots of their deleted items folder does that sort of convincing.

Reply

[-]

CompositeCharacter@reddit

If the recipient of the information doesn't trust the application to produce logs then the problem isn't with the admin that provided the logs, but the manager who is using an application that is in their opinion not fit for purpose. The universe isn't going to empower some IT wizard to write the evidence in the sky simply because some idiot wills it.

Reply

[-]

CARLEtheCamry@reddit

Not a law, but I know someone who was fired for breaching company policy. He was a BES administrator back when Blackberrys were a thing. Suspected his wife (also worked at the company) was cheating on him with her manager. Got into the BES server, got the incriminating PIN messages, turned them over to HR. The manager and the husband were both fired. Manager for the relationship with a direct subordinate, husband for breaking internal policies. Which then all of IT had to sign a revised data use policy to close some loopholes.

Reply

[-]

rswwalker@reddit

In that case the logs would be final. They said the email was delivered therefore the users are lying and should be disciplined for such.

Reply

[-]

Background_Dirt376@reddit

I have personally witnessed Exchange logs being incorrect. The user view and the admin view opened at the same time, with only the user's inbox missing a message. The admins at the time STILL blamed the user, somehow! Fact is, Exchange uses a deprecated database as it's storage system. This isn't an architecture that works 24/7, it has consistency problems. You have to use modern data structures to handle modern mail volumes, or things get weird. If every Exchange admin who got the "dropped message" complaint knew to investigate past the logs, we'd all be using something else by now.

Reply

[-]

rswwalker@reddit

In my experience the Exchange logs have been true. If the user has a rule that does weird things with the email after it arrives, then it is up to the user to fix it, but the email arrived to their mailbox. If you can’t trust the logs, then what’s even the point of logging? And if you can’t prove the system didn’t eat the email then everyone will think the system just eats email and management will start thinking you’re incompetent because of Exchange looses emails then Office 365 users would be up in arms!

Reply

[-]

IsilZha@reddit

Any time I've seen an inconsistency like that, I've always found that the user did something with it, or had an Outlook client-side rule that did something with it. That being said, getting into their mailbox to find it is what I'll do if necessary.

Reply

[-]

SilentLennie@reddit

> Exceptions can be made, but it requires a good reason I would say this is a good reason, but as you said, probably would need some other people like you mentioned to be involved.

Reply

[-]

So_frickin_tasty@reddit

Workplace email is subject to scrutiny in the states. It also makes sense why (given the Original post). Whether people are acting on the advantages of business emails being confirmed missing and the like is another matter entirely. Usually bad bosses just snoop and connive

Reply

[-]

shouldbeworkingbutn0@reddit

That's why you give people generic mailboxes like department + random digits (e.g. sales.0472@), and not named ones. That way there is no personal mailbox, and this argument gets rendered moot.

Reply

[-]

FeesShortyFees@reddit

I couldn't imagine working under those constraints. In the U.S., we simply expect you to use your personal email address if you want privacy.

Reply

[-]

GoodTofuFriday@reddit

I can't see how thats an issue. Any email that goes through company servers is company property.

Reply

[-]

Tymanthius@reddit

Depends on where in the world you are. US that's true. Most of EU, it is *not* true.

Reply

[-]

placated@reddit

In the US companies can do whatever the F they want to.

Reply

[-]

Stonewalled9999@reddit

>em bolster their lie. Hey OP, grant the admin account access to the other 3 mailboxes that are members of the DL and see when they deleted it. File your grievance with HR regarding the destruction of corporate information and then sit back and enjoy while individual # 1 twists in the wind and the other 3 flip on them because they don't want to be associated with an HR inquiry. Union stewards are not unbiased. At least not here.

Reply

[-]

uebersoldat@reddit

>Simply giving yourself access to users mailboxes is a big no no. If you don't have something in your IT policy that your staff have to sign that says they have ZERO expectations of privacy on a company-owned machine then you might need to look into that. We even have a mobile device policy for remote wipes if the company's Exchange account has been added.

Reply

[-]

fengshui@reddit

Certainly abuser could grant the admin permission to look in their box. In such a situation the admin would just ask "can I look in your email for it", or do it on their computer with the user watching. If they are sticking to the "I never received it" story, they would want to allow that, and would then be proven wrong in the same way. There are many ways to skin a cat.

Reply

[-]

CKtravel@reddit

All the better. Doing this in front of an HR person and someone from Betriebsrat means even more of a shitshow, especially in case either of them was to side with the lying bastards.

Reply

[-]

CaptainWart@reddit

Here in the US, you have zero rights or privacy on company owned resources. The assumption should always be that your employer/IT team sees everything you do on your device/account. I'm sure in some cases company or union policy dictate some protections, but there's certainly nothing in the law about it. The worst thing I ever came across - The company owner suspected a co-worker I worked closely with was communicating with and giving secrets away to a competitor and told me to search her email account for evidence. I didn't find anything like that, but I did find that she was having an affair with the pastor of her church. Definitely made me feel a bit awkward from there on.

Reply

[-]

ilililiiliililliliil@reddit

> Simply giving yourself access to users mailboxes is a big no no. Not as as formal an issue here as there is no expectation of privacy as you are (generally) not allowed to use business resources for personal use. I'm not saying it's a non-issue. Snooping around in user's mailboxes is a definite potential HR complaint and issue. In the OP's case he/she has a clear reason to do so - specifically to troubleshoot the issue that has been raised. OP didn't say but as long as the supervisor was informed of what was being done and the admin rights were removed from the mailboxes afterwards, it should be good.

Reply

[-]

inphosys@reddit

I certainly am on OPs side as well. That's very interesting to know about European / German law, here in the states our servers are considered corporate resources and admins are pretty much granted permission to do whatever is necessary, within reason, to properly insure that the systems are working optimally and any issue documented by an end user is not somehow part of a larger issue that has gone unnoticed. There are some exceptions to this rule when it comes to companies that work in highly regulated industries like banking and some healthcare (there are others, I just can't think of them at the moment because I'm just waking up and haven't had coffee yet), but those industries usually have a loophole or IT regulatory guideline that states something like, admin access is allowed to certain types of private data as long as there is a documented audit trail of the admin granting, accessing, and then removing access, and any information retrieved is summerized in the audit report and handled in accordance with the data governance rules. I did a job once for a university system and when I had to gain access to a secured student data repository I also had to have a second admin co-sign my report stating that they witnessed all of my actions while I had access to the data and that my report was accurate and true. Made the job take almost twice as long to complete but made me very happy about the security of sensitive, private, student records and I can definitely see how this rule could be useful in almost all industries when it comes to 'private' data.

Reply

[-]

wrosecrans@reddit

Not necessarily an HR issue. Just start asking accounting what billing code to use for charging their department for the wasted resources. If HR doesn't want those people funding IT's parties, that's their issue.

Reply

[-]

magiclatte@reddit

Definitely an HR issue. Wasting company resources on their incompetence....

Reply

[-]

GreatRyujin@reddit

This is going beyond incompetence, I'd say it's crossing into malice.

Reply

[-]

BrainWaveCC@reddit

It left the backyard of incompetence, crossed over Willful Conduct Avenue, and is now frolicking in the field of malice.

Reply

[-]

randomman87@reddit

Yeah. If HR isn't interested I bet legal is.

Reply

[-]

dyne87@reddit

Plot twist: the DL is hr@company.com. Had something similar where a new HR employee claimed that IT didn't train her how to connect to the VPN. The HR manager came to me to see if we have any documentation that shows the helpdesk team trained her. Popped into the auth portal and pulled the logs that showed her account was created the day before she was scheduled to work off site for the first time and then a couple hours later was added to her mobile app with a successful login less than a minute later. Yeah, they fired her the next day. Turns out she was already causing problems and spending an entire day off-site without working and then coming back the next day saying she couldn't work from home "because IT" was the final straw.

Reply

[-]

Cow_Launcher@reddit

> ...knowingly deleted a unwanted mail... Or - and I've seen this before - "I have an email. It is from someone unimportant and can therefore be ignored and deleted." Five hours later, the same person is screaming at the poor schlub who sent the email, "Where are the figures from July? You were supposed to send them to me five hours ago!" The arrogance is real.

Reply

[-]

Lee_Ars@reddit

> Multiple adults are caught lying about the basic fact of having received an email? And any further lip from them is being tolerated? To people like this, IT is just a bunch of computer janitors. They don't respect you and don't see you as an actual valuable part of the company—you're basically furniture.

Reply

[-]

worthing0101@reddit

>you're basically furniture You're overhead to these kinds of people and they hate paying for you because they don't understand your actual or potential value. Now that they can set up their own WiFi at home, get email on their iPad in 30 seconds, even manage their own cloud backups, etc. (all of which are positive things) they assume everything IT related at work is just as easy and question why it costs so much, takes so much time, etc. Even when you explain what you're doing and why and why it costs what it costs they either don't get it or just think you're lying or bending the truth. Must infuriating to me is they almost never think of or treat other departments this way. I've almost never heard anyone question the existence of or need for finance, facilities, security, etc. the way they question the need for IT. It's maddening.

Reply

[-]

CKtravel@reddit

There's an easy solution to ANY company that does this to you: leaving. If they don't consider your existence there important enough then you're better off e.g. managing the infrastructure of someone else.

Reply

[-]

worthing0101@reddit

I know a lot of people in this sub like to believe that anyone, anywhere, in any region, in any job market, regardless of their personal circumstances can just quit and go find another job easy peasy lemon squeezy but that's simply not always the case. For some people, even a lot of people, there's a lot more effort, risk and difficulty involved in changing jobs. I'm not saying those people are stuck forever, just that it's not always "an easy solution".

Reply

[-]

CKtravel@reddit

>there's a lot more effort, risk and difficulty involved in changing jobs. I never doubted this at all. But you should always ask yourself this: is jeopardizing your peace of mind and mental health worth it? Is putting up with the subhuman treatment of entit)ed shitheads worth the salary you're getting paid? No, it might not be easy at all, but at the end of the rabbit hole at a toxic company lies alcoholism, drugs, self-destructive behavior, depression and a declining private life (among others).

Reply

[-]

worthing0101@reddit

> but at the end of the rabbit hole at a toxic company lies alcoholism, drugs, self-destructive behavior, depression and a declining private life I actually got laid off mid-pandemic from a company I'd been with for ~15 years that I knew was toxic but I didn't realize how toxic until I got some distance from the job. I've been struggling to just be a functional adult since then and I'm definitely dealing with, or dealt with, 4 out of 5 from that list since I left. (But, weirdly, not while I was there?)

Reply

[-]

Nu-Hir@reddit

The problem is, most companies value departments based on the money they bring in. IT and Maintenance bring no money into the company, they only spend their hard earned dollars. They don't value IT because IT doesn't bring in money, despite the face that the other groups in the building are able to bring in money because of IT's hard work.

Reply

[-]

CKtravel@reddit

Yes, that's true. And it's a VERY good reason why I try to make it a point to seek jobs mostly at companies whose core business is either IT, is related to IT or very heavily relies on IT. Yes, other companies have a tendency to treat their IT departments as unwanted children.

Reply

[-]

Nu-Hir@reddit

I mean, with some companies, being treated as an unwanted child would be better than how they're treated. One of the things we're fighting for is to have a different budget for lifecycle replacements of new PCs and for new users. Near half of our PCs that we've purchased to replace old PCs are being used for new users. I've recommended that we lifecycle C Suite last so they understand why we can't have the PCs come from the same budget. "Sorry, we can't upgrade your 5 year old laptop, we had to use the last one for this new user. I have a 4 year old laptop I could let you use".

Reply

[-]

CKtravel@reddit

This kind of thing you're talking about seems to be more of a generic issue though. If IT has no budget for HW refreshment then no users get new laptops, not just IT. And apparently they aren't complaining much either. Also to be clear: what I meant is more like something along the lines of the story of the OP where users generally act towards IT like despicable bastards.

Reply

[-]

greet_the_sun@reddit

Any upper management that doesn't understand the concept of a force multiplier in the context of business technology probably got Peter Principled into the role in the first place.

Reply

[-]

TaliesinWI@reddit

The analogy doesn't work as much anymore, but I used to compare it to land lines. You don't think about the techs running the phone system until you pick up the phone and there's no dial tone. Suddenly you're VERY concerned about things.

Reply

[-]

garbageadmin@reddit

Same analogy works with the light switch.

Reply

[-]

Remarkable_Milk@reddit

Yup. I'm doing some good progress in sorting out the mess that my boss left us in Okta groups and their assigned rules while all of a suddenly I have to halt everything, because the CEO is in the lobby and the clicker wasn't pointing to the USB dongle when the speaker was trying to switch slides. "Hiiii why isn't application X accessible from my main dashboard" You fucks Because you slacked me an hour ago "HELP NOTHING IS WORKING IN THE MAIN LOBBY" sometimes, I just can't help myself and laugh on the spot I mean you can't kill my soul, just my mood buddy.

Reply

[-]

taggospreme@reddit

Reminds me of all the recent optimization at a particular blue bird

Reply

[-]

bigdeezy456@reddit

Or yesterday's jam

Reply

[-]

tankerkiller125real@reddit

I like to use the analogy of an HVAC tech. When you need them, you really fucking need them because your on fire/freezing. When everything is working fine your kinda forget they exist. But never forget, if you fuck with em, they can make you sweat until your stench is so strong no one wants to be around you.

Reply

[-]

Lakeside3521@reddit

A sad but true fact.

Reply

[-]

JSLEnterprises@reddit

office politics are the worst in any corperate environment.

Reply

[-]

Background_Dirt376@reddit

I know this is gonna be poorly received, but I've admined Exchange off and on since the 1990's. THEY MIGHT NOT BE LYING. This is classic, classic Exchange behavior. Logs say a message was delivered. It's "in the inbox," but not visible to users. Hours or days or WEEKS later, it shows up. Calling "user error" on a fussy, inconsistent, black-box server whose logs *frequently do not correspond to reality in realtime* creates a work culture where users don't trust admins, admins think users are lying, and no one's actually checking on the infrastructure software unless there's already a problem (for which we are already gonna blame the users). I stopped advocating Exchange because the platform gaslights admins and users. What happens to your workplace after that isn't the vendor's fault...

Reply

[-]

jmbpiano@reddit

> I found the message in their recover deleted items and it had been deleted four minutes after it was received. *This* doesn't sound like an Exchange issue. An "invisible" email in the Inbox, I can believe. I've seen weird Outlook caching issues cause that. An email deleted without the users knowledge due to a rogue message rule, I could also believe. But an email that was actually deleted, a full four minutes after arrival? That smells off.

Reply

[-]

draeath@reddit

Perhaps a busted client side rule?

Reply

[-]

jmbpiano@reddit

Not at all impossible, there's certainly some delays that could be introduced by the client, but that still seems an awfully long time for rule processing to take. If it were in the 30 seconds to two minute range, I would be much more inclined to attribute it to a client side rule.

Reply

[-]

dreamyangel@reddit

If you wait enouth problems will pass by. Blame it on anyone but you. After two weeks people will stop searching for answers and will move on. Except for IT guys. They will spend 8 hours a day, 7 days a week, looking to fix the issue. Don't play with IT guys.

Reply

[-]

jpmoney@reddit

[Obligatory XKCD](https://xkcd.com/705/)

Reply

[-]

databeestjenl@reddit

I had a whole department claiming they didn't get the email about a certificate expiring. We used a shared support@ group address, but unfortunately the programmers thought it handy to send all email from their reporting with the same from: address. 1st response, you didn't send the email. It's gone. reply: No, it isn't look harder. 2nd respons: Can you forward me the message reply: No, look harder, you will probably find the rest. They were not amused by my blunt response, but I was done with bullshit for the day. 3rd response, nope. It is not in our mailbox, you didn't send it. Walk up to user, permission to access mailbox, search for subject. reply: there is your email. So, because of the reporting the whole department made a rule to move all messages from support@ to a folder and none of them ever saw it. They then proceeded to blame the system for emailing out of support@ And yes, they did have a point, it turns out that over 15 years that was probably the main cause of emails from support@ to the entire organization were "lost". It explained a few incidents and awkard conversations of not being aware of maintenance.

Reply

[-]

ambscout@reddit

This is why it is always a good think to know how to review all audit logs. I have proven not my issue many times. Last week I got told some outbound emails were caught by the recipient's spam filter and that I needed to fix it. I responded to the users with a ticket from my email provider saying it isn't my issue.

Reply

[-]

qwerty_samm@reddit

Its always easy to blame IT. Stick to your guns though and make sure you keep the hard evidence to back it up.

Reply

[-]

Hjarg@reddit

It is never a good idea to blame IT. We have logs!

Reply

[-]

nibbles200@reddit

I had a co network engineer I worked with that I didn’t get along with. When I started I couldn’t believe there wasn’t a log server and all the devices were set to the lowest log retention possible. We would have these hard outages and it was my responsibility to root cause. At first I couldn’t because there were no logs. When I fight for a logging server the other engineer would gaslite the boss to convince him it’s a bad idea. Never made any sense until I made scripts to dump all logs to my desktop. I started producing root cause reports to the boss. It was all the other engineer playing in prod, trying new things or making changes on a whim. When I stated calling him out he had the nerve to actually question how I got the logs, because they should have rolled off. When I told him I was dumping them to my desktop his face went pale. He doesn’t work for us anymore.

Reply

[-]

steveholt480@reddit

...but why? Why play around in prod?

Reply

[-]

Mr_ToDo@reddit

Because they couldn't figure out how to setup a functioning test environment?

Reply

[-]

PhiberOptikz@reddit

I'll give the dude *some* leniency. I've actually been in a position where I didn't have the equipment to build out a test environment, and wasn't able to procure any. The solution I went with, was to only test outside of business hours, and never do it without having a fresh backup complete which I can roll back to. Never had a problem, but I felt part of my soul leaving me everytime I did. :/

Reply

[-]

Ryuluce@reddit

Something I'm glad about is that the application I support has not just prod. Not just prod and test. But ***fourteen*** environments. * Production and one backup copy * Several feature development environments * Several integration testing environments

Reply

[-]

PhiberOptikz@reddit

One may say 14 environments is too much. A seasoned sysadmin may say not enough. XD May be a lot of work to maintain, but that's worth it in my eyes.

Reply

[-]

Ryuluce@reddit

Yeah, issues in prod are rare. Though when prod *does* have a problem, that's all-hands-on-deck fire mode.

Reply

[-]

doweactuallycare@reddit

Saaaaaaaaame. Most places i've been just literally don't have 'test environments' at all, or they're not suitable whatsoever. I've done far more work on live production than in testing. The reality is, most places just don't have full testing environments and everyone can look at their jobs right now and i'm betting 50% who read this do NOT have suitable testing environments and theres always a percentage of "fuck it, we do it live" going on. Most are struggling to get production equipment replaced. Let alone enough allowance to have a fully functional testing environment that actually fits the production environment. Theres been plenty of times i've tested things in my homelab with fake data. I think even that is more than workplace testing environments. I hate it.

Reply

[-]

sagewah@reddit

> a percentage of "fuck it, we do it live" going on. More "If it doesn't work, it's dev; if it does, it's now prod".

Reply

[-]

doweactuallycare@reddit

Everyone has a dev environment. Some people have a seperate prod though.

Reply

[-]

PhiberOptikz@reddit

It's truly tragic just how widespread the problem is. Far too many people with pruchasing approval powers don't understand their company's IT needs, let alone the IT department's needs.

Reply

[-]

Oblong_Gatta@reddit

The concept of test environments was completely new to me until my current job. Prior to that, it was all Prod, and we had to fight to get new equipment. For the last 10 years I was there, I was finally able to double up on everything just for redundancy

Reply

[-]

mnvoronin@reddit

*Everybody* has a test environment. Some of us are lucky to have a separate prod environment.

Reply

[-]

davidm2232@reddit

Because there is not nearly enough time in the day to build and maintain both a prod and test environment. I was the only sysadmin for a small bank. We rolled out changes as slow as possible but it was always in prod. Just not enough resources to have a test environment. Luckily some of our new apps allowed for a UAT group which was awesome. But I would hit the limits of our SAN and hypervisors weekly and have to shuffle stuff around to make just prod work. Forget about having space to run a test environment.

Reply

[-]

TightBed8201@reddit

Arent you bound by regulators to have peoper support for your systems?

Reply

[-]

davidm2232@reddit

The regulations are very loose. We are pretty strict on security and data backups. Otherwise, they are very lenient. If we had outages all the time, management would have more to say than regulators. I was there 5 years and did 95% of our roll outs in the prod environment. I think we had 2 occasions where it went bad. One was a half hour of complete network unavailability and one was an after hours Exchange upgrade that put email down from 4pam to 8am. But we only had one Exchange server so no real other option when the CUs had to be installed. I didn't even know CUs existed for the first 3 years I was there so I had to do like 5 over a month to build back up to the current level.

Reply

[-]

TightBed8201@reddit

Dont know if you fall under pci dss and whatnot, buz my employer would be chewed up if we end with one person at support.

Reply

[-]

davidm2232@reddit

We did not fall under pci. It was a comparatively tiny bank so we got away with a lot. Now that I no longer work there, I pulled my money out. For several reasons.

Reply

[-]

BrainWaveCC@reddit

All that is fine and dandy, but when the approach that is being taken starts to have clear, adverse impact on the production environment, then it's time to use a more professional, disciplined approach. I was able to justify numerous lab / test environment (in whole or in part) because of persistent issues like the one described by OP. Shoot, even a functional change management program would minimize adverse customer/production/business impact.

Reply

[-]

davidm2232@reddit

The CEO of that company wanted to 'shut down IT' and go back to paper. I found a new job.

Reply

[-]

steveholt480@reddit

I get that, but it’s a far cry from messing around and deleting logs to cover your tracks.

Reply

[-]

dans_cafe@reddit

isn't deleting/editing logs a crime because ENRON?

Reply

[-]

INSPECTOR99@reddit

He did not CREATE logs....ergo, nothing to delete :-) He just obfuscated his chicanery into oblivion.

Reply

[-]

SuspiciouslyMoist@reddit

Depends on where you are, and what they're logs of. In some cases, having a low retention time isn't as bad as having a high retention time and actively deleting things. This isn't about logs, but in my organisation we currently have a headache caused by past mistakes: if you want to archive stuff, archive it - don't just keep backups indefinitely. We have *lots* of backup tapes containing a mix of stuff that we're legally obliged to keep for X years and stuff that we're legally obliged to delete after Y years (where Y<X).

Reply

[-]

Tymanthius@reddit

I'm right there with you. But also, that doesn't appear to be the problem in the above story.

Reply

[-]

fatcakesabz@reddit

everyone has a test system just sometimes its not seperate from prod....

Reply

[-]

calcium@reddit

It's faster then QA

Reply

[-]

mgb1980@reddit

It’s all good until the reveries trip self-awareness and Delores starts shootings real bullets.

Reply

[-]

Geno0wl@reddit

...except for the fact in that instance that is what the dev actually wanted to happen...

Reply

[-]

mgb1980@reddit

Indeed. In that instance it was all correct, but not necessarily good, but that’s subjective 😁

Reply

[-]

lando55@reddit

But still, no dev sandbox? Guy is hardcore if he was that confident going straight to prod

Reply

[-]

mgb1980@reddit

Who knows. He has lots of spares down in cold storage, plus his own little production facility in his replica house, off-net.

Reply

[-]

JSLEnterprises@reddit

sometimes qa isnt even near close to the prod environment setup. al just a clusterfk and ad is stoic for any changes because they honestly dont even know they systems they work on.

Reply

[-]

TrainAss@reddit

Why QA when the users can do it for you?

Reply

[-]

flugenblar@reddit

In the words of the World's Most Interesting Man, "I may not test all my changes, but when I do, I test in Production."

Reply

[-]

Nu-Hir@reddit

All companies have a test environment. Sometimes it's separate from Prod.

Reply

[-]

dam_broke_it_again@reddit

But frequently has the same EXPECTED SLAs' as prod :(

Reply

[-]

hardolaf@reddit

Downtime in dev is often just as costly as downtime in prod.

Reply

[-]

pdp10@reddit

Most organizations don't have substantial non-prod environments when those test environments cost money and/or time. Organizations that choose to buy $50k load-balancing appliances, surprisingly usually don't buy six more identical units for non-prod environments. Another thing you tended not to see duplicated in test environments was paired Fibre Channel switches.

Reply

[-]

Posting____At_Night@reddit

Serverless/cloud envs, for all their faults, are really good for this. I can spin up an entire fresh test stack in under 15 minutes for nearly free.

Reply

[-]

lesusisjord@reddit

Because we have full Dev, QA, UAT, and Prod environments in Azure for all our apps (we are in the healthcare/health insurance space so our clients have the money to pay for this infra), I fortunately never had to spin up a short term test environment from Prod. You mention the near-free aspect of doing it that way which is a HUGE plus for cloud-based environments. If you only need the resources to exist for as long as testing takes (might be only a few hours), you aren’t going to generate much of a bill for the short term rest environment.

Reply

[-]

Posting____At_Night@reddit

We still have the persistent dev/prod/qa envs but being able to spin up a deployment for any given feature branch has proven to be extremely useful for allowing devs to have a live environment to test their changes without stepping on anyone else's toes.

Reply

[-]

af_cheddarhead@reddit

If you are talking F5 then they offer a $75 lab only license for a VM based appliance that is fully licensed for all functions. It's about the only good thing I have to say about F5.

Reply

[-]

pdp10@reddit

That's a good option, now, as long as you architected your production infrastructure to use virtual appliances, so prod could be identical to dev and QA environments. I just use HAproxy, Varnish, Stunnel, *etc.*.

Reply

[-]

af_cheddarhead@reddit

Fortunately (sort of) my prod environment minus the physical switches is in vSphere. But yes that can be an issue.

Reply

[-]

secretlyyourgrandma@reddit

move fast, break stuff, leave a smoking crater

Reply

[-]

DJPalefaceSD@reddit

If it works then you're done and can go home early

Reply

[-]

richhaynes@reddit

Sounds like a classic case of someone getting a job because of connections rather than being good at it.

Reply

[-]

Zoltan_Kakler@reddit

Dumbassness

Reply

[-]

Aim_Fire_Ready@reddit

>When I told him I was dumping them to my desktop his face went pale. TFW you know you're busted!

Reply

[-]

sirsharp@reddit

It was once said in here I don't know who said it but it stuck. "Everyone has a test environment. Some are fortunate to have production also."

Reply

[-]

1z1z2x2x3c3c4v4v@reddit

Had the same thing happen with almost the same result. Logs are a wonderful thing. Only I got a promotion when the "Senior" a-hole was fired...

Reply

[-]

bionic80@reddit

I dealt with that with our ops team recently. they would get a ticket, and 5 minutes later push to engineering with a laundry list of their "troubleshooting" - I proceed to dump the logs showing that not only did no one log into the server(console or otherwise), but the last person who DID log in was over a month ago (me, as it were). I presented the logs and our additional NSXT logs showing there was zero 3389 traffic to that server. It is good to know where to go for logs.

Reply

[-]

anonymousITCoward@reddit

Similar happenings here... until one day... I got tired of being blamed for shit and not having the logs to back me up, I took it upon myself to get logging to where I needed them, not where they should/could be, but just enough to prove my point. Then was able to prove that it wasn't my fault, or the fault of my department... everyone but us were pissed that I did that... the big ego has hunted down and killed off my logging, and removed access for me to do what I need to do, so I now enjoy being blamed and reminding people that I had my access revoked... and if they wanted my help they need to first give me access, and second, let me log what's happening... still no access =)

Reply

[-]

rohmish@reddit

The thought process of some people honestly baffles me

Reply

[-]

uebersoldat@reddit

Justice was served! Well done!

Reply

[-]

sagewah@reddit

It's fun when people try, gives me an excuse to go full CSI on the logs to prove someone need a-firin'.

Reply

[-]

flugenblar@reddit

it's 2023 and people still do not understand this. there was a guy downloading porn on the company network, taking up all the bandwidth. he had even been caught once before doing this. but... he got caught again. as if... as if nobody watches for that kind of thing. shees.

Reply

[-]

Myte342@reddit

Litigation Hold in 365 is fun. People think they deleted an email and we can pull it up years later.

Reply

[-]

Oshkalan@reddit

Business: I don't trust the logs

Reply

[-]

p4ttl1992@reddit

BuT the LoGs ArE BROkeN tOo

Reply

[-]

ivanraddison@reddit

They're only broken when you need to save your ass 😬

Reply

[-]

kremlingrasso@reddit

this is one of those "the wizard always arrives on time" things...

Reply

[-]

archiekane@reddit

But the email didn't, "apparently" .

Reply

[-]

kremlingrasso@reddit

the email does or doesn't arrive precisely when the wizard wants it.

Reply

[-]

ChefBoyAreWeFucked@reddit

Also, the wrong file didn't arrive.

Reply

[-]

samurai_ka@reddit

Sounds like the MAGA playbook 101

Reply

[-]

BoulderBurger@reddit

Fuck off loser

Reply

[-]

samurai_ka@reddit

No thank you. Unlike the leader of the MAGA cult, who is a convicted sex predator, I don't want to do your mother.

Reply

[-]

RedFive1976@reddit

No he wasn't. Being found financially culpable in a civil court is not the same thing as being convicted of a sexual crime in a criminal court.

Reply

[-]

Iintendtooffend@reddit

Yeah, but he's still a sexual predator

Reply

[-]

RedFive1976@reddit

Is he? Lack of convictions would seem to indicate otherwise.

Reply

[-]

Backitup30@reddit

He literally admitted it in that town hall, LOL what are you talking about? There are people like you still out there?

Reply

[-]

RedFive1976@reddit

Bullshit. If he had, it would be all over the news. But instead, they're still laughing about him crying about election fraud.

Reply

[-]

Backitup30@reddit

It was all over the news, just not your news because people like you are so fragile you would change the channel if they said something bad about him. The same reason you can’t seem to accept reality that he did it is the same reason Fox just lost $870,000,000.00 in their Dominion lawsuit. They got caught lying to people like you and since that didn’t change your mind yet again, the only conclusion is that you have made so much of your personality based on these politics that your psyche would literally crack if you admitted you were wrong. It would be fascinating if it wasn’t so…. Sad?

Reply

[-]

KakariBlue@reddit

And also that you can only be something negative if you've been convicted of it.

Reply

[-]

Iintendtooffend@reddit

even if he had been convicted of it, they'd be loop holing how that one doesn't really count because of some reason or another.

Reply

[-]

RedFive1976@reddit

> It was all over the news Again I call bullshit, because everyone is talking about the "election fraud conspiracy" and not wanting Ukraine to win the war, and being pissed off at CNN for even giving him the time of day, but not one word about this supposed "admission". If it's true that he said that, you would have no problem adding a link to some video, would you? > They got caught lying Again, there was no admission of any guilt, there was a settlement from Fox outside of court. > you have made so much of your personality based on these politics What makes you think you, some random internet idiot, know anything about me?

Reply

[-]

KakariBlue@reddit

>> They got caught lying > >Again, there was no admission of any guilt, there was a settlement from Fox outside of court. So Fox wasn't in the wrong they just made a small donation to a company they'd reported on in the past? And there definitely wasn't a judgment stating: >While the Court must view the record in the light most favorable to Fox, the record does not show a genuine issue of material fact as to falsity. Through its extensive proof, Dominion has met its burden of showing there is no genuine issue of material fact as to falsity. Fox therefore had the burden to show an issue of material fact existed in turn. Fox failed to meet its burden. The evidence developed in this civil proceeding demonstrates that is CRYSTAL clear that none of the Statements relating to Dominion about the 2020 election are true. Quote p42 of https://www.documentcloud.org/documents/23736885-dominion-v-fox-summary-judgment I also recommend the section starting on 17.

Reply

[-]

IWASRUNNING91@reddit

I'm surprised the backup plan to scream "Fake news!" hasn't come in yet. "Tucker left of his own accord because he has a bigger following than all of Fox. Him and Trump will start a super network of truth!"

Reply

[-]

Superb_Raccoon@reddit

That was not the claim. You sound like the users in the OP story... moving goalposts.

Reply

[-]

Iintendtooffend@reddit

I'm not referring to the claim, I'm referring to you know, they things he's said out loud, with his mouth, and admitted to doing.

Reply

[-]

Superb_Raccoon@reddit

YOU were. The OP was not, he said "convicted" and you moved the goalposts.

Reply

[-]

Iintendtooffend@reddit

if the baseline for someone to be a sexual predator is a conviction, then anyone who is a sexual predator, but just hasn't been caught and convicted, isn't by their definition a sexual predator. Is that the line in the sand you want to draw? No one is a predator unless they get caught AND convicted?

Reply

[-]

Superb_Raccoon@reddit

That is the LEGAL definition yes. That is the basis of our legal system, innocent until proven guilty. That was the original claim, CONVICTED. But in your head it is backwards, guilty until proven innocent. Man, you gonna be mad when people apply that standard to politicians you like and Not Orange Man Bad.

Reply

[-]

Iintendtooffend@reddit

Nope, I won't be mad, I'll be disappointed, and then hold them accountable for their actions. The original claim was not convicted, that was what the line that /u/redfive1976 drew as what was needed to qualify him as a sexual predator. I don't need that qualification, because I believe him when he says he's been a sexual predator.

Reply

[-]

ALurkerForcedToLogin@reddit

If you knee-jerk react with hostility to everything you see that you don't like, you aren't going to make it in IT, and you are going to be eaten alive on the internet, especially on Reddit. Besides, the "missing" email is for some random company, not Hilary, so you haven't blown the case open yet.

Reply

[-]

BoulderBurger@reddit

I've been in IT for a long time and I'm not going anywhere.

Reply

[-]

xGovernor@reddit

Butthurt

Reply

[-]

HellishJesterCorpse@reddit

I don't know what's funnier, the fact that the comment is accurate, they do destroy evidence, or how easily and bigly, old mate here was triggered...

Reply

[-]

Benutzernutzer@reddit

This is a sysadmin forum. You country's political shitslinging can stay where it belongs, which already is the entire rest of reddit. Yes, this "triggers" me.

Reply

[-]

VCoupe376ci@reddit

Says the person that supports the party of: “Did you wipe your e-mail server?” “What, like with a cloth or something?”

Reply

[-]

HellishJesterCorpse@reddit

Mate, I'm Australian, I don't support any of them, but it's funny how easily triggered the MAGA folk are.

Reply

[-]

Get3DPrint@reddit

What email server in the closet?

Reply

[-]

GhostNode@reddit

Settle snowflake.

Reply

[-]

tuxedo_jack@reddit

Sysadmin law 2: >Users lie. Logs don't.

Reply

[-]

mpdscb@reddit

And the root password.

Reply

[-]

Ron-Swanson-Mustache@reddit

And they're big, heavy, and wood!

Reply

[-]

hva_vet@reddit

And long memories.

Reply

[-]

Bradley-Mayhem@reddit

this

Reply

[-]

DigitalWhitewater@reddit

[everyone needs a log](https://youtu.be/OSBCU7-iBX4)

Reply

[-]

ripples2288@reddit

CYA before all else.

Reply

[-]

pier4r@reddit

strangely though, while IT should defend itself and prove everything, the accusers have only to make easy claims. If the justice system would work like this (and it is already far from perfect), it would be just hell.

Reply

[-]

NaiaSFW@reddit

Everyone would be in Jail or burnt at the stake.

Reply

[-]

RevWaldo@reddit

They forgot the most important thing - https://youtu.be/WXe4aV7y7eY

Reply

[-]

set-271@reddit

Like over 15 years ago, I received an emergency phone call at 7pm that one of my museum client's vip board members couldn't email the Director because the exchange server was down. I panicked, troubleshooted for a half hour, then contacted the VIP Board Member on the phone to see what was up. He's a really snobby, uppercrust, rich asshole who always talks down to everyone and took the position that I should be grateful to him because he awarded me the IT contract. Anyways, he kept saying the email he sent to the Director bounced back, therefore the email server is down. So I asked him to forward me the bounceback message, which he did, and all it was, was a misspelled email address. So I didn't bother calling him back and just replied to him, "you misspelled the email address, watchout for typos on your Blackberry, happens" and didn't think twice about it as I had other shit to do. Well, my response suddenly caused a flood phone calls from the Director berating me..."How dare you accuse Tom, our board member, of misspelling an email address, do you know who HE is?!!!" Then, a flood of emails saying the same thing. Then Tom, the board member called me all uppity, proclaiming it was impossible for him to make such at mistake and that it was somehow my fault...??? This went on till about 11pm and all I kept saying to everyone was, just spell the email address properly and this problem won't happen again, which just further infuriated everyone and just further infuriated me. And it never has never ended since. 15 years later, just last Christmas, the museum had a holiday party and I reluctantly attended, as I find their whole staff just plain stupid like Tom. And there was Tom, the stuffy board member, making the rounds with a wine glass in his hand, acting, the refined, important, wise sage with his refined beard, suit, and spectacles. And when he saw me, he went right back at it again with me, mocking me in front of everyone..."Ah yes, the one who crashed our email server!" I countered bluntly, "Ah yes, the idiot who can't spell." Call me rude, but after 20 years in NYC dealing with similar Tommy boys, I just dont give a shit and call it not just as i see, but how it is. And I ended my contract with them last February, which they were absolutely livid over, because their egos are so fragile, they would've wanted to fire me first. I'm hear to tell you right now, get rid of that lying client asap. Despite all the woke statements professing otherwise, "it doesn't get better. It only gets worse". Fuck em.

Reply

[-]

newton302@reddit

>And there was Tom, the stuffy board member, making the rounds with a wine glass in his hand, Excellent writing, you!! I worked IT for a nonprofit for many years, and that dynamic with the board absolutely sucked. I feel like if it came to it, my boss might have been willing to kill me in order to save face with the board. Anyway I too believe there is a very special place for people who gaslight, sucker punch, otherwise-slander or underfund IT.

Reply

[-]

set-271@reddit

Amen brother. Amen. There are two things I'll add... One, pretty much most 501c3 nonprofit museums in NYC are tax shelters for big corporations...in my case, a Big Agriculture magnate. Once I learned about that fact, it really opened my eyes to what was really going on with the museum client, and how the Director was in on the scam. Two, every year the museum client held a fundraising gala, which I never attended, but I would donate 1 month of what they were paying me out of goodwill gesture. Yes, they were such a jerk client, but back then, I was young and had the attitude I could kill them with kindness...which ultimately never worked because in the end, they always treated me like shit and gaslighted me with endless supposed tech issues. And it was really only till a few years ago, when I was serving their office computers after hours, that I learned I was one of the biggest donors of the gala. The museum's accountant, who was the only kind person working there, was still working late at the office and greeted me with a smile and said, "How come you never attend the gala?" I told her, "Gala? I never get an invite." Then she told me, "Never get an invite? You're one of the biggest donors! Everyone else just donates $200, including Alec Baldwin!" Though I smiled, pretending to be oblivious and care less about what she just said to me, it really felt like someone punched me hard in the gut. It doesn't get better. It only gets worse. Fuck em.

Reply

[-]

shanghailoz@reddit

Its not what you say, its also how you say it. A bit of tact goes a long way. Hey, from what I can see, its possible that a typo has been made, have you tried [correctspelling@domain.com](mailto:correctspelling@domain.com) to see if that works? Let me know if you continue to have problems, thanks. Don't put the blame on a specific person, give them an out to save face. I agree, people should be able to take criticism, but most can't, so you need to be smart about phrasing unfortunately. Its a life skill you need to learn, and just as important as "being right".

Reply

[-]

newton302@reddit

>have you tried [correctspelling@domain.com](mailto:correctspelling@domain.com) to see if that works? And here’s a variation on the response you’d likely get: “This is a very important person I write to REGULARLY. I find it annoying that I am being held responsible. ED I have cc’d you on this email. Can we get the issue resolved?” A careful read of the story told by set-271 exposes the board member as unwilling to assume responsibility in any way, for reasons they alone choose. No amount of ass kissing makes a difference with a person like this. I hope you never have to feel it.

Reply

[-]

set-271@reddit

Well, I used to have your mindset...I always believe in being tactful (still do in many ways), and I always left people a huge out (as well as the benefit of the doubt) when issues as such came up. And then I moved to New York City. Being tactful here or playing easy going or the nice guy doesn't work at all in NYC whatsoever. Time is of the essence in New York, and more importantly, your time is your money...so it's always best to be short, direct, and to the point. And document everything. Still be polite, but be stern and direct call out the b.s. as it happens, immediately when it happens, otherwise 9 times out of 10, it will escalate. And something I never did before, but learned always to do now, which absolutely works is to always place blame on the person when blame needs to be placed on them, because if you don't, that person will just see that as an opening to the blame unjustly back on you (which the Board Member did for 15 years). I had to say what I did at the Musuem holiday party in front of him, to not give him an out, and to call him out on his bullshit, in front of everyone. Simple as that. I only regret not being more stern with him when his b.s. issue first came up. You have to let people know their petty dalliances takes up people's time and costs money, cause if you don't, they will take up all your time with even more of their bullshit, which will just cost you money. Perhaps being tactful works in your neck of the woods, but it definitely does not in New York City. And I make no apologies.

Reply

[-]

newton302@reddit

"...tax shelters for big corporations..." Yes, Greenwashing...

Reply

[-]

newton302@reddit

Greenwashing, they call it

Reply

[-]

rgb255469236@reddit

Man this really touched a nerve. Really fucks me off how people who are good, want others to have a good time (and therefore go out of their way to help them) get treated like shit and trampled on in our society. You have to be a shark.

Reply

[-]

set-271@reddit

Thank you. This New York, where everything is a shit show hierarchy and nobody respects anything but big money. And the thing I've learned is, they are very clever at getting it via 501c3 non profit tax shelters, cultural galas where they honor themselves, fundraisers, etc. It took me a few years after I learned about being the Gala's biggest donor, that they really were just shitting all over me. When I gave them sudden termination notice, they were mad and threw all these b.s. accusations at me down to the very last second I walked out of their offices after handing over network passwords. Now, when I see all their press photos online and in the news, I just think they are scum and are running a very clever racket. I'll rephrase and add to what I said previous, as I am still an optimistic person... It doesn't get better. It only gets worse. So get rid of them and get out... and work only on bettering yourself. 🔥👊🔥

Reply

[-]

Bradddtheimpaler@reddit

My only experience working for a “non-profit” was an absolute disaster. You’d be putting “non-profit” in quotes as well if you’d seen the Porsches and Mercedes in the parking lot. Should have known from that alone something was off.

Reply

[-]

countextreme@reddit

But they always have no IT/infrastructure budget available.

Reply

[-]

LetMeGuessYourAlts@reddit

Same for mega churches. Something about not paying taxes and being able to tell everyone else that they shouldn’t be in it for the money and thus accept lower wages draws the shadiest people.

Reply

[-]

Bradddtheimpaler@reddit

Yep. It was my first IT job. Parked next to those getting paid $38k.

Reply

[-]

tankerkiller125real@reddit

I worked for a school system where the superintendent was a snobby piece of shit just trying to enrich herself. She tried at least twice to threaten me and my job because I wasn't "doing my job fast enough". Both times I reminded her that she worked for a public school system, and every email she had ever sent and received was public record that could be pulled by a news agency at anytime which shut her up. When I quit I did send an email to the local newspaper suggesting that they should put in a FOIA for the superintendents email. As far as I know they never did do anything with it, but I have no doubt that she was up to some shady shit.

Reply

[-]

pdp10@reddit

> "Ah yes, the one who crashed our email server!" Your principal was repeating untruths among an audience where you might not have attended. This is how people believe untrue things. The audience doesn't know better, and what's more, they don't care to find out the truth, but they will remember the lie. Besides having logs and hard data that nobody cares to examine, I suppose the only way to counter this sort of thing is to get out in front of it with good PR.

Reply

[-]

seizethecarp_1@reddit

what's also crazy is him holding onto this interaction for 15 years this little event lived rent free in his head is almost old enough to graduate high school and it's still the thing he associates with set

Reply

[-]

set-271@reddit

Let me just say, these people are so rich and entitled, they have all the time in the world to hold grudges against people who call them out on their frivolous bullshit. It's a weird dichotomy, because yes, it lives rent free in their head, but they can afford the rent because they have nothing else better to do, than to fight over petty issues.

Reply

[-]

set-271@reddit

Thank you...your absolutely right. Thing is, the museum never cared about facts. Ever. It was all about status and pecking order. That is only what they listened to and you really can't beat it as an independent consultant. I have plenty more stories about them, like when I documented wrong doing by a Department Head (she was erasing another department's files), only to be berated for accusing such an important person, and then nothing ever being done about it. It just magically never happened again. At the time, I was younger and needed the client. And it took me a very long time before I came to my senses and realized their shenanigans were organized and deliberate. I only wish I got rid of them sooner. It doesn't get better until you get rid of them!

Reply

[-]

DrAculaAlucardMD@reddit

"Tom, do me a favor, read line X of the email where it says the address. Read it slowly as if you were talking to a child of a low income household." Oh hey, that looks wrong. "There ya go, fix the spelling and you'll be fine."

Reply

[-]

set-271@reddit

Haha....I get you. And trust me when I say I did that with Tom and the museum staff already and forwarded the bounceback email highlighting the misspelled email address. But Tom is Tom, oh so important Tom, with his ivy league degree and family money so all Tom had to say (as well as everyone else at the museum) back to me was "well, how could've it been misspelled when it always worked before? It sounds to me like something is wrong with your system? Maybe Microsoft Exchange is the wrong solution for us, because I've heard Gmail is so much better." It's an endless gaslightiing situation that never ends. It's so silly, yet took up so much of my time when I was younger. Other clients would do this too, but the museum hands down was the worst, as the entire staff of 50 people, many of them with Ivy League degrees, would do this in rotation, claiming they couldn't get their work done because some existential problem that did not exist with the network. And it never ended, just continuously repeated for 15 years till I grew the other half of my brain and terminated my contract. Thus my mantra, it doesn't get better, so get out! No amount of money is worth putting up with these time wasters. Fuck em!

Reply

[-]

Prudent_Highlight980@reddit

>Despite all the woke statements professing otherwise I'm so sick of this word. Why is everything someone doesn't like "woke?"

Reply

[-]

set-271@reddit

sorry about that...i was doing a play on the often used phrase/slogan "it gets better"...

Reply

[-]

NotSinceYesterday@reddit

Haha, I've had similar before and just added the misspelled address as an alias. Fixed the server!

Reply

[-]

chillyhellion@reddit

Except that there is now precedent for you to add every variation of every user's email address in the future. After all, you fixed it for so-and-so!

Reply

[-]

sunny_monday@reddit

Ive done it.

Reply

[-]

wallacehacks@reddit

Clever but infuriating.

Reply

[-]

syphilicious@reddit

Have you considered a career in consulting? This is top tier problem-solving lol.

Reply

[-]

reddit-dip-dip@reddit

But he forwarded email to you that show the wrong spelling , no? I know I know, just been "your system sucks !!" near personal attack last week, all turn out the attachment was too big that even gmail don't bother to forward in. I kept telling him (including poker faced boss), when email bounced, there is an error code prefixed to it, telling you why. just tell me the code ... but nobody listen. When you don't need the job, the decision is easier.

Reply

[-]

set-271@reddit

Didn't you know?! You're supposed to call Gmail and get it fixed!!!!

Reply

[-]

reddit-dip-dip@reddit

The chat robot would outsmart me when complained

Reply

[-]

Nu-Hir@reddit

The chat robot always asked me if I googled the answer first.

Reply

[-]

countextreme@reddit

I generally try to offer people like this an out because they are physically incapable of admitting to their own mistakes. Something like highlighting the incorrect email in the logs and saying "It looks like the email address was somehow misspelled. It's possible that it got put into your contacts incorrectly, but if it's resent with the address corrected, it should go through." Note how I didn't use the word "you" anywhere in there and I offered a plausible sounding alternative bullshit explanation for how it could have happened. The second you imply they are part of the problem rather than the problem just existing on its own is when they go on the defensive.

Reply

[-]

ThatCrankyGuy@reddit

I do not condone violence, but I would've choked that old fuck with my own hands.

Reply

[-]

McGarnacIe@reddit

This was a well written and fantastically entertaining read. Thank you for sharing.

Reply

[-]

set-271@reddit

You're welcome! I'll add, part of the board member's last name rhymed with "fraud", so I always deliberately mispronounced his last name accentuating the word "fraud". Haha

Reply

[-]

UbiquityDDD34@reddit

Bruh do you even PowerShell? Jokes aside even though search-mailbox is deprecated it’s still a useful tool. You can also use compliance center if it’s Exchange Online. In these situations it can save a lot of time.

Reply

[-]

almathden@reddit

\> I gave my adm account access to the mailboxes and logged into each one and found the message  Good win, take the W, but isn't this exactly what purview/compliance is for? And also limits the scope rather than exposing yourself to other content

Reply

[-]

EnusTAnyBOLuBeST@reddit

Can I throw a small monkey wrench in here: [Similar Issue](https://www.reddit.com/r/msp/comments/13ry89n/outlook_dropping_emails/?utm_source=share&utm_medium=ios_app&utm_name=ioscss&utm_content=2&utm_term=1)

Reply

[-]

sergiucrs@reddit

Daily story at my company

Reply

[-]

fliegende_hollaender@reddit

A new colleague in our IT Support dept once clicked on a phishing link in an email (the link was blocked by the antivirus). Then she got scared, deleted the email pretended it never existed. But we knew about it because of antivirus report. So I asked my Windows admin to dig up the Exchange logs, and the new colleague was asked once again if she wants to tell us something. She didn’t. Her boss then proceeded to show her the logs proving she is lying, and fired her on the spot.

Reply

[-]

lurkeroutthere@reddit

A lie unchallenged becomes people’s truth.

Reply

[-]

DrAculaAlucardMD@reddit

Politics in a nutshell.

Reply

[-]

MrYiff@reddit

Yeah, this is something ive seen a few times before and people always forget IT has access to everything. One thing I would suggest for future reference is to avoid giving yourself direct access to peoples mailboxes (and definitely don't tell the user you did this as they will just try and insist it was IT that 'planted' the missing email back), instead use the Exchange E-Discovery feature, this lets you run searches on mailboxes and then copy any results to a special discovery mailbox (which includes the folders they were found in), so you get the same results but you can do it in a HR safe manner. If you haven't used ediscovery before you may need to be granted the additional role in Exchange as by default even Org Admins don't have it assigned automatically.

Reply

[-]

DrAculaAlucardMD@reddit

E-Discovery is amazing, and stupidly powerful. 10/10.

Reply

[-]

jimicus@reddit

It’s not that they forget. It’s that they honestly don’t know. I have literally had to explain to people that yes, I can read their files because I need this ability in order to back them up. You want to guarantee I can’t, you encrypt them - and accept the risk that if you lose the keys, you lose the files.

Reply

[-]

Myte342@reddit

General rule to live by: If you do not own it then assume whoever does can see everything you do. You do not own that email account that PC, that internet connection etc etc. Assume that the company that does own it can see what you do and act accordingly.

Reply

[-]

jimicus@reddit

Wise words. But it's astonishing how proprietorial people can get over their PC. I've seen an end-user come very close to submitting a formal grievance because I had the temerity to log into his PC under my own user account.

Reply

[-]

SoftwareHitch@reddit

Then they proceed to try and store all their files on USB and wonder why the drive stops working and they lost all the important things they were working on. It’s because you included the 100gb of rips of movies we didn’t realise you have on your company computer, *Claire*. These drives can only be written to so many times and when you delete and copy all your files from and to it six times a day, you’re really running those operations low. Lucky we don’t log it as data exfiltration and block your account’s ability to use USB ports.

Reply

[-]

lukasnmd@reddit

Oddly specific, Claire.

Reply

[-]

SoftwareHitch@reddit

This is a true story, the name was changed to protect the guilty though.

Reply

[-]

rjam710@reddit

This. Giving myself access to a users mailbox feels wrong, and can be argued that you could have interfered. Ediscovery tells you exactly where the message is, let's you download a copy of it, all while never having to open that user's mailbox yourself. One of my favorite "told you so" tools lol

Reply

[-]

Rippedyanu1@reddit

Ive only done it to clear out read only shared mailboxes of spam emails with malicious attachments. Otherwise yeah it feels wicked weird to do.

Reply

[-]

galland101@reddit

We had a high-level user who was using their Deleted Items folder as a "triage area" for his e-mails instead of, you know, a regular folder. One day someone changed the retention on Deleted Items to be a lot shorter and it purged a majority of his e-mails. He got pissed. I don't understand how some people couldn't be bothered to learn the correct way to do things and instead just lazily do stupid things like this.

Reply

[-]

swimmingpoolstraw@reddit

So typical.

Reply

[-]

Bagel-luigi@reddit

Had a similar scenario here last year. 5 different people, multiple emails sent directly to them all, fast forward 4 months they claim they hadn't received anything (despite 2 of them replying once) Remoted to each computer, found the email in their email archive (auto archiving after 3 months). Showed them the date each of them opened the email, and all 5 of them stuck to the story that they hadn't received it.......While I'm remoted to 3 of their laptops physically showing them that they've received it and opened it. My team and my boss backed me up pretty well and nothing came of it despite various service complaints against me, but really leaves a sore taste whenever I have to deal with those 5 people now. 4 months of multi departmental issues cause those 5 people couldn't follow the simplest of instructions I sent them multiple times. As simple as 'click here, then here, type username, recieve email, click the approve button" People mannn, people

Reply

[-]

Capable_Buddy_5683@reddit

IT gets the blame for everything...It's always the best when they blame IT yet have failed to let the IT department know there is actually an issue...That's one of my favorites...Mind reading is not one of our strong points!!

Reply

[-]

Geralt_Amx@reddit

Same happened to me once, luckily had the email traced at Mimecast and O365 and it showed that it was delivered successfully to all the recipients. The people who received the email deleted them from their mailbox and from deleted items but we had substantial evidence that it in fact was delivered.

Reply

[-]

pertymoose@reddit

>Now the department is blaming us because the file doesn’t have the right content- sorry that is not even remotely the IT department’s responsibility- please find another way to spin it how you f**ked up and don’t blame IT! This is the very first skill a politician learns. Lie, get caught, misdirect, new lie.

Reply

[-]

StoicBloke@reddit

in their defense, immediately deleting your email usually works wonders to your preceived workload.

Reply

[-]

ManateeMutineer@reddit

Heh. Had a reverse problem when IT will blame everyone for their shortcomings. They hated my guts for being able to call their bullshit repeatedly... And then we got a huge project no one wanted and a puddle of bull turned into an ocean with our team being blamed for sloppy development and them ignoring key requirements. Came to a head during the meeting with CTO and CEO when a couple of us (manufacturing dept - so really low on the totem pole and working a few thousand miles from the C-suite) firsst laid open all the bullshit - only to then turn around and defend the IT from the barrage of CEO grade-A BS ideas. And that's how we got the damned project moving after 3 years of bullshit sifting. For anyone in manufacturing worth their salt knows to keep the log and copy the emails - even more so us QC jockeys.

Reply

[-]

Cheesetoast9@reddit

It was the sales department wasn't it? They always want to cover their asses and don't like to admit they made a mistake

Reply

[-]

7A65647269636B@reddit

You should work at every company. I'm in deliverability and often get cases escalated where the recipient complains to the sender (our customer) that they never got the mail. So I send them the logs showing 250 OK from the recipient server (also make sure that there are not issues with DMARC/DKIM/SPF etc that would sentence it to the spam folder) and tell them to have their IT see what happened to the mail after their server accepted it. 2 weeks later: "Our IT can't find the mail." Well, that's because they are idiots who don't know the difference between mail from and header from, if they bothered to look at all.

Reply

[-]

BecomeABenefit@reddit

Odds are that their IT is perfectly competent and they never asked or misrepresented the issue to them. Rule #1: users lie

Reply

[-]

CryptoRoast_@reddit

When everything's going badly: "what do we even pay you for?" When everything's going well: "what do we even pay you for?"

Reply

[-]

Aim_Fire_Ready@reddit

To paraphrase an old saying, "Logs don't lie." (In Accounting, it was "numbers".) I'm so tired of people shifting blame when they don't know how to use basic work equipment!

Reply

[-]

WhiteDogBE@reddit

They can still claim you planted that email there afterwards though. Now if this information was that important, they should have asked to 1. acknowledge 2. follow-up 3. called, texted, chatted

Reply

[-]

duDahl@reddit

I work in k-12. We use JobRouter for various workflows, one being transportation requests. In the middle of our administrator meeting a fellow admin goes off about not receiving emails related to transportation requests needing approval in front of everyone saying my system is flawed and we need to move to something else. Knowing with 99% certainty this guy is not one to read all of his emails based on previous experience, I step up to give my presentation but first open up Google Vault, run a query on his account for the notification emails and show all of my fellow administrators on the projection screen that he did in fact receive those emails and that the system labels applied to those emails were: delivered, unread, trashed..... I am not one to start shit, but if someone wants to go, I have no problem playing by the rules they lay out. It is way too easy to blame IT for problems and for non-IT people to accept that as a reasonable explanation...

Reply

[-]

Sirbo311@reddit

Mess around and find out, as they say...

Reply

[-]

Sirbo311@reddit

If I had a nickle for every time, at my last job, I had to hunt down the 'so and so remotely important claims they didn't get an email that was sent to the company'. Well, Head of HR, here's a report showing that this manager has it in their deleted items, this person and this person still have it in their inbox, and this person deleted it x minutes after receiving it, it's in recover deleted items. To the folks asking me credit, it was pretty much "I'll take it from here." Didn't stop the requests from coming in next time an email went out, but our folks were smart enough to understand what we and the report were telling them. Also, content search in M365, you don't have to take access to their mailboxes. Just pick them all to search, put in your parameters, and then export the report only when you're done. We're E5 licenses, so I may get the good stuff.

Reply

[-]

ArsenalITTwo@reddit

So. Uh. It's in soft delete. Do you have any API based or Defender for 365 deployed that accidentally deleted it?

Reply

[-]

whoami123CA@reddit

Does your boss take your side?

Reply

[-]

SeanMcAdvance@reddit

Is it in junk? We’ve been having an issue with a lot of emails going there.

Reply

[-]

0RGASMIK@reddit

I love me a good log.

Reply

[-]

javaqthrowaway@reddit

This guy shits

Reply

[-]

hume_reddit@reddit

It rolls down stairs, alone or in pairs...

Reply

[-]

RevWaldo@reddit

*What rolls down stairs, alone or in pairs...*

Reply

[-]

agoodrich5@reddit

**Jooooooy!**

Reply

[-]

IdiosyncraticBond@reddit

Are you a lumberjack?

Reply

[-]

SJHillman@reddit

I am, and I'm okay. I sleep all night and I work all day.

Reply

[-]

VariationFun475@reddit

Sounds like these "adults" need a time out and a nap. Maybe no recess for the rest of the week either.

Reply

[-]

TightBed8201@reddit

What toxic enviroment are you working in? I always blast our servicedesk and users with documented proofs against their stupid claims. It is mostly about file share permissions and O365 not working. From the system engineer point of view users are enemies and should be treated as such

Reply

[-]

workerbee12three@reddit

I like how outlook now confirms before you drag an email in to a folder, saved us admins hours of searching peoples email folders for emails that "just vanished"

Reply

[-]

alestrix@reddit

I've never seen that confirmation and I'm glad I didn't, would make my daily work extremely tedious.

Reply

[-]

caponate@reddit

I remember in my exchange admin days at an MSP, a client was severely behind on their invoices. The accounts payable claiming they never got them, they miss all kinds of email from us, blah blah. So I check the message trace, sure enough they're all there. I check and found THEY MARKED OUR COMPANY DOMAIN AS JUNK/Blocked sender. Coincidentally they were terminated a month later.

Reply

[-]

hardingd@reddit

Been there before. But there is solace in the old adage “logs don’t lie, bitch”.

Reply

[-]

EyeJustLikeToWatchTo@reddit

It's always Network's fault. 😛

Reply

[-]

MattWey@reddit

Smells like a coverup or/and people trying to get away with shiet

Reply

[-]

BifronsOnline@reddit

I've head to pull message tracing due to pea-brained users as well. Most of the time it's some bullshit fucking rule they set up that moved the message to another folder, so it didn't land directly in their inbox. You would think they'd know this since they created the rule, but you know users.

Reply

[-]

RevLoveJoy@reddit

In a previous life I was an Exchange admin (back when most was on prem). I have been in your shoes **countless** times. "Exchange messed up and cost the company money" is basically code for "my team are lazy and they conspired to not do the work and then conspired to blame email / IT." I've seen it dozens if not hundreds of times. It got so bad at one point my default course of action would be all the steps you took. That was my first move. The caveat to this is, we billed internally for not my circus type events like this, so we'd bill them $500 across departments for our time (which, after the first dozen or so times this happened got pretty automatic - maybe 15 minutes, uninterrupted). I'm so glad I took off, doused in gasoline and burned that Exchange Admin hat. Email is, IMO, the #1 corporate scape goat and the people running it are, to the corporation, everyone's whipping boy.

Reply

[-]

BerkeleyFarmGirl@reddit

Did the frequency go down with the chargeback? Or at least you didn't have frequent fliers?

Reply

[-]

RevLoveJoy@reddit

Your latter point was exactly it. The chargeback curbed the ~~lazy fucks~~ excuse me distracted coworkers whose first excuse for "I didn't do my homework" was "IT ate my homework!" I would say overall volume, slightly impacted. Long time ago, I don't recall the exact metrics.

Reply

[-]

iamkhris@reddit

But even though I deleted the email, it's still IT's fault.

Reply

[-]

CuteSharksForAll@reddit

Sounds like that user I had who kept the only copy of his important project files on an external hard drive and then cried when it stopped working. My first question was.. Why didn’t you keep this on your backed up network project drive? I guess it was slow to access over VPN from his cruddy home internet. My second question being… Why not come in the office once a week and use robocopy to keep them up to date? Lazy people blaming IT for their idiotic problems

Reply

[-]

MrScrib@reddit

Mister, I have audit trails, logs, and other information which cannot be falsified. I also don't care to get in the way of you doing your job, just that the systems I administer continue to run smoothly and with minimum fault. You have your feelings.

Reply

[-]

dcv5@reddit

Logs don't lie. People lie.

Reply

[-]

ChokladHatt@reddit

I've seen logs lie plenty of times.

Reply

[-]

splenetic@reddit

Logs sometimes lie. I once had a DHCP server that logged that it had received a discover from a particular subject and sent out an offer but the client never saw it. After checking the client, routing, ACLs and a couple of firewalls I resorted to wireshark and found that the offer never got sent out. Wanker. That being said I'd still believe a log over a user every time.

Reply

[-]

dracotrapnet@reddit

A few years ago Unifi access points ate DHCP replies from the DHCP servers. We wiresharked the client adapter, a mirrored part to an access point and the link to the access switch from core. Saw the core kick the packet to the access switch, saw the access point get the packet. Client never got it. They fixed that firmware. Then somewhere along the way their switches did the same thing for a bit. I had to downgrade firmware and hold off on firmware updates for a while until that was fixed.

Reply

[-]

ProfessorHuman@reddit

Pcaps never lie.

Reply

[-]

FarkinDaffy@reddit

Cisco Proxy ARP does things like that. It guessing what the end point is going to do and can request a DHCP even if the client never does.

Reply

[-]

splenetic@reddit

This was definitely the DHCP server. I did a packet capture at the server's switch port and could see the discover packet from the client reaching the server but, despite what its log entry said, the server never sent out the offer. Once I'd realised the log entry was a dirty lie I dug into the server health in more depth. It turned out there was a replication issue between the redundant servers that had screwed up the config for that pool.

Reply

[-]

inphosys@reddit

Windows Server DHCP? Yup, sounds like a familiar problem. I've seen similar issues in several environments. Server A supposed to issue IP, says that it does, but Server B issues the IP first and they enter this twilight zone deadlock and client never gets IP.

Reply

[-]

Dal90@reddit

People are, by our nature, routinely unfaithful narrators. They lie and misremember. They often misinterpret information through their biases developed through upbringing, education, social groups, and experience and then are confident in expressing their misinterpretations. The brain itself is not immutable in its memories. Computers are seldom unfaithful narrators. More likely they never recorded the data to review. But any single system can -- in rare instances -- be unfaithful; so the higher the stakes the more important it is confirm the why, what, and how of it's logging. In the case with the OP, even if the Exchange log had lied about the delivery...the deleted items folder did not. The next check then becomes if there is anything like a rule that could explain non-human intervention doing so.

Reply

[-]

0RGASMIK@reddit

Google logs would like a word. Had a clients admin account no one uses log in and make hundreds of changes according to the “logs.” Also according to the logs no one has signed into that admin account since the orgs creation. Google support claimed it was cached logins and closed the ticket despite use telling them no one’s ever logged in. We tried to replicate a scenario where these actions line up with the logs and could not do it.

Reply

[-]

SteveJEO@reddit

My first thought would be a Rogue dev/service. Seen something like it before where logins were being recoded from dead accounts. Turned out what had happened was that an old machine image was restored and the service auto loaded, the service then tried to run it's config script and you had these logins start flying off all over the shop. The actual devs themselves didn't have a clue it was happening. They were just trying to get the machine back up.

Reply

[-]

alpha_dk@reddit

My first thought would be "there's no guarantee someone at your client was the one using the credentials, change 'em for safety"

Reply

[-]

SteveJEO@reddit

Old account shouldn't be active in the first place. It should be dead and retired.The logs should still record the attempts though. It's basically the argument though isn't it. Logs don't generate themselves. The information is coming from *somewhere* so where?.

Reply

[-]

alpha_dk@reddit

Doesn't say "old account", says "admin account". Nothing leads me to believe that there were not valid login attempts from someone, somewhere. It's not like only the people at the company are capable of it.

Reply

[-]

TMITectonic@reddit

>Logs don't lie. People lie. While *almost* a universal rule, since eventlogedit (NSA/Shadow Brokers) became public, and has since been improved, definitely never forget the *even more* universal rule of: **Trust, but verify.**

Reply

[-]

TheGooOnTheFloor@reddit

I'm going to frame that and put it above my desk: "Logs don't lie, people do."

Reply

[-]

rat4204@reddit

Unfortunately we seem to have found ourselves in a culture where someone's feelings are accepted above facts based on hard evidence.

Reply

[-]

Liquidretro@reddit

I would argue it's more that that, I want things to run smoothly and you be able to do your job because it makes mine easier/better when I'm not interrupted with problems all the time or angry people.

Reply

[-]

ChokladHatt@reddit

Could be a bug in the mail client. Microsoft is generally quite shit, so I wouldn't ignore that possibility. The blame game is completely pointless though.

Reply

[-]

nullvector@reddit

When I was an Exchange guy way back in the 2003/2007 days, it was a great product to administer, but the user side of it was frustrating. Got that same thing all the time. "I never got it!", "It didn't deliver!". I'd provide email receipts, logs, etc...people would always insist they never got it. Then we got an email archiving system that could restore point-in-time mailboxes to PST files, down to the minute. Absolutely brilliant for the time. Restoring the mailbox and showing them point blank that yes, it was in there at 3:01pm last Friday before you deleted it at 3:05pm (here's the logs!). We had users store things in the trash, intentionally too, and then complain when the system automatically cleared out trash content every couple weeks.

Reply

[-]

BerkeleyFarmGirl@reddit

My first Exchange restore - back in the 5.5 days when you had to spin up a recovery AD forest - was for someone who stored important stuff in the trash folder.

Reply

[-]

nullvector@reddit

Oh no....that would be terrible. I didn't start working with Exchange until 2005-2006, and then did the MCITP for 2007. We moved off of Exchange after that, and I stopped doing email admin stuff at that time. Restoring a forest to get a mailbox back would be awful!

Reply

[-]

BerkeleyFarmGirl@reddit

Yeah we wouldn't have done it if the person weren't the big boss of the org, for sure. (I keep hearing of this happening, and I shake my head EVERY TIME.) I also got a lot of "email never arrived!" complaints. One was that the user had sorted alpha by sender so never got down to the S sender.

Reply

[-]

nullvector@reddit

Haha, definitely have been there. Back when we ran exchange in the early 2000's we had to rebuild a whole Novell Netware/Groupwise environment (from the previous email environment I didn't administer) to look for an email from 7-8 years prior (which was never found). 2-3 weeks of work to look for an email...ugh. I'm so glad I moved over to VMware/CiscoUCS/AWS and don't do email anymore. Definitely have a soft spot for those who still do, so if you still do manage that kind of thing, I salute you! :)

Reply

[-]

BoltActionRifleman@reddit

> FFS are users this dumb? Is this a serious question? 😂

Reply

[-]

serverhorror@reddit

Sounds like you ned to provide some content to https://thedailywtf.com/

Reply

[-]

ayelmaowtfyougood@reddit

Tell manager hey make sure to download or sync any important files because the users account will be wiped automatically after 30 days. 90 days later, manager states they can't use a file that they were able to access before but can't anymore. You know where this is headed.

Reply

[-]

kurjo22@reddit

I had a similar case of missing mails. It turned out that mails that where filtered through office 365 security part where not displayed in the exchange mail flow. Because of this I always enable monitoring in the compliance center. Maybe it helps you out :)

Reply

[-]

lefthanded_engineer@reddit

They all assume shit rolls down hill to IT. It ate their homework.

Reply

[-]

BerkeleyFarmGirl@reddit

I have been saying ""The email never arrived" is the new "my dog ate my homework".

Reply

[-]

Ok-Wheel-476@reddit

This happened all the time at my last job and get this I’m in security and they would blame security for emails not making it in.

Reply

[-]

maxtimbo@reddit

"it's the firewall..." Look Bob, you keep saying that word, but I don't think you know what it means.

Reply

[-]

machacker89@reddit

hahaha classic "Princess Bride" comeback ;) I'm glad someone else use that phrase. like:"not my monkey, not my circus", FUBAR, RTFM and my personal favorite: PEBKAC. Glad my computer tech teacher taught me early on.

Reply

[-]

nstern2@reddit

When we used to have exchange on premise we would get these every now and then. Our front line guys saying so and so isn't receiving mail and 99.9% of the time they weren't sorting their inbox by date. I was so sure that's how this story was going to end.

Reply

[-]

elevul@reddit

Whenever that happens I just pull the logs out of Exchange and move on. No reason to waste time further.

Reply

[-]

bluehairminerboy@reddit

> FFS are users this dumb? Yes.

Reply

[-]

davidbrit2@reddit

> Now the department is blaming us because the file doesn’t have the right content- sorry that is not even remotely the IT department’s responsibility- please find another way to spin it how you f**ked up and don’t blame IT! I think at that point I would just close the ticket with "NMFP".

Reply

[-]

DumbshitOnTheRight@reddit

NADS - No Abnormality Detected in System

Reply

[-]

SamR1301@reddit

"Layer 8 error"

Reply

[-]

DonCBurr@reddit

LOL.. 😁

Reply

[-]

SsP45@reddit

We had a C-level person at our company claim the same. They never got a really important email. Ran a PS command that searched for the message and we found it in their archive folder. Turns out they had a mailbox rule that auto archived it.

Reply

[-]

BrainWaveCC@reddit

Logs. aka Receipts.

Reply

[-]

deskpil0t@reddit

I’d be asking for 20% of that sla fine for my budget

Reply

[-]

moltari@reddit

remember, this cost the company a "Substantial" amount of money! they should be on the line for that, not you .

Reply

[-]

vrtigo1@reddit

This is a monthly occurrence. "I never received that e-mail!" "Then why is it in your deleted items folder?" Extra points when mail flow tells us that the message was automatically deleted or redirected to a folder the user never looks at because they created a rule.

Reply

[-]

hooshotjr@reddit

I don't know why people can't say "I missed it" or "didn't see it". Someone tried to throw me under the bus in this manner, tattling to my boss that I never sent them something. I then have an admin confirm that I sent it, and it was recieved...and as soon as I point this out it's all "well I think it's time to move on".

Reply

[-]

_Beelzebubz@reddit

I had a user submit a Ticket saying they were missing a full year’s worth of emails. I asked them the basic question of if they accidentally deleted it and got the typical “no, of course not” response. With their permission, I granted myself access to their mailbox and found all the emails in the recently deleted portion of the deleted items bin. Thankfully I was able to restore all of them.

Reply

[-]

DatDing15@reddit

Had a similar report but different issue. It was the rarely known Exchange "feature" called "BccFolderingAgent". (Exchange 2016) The incident report was, that an email from an internal sender, sent to a DL for internal recipients as BCC (BCC so people can't "reply-all") never arrives to the recipients. Those were still Exchange on-premise mailboxes so the search was easier. So I conducted a Search-Mailbox and found the messages in different subfolders among the recipients. Turns out, people were only searching in their inbox. But there was no rule on their Outlook clients set, that would move the mail from the inbox to these subfolders. Turns out the sender used an old e-mail as a template. And the Exchange server sees this old "In-Reply-To" Header, which would correlate this new mail to this old mail. So the Exchange server moved this mail not even to the inbox, but instantly to the same subfolder where the old mail is. This behavior was seen on Exchange 2016 on-premise mailboxes. But not on our Exchange Online mailboxes.

Reply

[-]

uwu_4_rance@reddit

Wonder if that one's documented anywhere outside of this reddit post...

Reply

[-]

DatDing15@reddit

Two things I could find, that brought a solution. But there's really not much further information I managed to find. I asked our Microsoft service provider's Exchange experts afterwards and they couldn't find any official documention nor ever even heard about it. They also confirmed the behavior on their end. https://community.spiceworks.com/topic/2031107-outlook-is-automatically-organizing-messages-without-a-rule https://techcommunity.microsoft.com/t5/exchange/bcc-foldering-agent/m-p/196212 Is an extra feature by design. But for me personally seems so unintuitive, no idea why Microsoft specifically chose to implement this. But as mentioned couldn't reproduce this on Exchange Online. So either the forgot about the feature or dumped it. The solution is also in the links: New-SettingOverride DisableBCCFolderingAgent -Component MailboxTransport -Section StoreDriverBccFolderingAgent -Parameters @('Enabled=false') -Reason "Custom override to disable BCCFolderingAgent"

Reply

[-]

uwu_4_rance@reddit

Thank you Microsoft for keeping so many of us in a job.

Reply

[-]

tankerkiller125real@reddit

See, a lot of IT people hate Microsoft with a passion because of all these weird edge cases, bugs, and other issues. I thank them daily for keeping me employed in a well paying job sector.

Reply

[-]

DatDing15@reddit

After enabling the Override (disabling the "feature") I wondered how often this has happened previously? And I just labeled the end-user off as lazy, or too incompetent to search properly of for accidental drag&drops, when I've found their "oh so important missing mail" they've just received, in a subfolder??

Reply

[-]

StabbyPants@reddit

> So the Exchange server moved this mail not even to the inbox, but instantly to the same subfolder where the old mail is. too clever by half

Reply

[-]

ITechsXpress@reddit

Damn!

Reply

[-]

superadmin_1@reddit

recommend an external SME show what happened. Keep all logs and evidence now.

Reply

[-]

AHrubik@reddit

Reminds me of the time I got blamed for a dept using a web server as a file archive and dept share folder. I got a low storage warning one day and cleaned out all the old documents that were "expired" by policy and they lost years worth of files.

Reply

[-]

ElectroSpore@reddit

For us it is usually user mailbox rules. Which it is nice that Exchange Online message trace clearly shows what folder the message was delivered to. Anything but inbox and the user probably has a broad text match filter dumping the message to trash or something.

Reply

[-]

laybek@reddit

I had a mystery of disappearing emails - turns out user is clueless about focusing windows and is trying to edit text in word while having outlook in focus. And of course, pressing backspace deletes emails...

Reply

[-]

mrbiggbrain@reddit

Backspace archives emails.

Reply

[-]

laybek@reddit

Not on Outlook on Mac. At least that version that was used..

Reply

[-]

Kardinal@reddit

I think backspace for archive is a relatively recent change. Outlook didn't really have an Archive function for a long time. Maybe three or four years ago it was added? I never used the backspace shortcut before archive, but I suspect that before it became the archive button, it was another delete button. It often is.

Reply

[-]

Nu-Hir@reddit

Well, on a Mac, the key that's labeled Backspace on a PC is labeled Delete on a Mac. Just like the Enter Key (not on the 10 key) on a PC vs Return on a Mac.

Reply

[-]

Ron-Swanson-Mustache@reddit

Outlook for Mac and Exchange on prem. Name a more iconic rivalry.

Reply

[-]

Mikash33@reddit

Funny, this happened to me last week as well. User tells me an email to a government agency never made it there in time, and that it is going to cost us funding money. They want me to go look in the system and see if it ever left Exchange. They had sent it the week before, closed up shop and went on vacation. Check Exchange, and sure enough it did not go through on the day in question, but on the day they called me. So it looks like it didn't leave the mailbox, went into outgoing, they prematurely closed up the machine before it sent and went home. I explain this to them, and they tell me that because they use a Mac, and heard the "Swoosh" sound, they knew it sent. Wrong. I have the proof that it didn't send. Here's the proof. Not good enough, they had me expand the parameters, check the system again, so I did. I humored them. It's then that I notice that the email had a second recipient: The sender. "Did you CC yourself in the email as well?" "Yes Mikash33, I did." "I see. Did you receive the email before today?" "No, I didn't." "OK, so think about that for a moment; You sent this very important email to them and yourself, didn't get your copy, and didn't think to check if it sent until a week later?" Silence on the phone. Checks watch, 10 seconds go by before I bust out: "Is there anything else I can help you with today?"

Reply

[-]

Nik_Tesla@reddit

I've never had this level of blowback, but I can't tell you how many tickets I've gotten where the employee claims they never got something, and blames IT, and then I find the logs that say that they added that address to their spam list or made a rule sending it to a folder. Usually they act humble or bashful, I don't think I've ever had them get even more combative.

Reply

[-]

ElderHakunin@reddit

Sorry, but why do you even care when there's rock solid evidence of their wrongdoing?

Reply

[-]

OGReverandMaynard@reddit

One of the earliest lessons I learned in IT is have a way to prove something did or didn't happen. People love to blame IT for their bullshit so they don't have to accept any personal accountability. If you can hit em with facts and expose their bullshit, most sane people will slink away in shame. Sounds like these people are doubling down on their bullshit, so I'd take it up with whoever supervises them and show *them* the proof.

Reply

[-]

Cyberhwk@reddit

Authentication, Authorization, and **Accounting**.

Reply

[-]

CeldonShooper@reddit

Small environment here but I've had several allegedly lost emails and could always prove via our immutable email archive that the mails indeed were received and archived. Folders are a big feature here for the reception so sometimes one of the employees simply pushes the email to some random folder and it 'disappears'. Same for customers who complained they 'never received a return call'.

Reply

[-]

Revzerksies@reddit

I love it when i can prove i have hard evidence that things were done properly

Reply

[-]

Thedguy@reddit

I’d sometimes want to calculate the amount of time I’ve had to spend proving people are trying to blame IT for their absolute failures. My favorite was probably the weeks of effort to solve the “phones not working” problem that ended up repeatedly being people turning off the ringer or unplugging their phones, or just flat out refusing to answer the phone. They wanted IT to solve this issue.

Reply

[-]

Oshkalan@reddit

In another episode of Office Politics....

Reply

[-]

CaffineIsLove@reddit

To give him the benefit of the doubt maybe he had an e-mail rule on his mailbox to auto delete those types of emails?

Reply

[-]

donithansewell@reddit

[Hahahaha](https://old.reddit.com/r/sysadmin/comments/13pqn2s/good_at_your_job_vs_skilled/jlb6f7p/)

Reply

[-]

8FConsulting@reddit

" FFS are users this dumb?" Yes....yes they are.

Reply

[-]

tickpig@reddit

If it was that important, a phone call and a Fed Ex envelope would be the way to conduct the transaction.

Reply

[-]

mediaogre@reddit

This is a classic case of being forced to provide a technology solution to address a behavioral problem. I see it far too much.

Reply

[-]

DK_Son@reddit

It's the kind of exposure a newcomer would need in order to realise how evil end users really are. A Service Desk engineer a few months on the job would be like "OH wow. Now I understand. We look after complete idiots. Complete idiots that WORK IN PACKS, LIKE WOLVES.".

Reply

[-]

OMGItsCheezWTF@reddit

Users setting up blanket deny rules to IT speaks to a larger organisational issue. What emails are you sending out that cause users to become annoyed enough to block you? An email from IT to the average user should be an exception (new laptop time, rare outage etc) not something that causes users to go ["oh you're joking, ANOTHER one?"](https://www.youtube.com/watch?v=d3PKE8uTSp8)

Reply

[-]

Kardinal@reddit

I love that attitude. It's like shadow IT. Shadow IT usually happens when we don't give our users what they need to do their jobs. So we should ask ourselves how we can meet that need so they don't try. Yes, of course, sometimes the answer is to block it, but we should start with ourselves. Yeah, if our users are deleting (or even just not reading) our emails we should be looking at the content to find out how to make it relevant. Often by sending less emails. The problem I face is that management wants the emails for CYA purposes. CYA isn't more important to the effectiveness of the company as effective communication. But we all know that our lives are driven by incentives. There's no incentive for effective communication, but there's definitely an incentive not to have that uncomfortable conversation with your boss (at whatever level) because you didn't CYA.

Reply

[-]

poldi1405@reddit

The answer to too many IT issues is unfortunately: management. Sorry Jen, it wasn't me who set up that asinine 90 day password policy. Sorry Jim, I know your laptop smokes and sounds like an old Mercedes with engine problems and only works once a week, but I can't give you priority because the general manager needs a new work iPhone first.

Reply

[-]

Kardinal@reddit

> The answer to too many IT issues is unfortunately: management. To be fair, they are the cause of many issues, but also *one of* (not the; not the only!) the major reason for our existence and for our successes. I have been lucky to work for some pretty good managers who keep the BS off my plate, represent my work, and get me what I need to deliver. Yes, they also ask for silly CYA sometimes, but nobody is perfect. I do remember arguing with my boss many years ago that "The CEO not being able to print is not as important as no one being able to receive email." He advocated That the CEO's print issue was objectively more important, I disagreed. I asked the CEO about it later and he said "Of course no one getting email is more important."

Reply

[-]

DK_Son@reddit

Fair point. Most of these were in previous jobs where I/my team wasn't in control of what was sent, and I can't recall exactly what was sent manually. We were just the ones who found the blocked rules. What I think a lot of them were was they'd get email updates for Service Now tickets they raised, and they'd block those. Then some of them would go on a blocking spree in order to keep their mailboxes cleaner. So they block other IT-related addresses in the same swoop. They don't realise they're blocking the address indefinitely. They don't realise that even when they are addressed directly with "SUSAN. YOUR DATA IS BEING MOVED", that those emails will still get filtered out. It's been a long time though. I haven't had to deal with that for a while.

Reply

[-]

CKtravel@reddit

It ALWAYS boils down to how toxic the company in question is and how much they allow rats like these get away with this shit. Honestly if a user has tried to pull such shit on me and gotten away with it I'd quit from that hellhole soon after.

Reply

[-]

jr49@reddit

If not receiving an email internally can cost the company lots of $$ then that process is broken. should be no dependencies on users reading their email. files should be automated to specific locations where they need to be. if it's a report that provides info for them to make financial decisions move that report to an appropriate location not in an email inbox.

Reply

[-]

YumWoonSen@reddit

They're blaming you because " cost the company a substantial amount of money" means someone is getting canned and they don't want it to be them. Back when monitoring was part of my team we implemented a policy where every single alert that went out would BCC a mailbox we had created for CYA. I can't tell you how many times we heard "an email alert never went out!!!111" and we'd forward a copy saying, "It sure did. We got a copy."

Reply

[-]

bmyst70@reddit

It's the time honored tradition of finding a scapegoat for their mistakes. Facts have nothing to do with their actions. Good thing you have logs proving they're wrong.

Reply

[-]

MyMainMobsterMan@reddit

They clearly fucked up and are using you as a scapegoat.

Reply

[-]

FeesShortyFees@reddit

I can't prove you did NOT receive an email, but I can certainly prove if you DID. Bad call by the department.

Reply

[-]

Shnazzyone@reddit

Next up is, "somehow it was deleted, BUT NOT BY ME!" Response with that is to have them check their sign ins and go through security hardening as they just claimed they allowed their account to get compromised. Probably from opening spam.

Reply

[-]

lurker1B@reddit

What no mandatory security training and testing with full account lockouts until successfully completed?

Reply

[-]

SmallFormalWear@reddit

Do you have retention policies set? Produce the logs where they all deleted it at different times to cover their story. Stick to your guns don't let a department team up against you. Request to log that departments activities more closely from the head of IT because that department is black balling you and you are here to protect company then department and these assclowns are putting both at risk by lying and potentially threatening our futures with the organization. Nah, this shit is war.

Reply

[-]

SmallFormalWear@reddit

Politics in business is like selling cars. You want to sell the car but you don't want to look like the salesman. Honor before retribution, company first voice activated, to the executive wing bat boy!

Reply

[-]

JerRatt1980@reddit

They blame you, that's when I start the "the way through logs show the facts means the only two ways this happens is either gross negligence on the receiving end, or outright intentionally malicious action."

Reply

[-]

AhmedBarayez@reddit

As long as you have your logs saved and proves the system is all good, you’re good 👏👏

Reply

[-]

theborgman1977@reddit

Sounds like that department needs to be put on legal hold. So that you can prove even if they delete the email they recieved it .

Reply

[-]

monoman67@reddit

Legal Hold for everyone possible.

Reply

[-]

Polar_Ted@reddit

For these lost email tickets I almost always include a mailbox search showing exactly where it currently is and sometimes a copy of the inbox rule they made that auto deleted it.

Reply

[-]

nighthawke75@reddit

Pull the delivery logs, assemble a grievance report, and hand the whole smash to HR. Then sit back and watch the guilty party squirm and writhe on their own spears.

Reply

[-]

topknottington@reddit

i love these tickets. please mfer... give me a reason to audit mailflows... i love this crap, its a perk of the job

Reply

[-]

DonCBurr@reddit

to the user; "It was THAT important and you did not pick up the phone to make sure the message was received and understood?"

Reply

[-]

ajoltman@reddit

Do they not know we potentially log everything? It is always fun hearing their story and it doesn't match up with the timestamps of when those actions were logged. So, we're going to believe the logger failed and the person is being truthful.... come on now.

Reply

[-]

fail4ward@reddit

Lies... that wasn't there! You did something to make it show up. 100% likely the response after finding it.

Reply

[-]

ThatCrankyGuy@reddit

If you're collude, at least add a person of knowledge into the conspiracy circle. This is hilarious. Office politics can get fucked, though. It's such a drain and these morons think they can get away with it (and if you have a dumbass boss, they probably will and you'll be under the buss - RECORD AND LOG EVERY SINGLE THING ABOUT THIS INCIDENT).

Reply

[-]

lizzieismydog@reddit

Or, if dealing with financial systems, on the front page of the New York Post.

Reply

[-]

D_Humphreys@reddit

I always send a screenshot of the message trace to the user, particular the part where it says "message was delivered to user's inbox", then ask, "Is it possible you have a rule that put in a folder or sent it to junk?"

Reply

[-]

Fatality@reddit

365 will tell you the folder it's in too

Reply

[-]

torewasa@reddit

jag@erling.se 👍 l köp öländska håll å ååpäoåmiökåöpäpåå På åpp å å ooo å å kl klä k l l okå(å å på kl å kokplattor l 8 jag 🙋 jag dä 🅾 å så att även jag 🙋 m i8oklkooåkålōpõlppplplp0lll0l0lplplllpplolpllpkopllplåkpkplllplpjllpklllllplpkölpkplp) lök öl öl ľ kö lä på å öl åå på pöå iöoiioiïplkåkoåliåkåklåiåoåoåóooåooåioåoooåooåooåopioioioåoioåoooioiooåipiäåiiokpåiloåïooiioåiooåioåkoioåpäålrwåååååzåoÅpä, Hemma åå , nååpäåå

Reply

[-]

texan01@reddit

I see...and how do you feel about that?

Reply

[-]

palordrolap@reddit

I asked Google Translate for help with this. Best I can tell, someone from Sweden came to Britain for the standard night out of beer and an onion bhaji, but had far too much beer. The very long words seem to suggest that they ended up at a fairground and had a bad time on the teacups. They're home now and still don't feel so good.

Reply

[-]

Common_Dealer_7541@reddit

I got nuthin

Reply

[-]

TheOnlyKarsh@reddit

I love when folks say I never sent the email. I just find it in sent items and forward it to them.  Karsh

Reply

[-]

texan01@reddit

Yup.. I've learned to keep all the emails I send/receive. cuts out a lot of bullshittery with customers/coworkers.

Reply

[-]

TheoreticalFunk@reddit

DL... I could come up with a lot of different meanings for this. Wat? Is this something specific to Exchange?

Reply

[-]

oni06@reddit

Distribution List

Reply

[-]

creamersrealm@reddit

Ah yes this reminds me when I did email. Everyone kept complaining to the higher ups that they never received it. I did a mail trace and screenshoted the exact rule that moved it to a folder/deleted it. They eventually stopped complaining.

Reply

[-]

DaLam@reddit

You check their inbox rules?

Reply

[-]

Redemptions@reddit

> I gave my adm account access to the mailboxes and logged into each one and found the message. The person causing the most stink- I found the message in their recover deleted items and it had been deleted four minutes after it was received. > Now the department is blaming us because the file doesn’t have the right content I was expecting the "It's your team's fault, it shouldn't be possible to delete important emails."

Reply

[-]

MGNurse25@reddit

So easy to blame IT. I like the classic, I’m typing in my password correctly but it’s saying it’s wrong. 🤦🏼‍♂️

Reply

[-]

CKtravel@reddit

It's simple. If those four mofos don't find themselves in a sea of pain after they've thrown all this shit then you should start looking for a new job. It means that not only this company sucks big time but that they're toxic as hell too.

Reply

[-]

lennert1984@reddit

Classic! Reminds me of the time a user was ranting for half an hour on the phone that I "lost some of her incoming e-mails" and that I was "bullying her by sabotaging her". Turned out she made a rule on her mailbox to automatically move e-mail from a certain person to a folder. She apologised for ranting but from that day on she got on my shit list ;)

Reply

[-]

Kardinal@reddit

I've been doing Exchange for a very very long time. Seems like almost every report of an email that a user didn't receive was because they moved it without noticing or thinking, or a rule. I've had to coach dozens of help desk techs to confirm the email is not in the mailbox before we start a trace. Thankfully nearly every user I've ever worked with on this type of issue is just grateful to find the email and I don't think I've ever been blamed for it when it was in fact user error. Lucky me I guess.

Reply

[-]

dnalloheoj@reddit

> Turned out she made a rule on her mailbox to automatically move e-mail from a certain person to a folder We've seen this numbers of times. Say you just migrate them from on-prem over to 365, everyone's like 'ooo fancy new inbox better keep it clean.' Someone will make a rule deleting junk emails they receive daily from whatever, Lifetime Fitness or something, but instead of blocking the sender they just make a blanket rule deleting all emails including "Lifetime" in the subject. Fast forward two years and they miss out on an email containing a contract (and therefore $$$) from 'Lifetime Food and Wellness' or whatever, raise a stink, blame the "new email system" that's 2 years old, and then quickly get shown their folly.

Reply

[-]

maxtimbo@reddit

Fuckin make me waste a day to figure out you're an idiot liar. That shot makes my blood boil

Reply

[-]

benderunit9000@reddit

Sending emails for urgent matters. yea, that was mistake #1.

Reply

[-]

whycantdeerfly@reddit

Do you work with me? I get the same line from our BD people just about once a week.

Reply

[-]

Tx_Drewdad@reddit

"I sent this critical piece of information by a method with no guaranteed delivery, and no guarantee that the recipient will look at it. I took no steps to confirm the right people received the information, or read the message. It's all your fault."

Reply

[-]

rswwalker@reddit

I used to get these all the time. My first response is to have them share their screens, first the sender, then the recipients to show that they sent it and then didn’t receive it. I usually find at that point that it either was never sent or deleted upon receiving. I guess users starting realizing I’m not going to fall for any banana in the tail pipe and these stopped.

Reply

[-]

BisonST@reddit

You could also run an eDiscovery search for a more report-like record. Instead of having to break into accounts and giving screenshots. Keep sticking to your guns, don't let them blame ya'll!

Reply

[-]

jkalchik99@reddit

I wish I had a nickel for every time I've asked for a mail trace to show delivery. And if you'd take away a 5 dollar bill for every time it wasn't delivered..... I'd still have all of those nickels.

Reply

[-]

stimj@reddit

Nothing makes me roll my eyes harder than "I'm on this distribution list and I didn't get this certain email". You know how many times that's turned out to be an Exchange problem? NEVER Exchange doesn't just randomly NOT deliver to one (or some) of the members on a DL. It goes to all of them. It's not a person, it doens't "forget" or "decide" not to send them. Emails to a DL go to all recipients. In every case across a 20+ year career now, the user always got the email, and usually either deleted it or forgot they had a rule that would move/delete it.

Reply

[-]

arkain504@reddit

You don’t have to give yourself admin access to the mailboxes. You can search them all at once and have the results sent to your email in a CSV. I use it to get rid of phishing attacks that get through to delete them from peoples mailboxes who haven’t opened it yet.

Reply

[-]

Kelsier25@reddit

This is actually a large part of why I'm looking to transition out of my current role. IT is always the scapegoat here. Every single time someone screws up. Screwed up data input? "the system shouldn't have let me put the wrong thing in". Running late on a deadline? "my computer is too slow". Forgot to respond to an email? "I never got the email". People never just own up to it. I'm lucky in that the org always trusts me once I explain what happened, but it's so incredibly irritating to have to stop what I'm doing every day to pull logs and show that people are lying about why they messed up.

Reply

[-]

itsmuddy@reddit

If the email was so important why didn’t they have receive confirmation turned on for it.

Reply

[-]

njaneardude@reddit

Blame IT - exactly what the firewall would say.

Reply

[-]

catwiesel@reddit

escalate. escalate until the level both dept. answer to (owner level?) then, demand either you are believed or unbiased third party forensic evidence specialists are hired, and determine the truth and since you obviously cant be trusted, the it dept. will not use any account that requires admin rights to do anything until the issue has been resolved. preferably with the guilty party being terminated

Reply

[-]

Enter_shiitake@reddit

Remember this next time they submit a ticket.

Reply

[-]

chuunibyou_edgelord@reddit

I guess they don't know about logs, recovering deleted items and backups.

Reply

[-]

XanII@reddit

I love these events. Very rare but they tend to have funny outcomes. So foolish to mess with the BOFH people who usually play nice but no longer have a reason to do so.

Reply

[-]

NullNova@reddit

Always nice when you can show them that they fucked up. Hope they back off from you now.

Reply

[-]

this_barb@reddit

If 4 people can lie about an email in their inbox, then NASA can lie about the moon landing or something like that.

Reply

[-]

twhiting9275@reddit

Sounds like someone has some shitty filters!

Reply

[-]

TheJizzle@reddit

I use Google eDiscovery when necessary to chase down lost emails. I love replying to liars with screenshots of the results where it says "unread, archived" without context, just so they know I know they lied.

Reply

[-]

StaffOfDoom@reddit

Never blame IT! Any tech with any experience knows how to CYA! Also, always let users know at any and every opportunity about non-repudiation and email tracking. If they know we can see every step an email takes, they’re less likely to BS about it not getting there. Content is not an IT function, we’re delivery so don’t shoot the messenger! We might not help the next time you’re down!

Reply

[-]

SXKHQSHF@reddit

An unread email to 4-person DL "cost the company significant money"? Don't these people have phones? Don't they follow up when there's no response to a critical communique? Was the message even marked urgent? Did they have meetings on the topic in their calendars?

Reply

[-]

AlmostRandomName@reddit

I've been accused of never sending an email multiple times, and also been accused of having received an email and not acting on it (they never included me on that critical email). Why do users keep making stupid email-related accusations about the people who have the power to *check* and prove themselves?

Reply

[-]

housepuma@reddit

LOL! At least if you're gonna lie remember to empty the deleted items. I hate meatspace sometimes.

Reply

[-]

PedroAlvarez@reddit

Sounds like someone's lie got out of hand.

Reply

[-]

jdptechnc@reddit

It is one thing when a user deflects blame for their lack of progress on whatever task toward some random IT glitch. If your leadership is still entertaining this BS from them after proving their lies, then you have some very, very crappy leadership.

Reply

[-]

Mr-RS182@reddit

If I had a £1 for every ticket that has come across my desk for "Oh I have not received an email" only to find it sat in a folder somewhere on their mailbox etc Had users realise their mistake and try delete the email to cover the mistake but not know about recoverable items in Outlook. Or us just being able to simply check the logs to see the email has been delivered. Had one user state the email system being broken and a client is not receiving their emails. Bare in mind this user has no proof they didntreceive the email, only the fact they have not replied. Told the user to call the client and asked if they got the email then never heard from the user again.

Reply

[-]

SoYorkish@reddit

Do they have any plugins for Outlook that might be processing incoming emails incorrectly? Like if they have something that pushes the email into a CRM or something then deletes the message after. It'd be odd for 4 people to do the same action, but not odd for plugin to consistently misprocess an email.

Reply

[-]

StabbyPants@reddit

> Now the department is blaming us because the file doesn’t have the right content "you proved me a liar, so i'm going to make up a distraction". what's the email equivalent of rolled eyes?

Reply

[-]

GremlinNZ@reddit

Today's one. Panicked business owner: staff member is missing their contacts and appointments in calendar. I'm driving, so basic troubleshooting, look in deleted items etc. Nope. OK, I'll give you a call and remote in later. Remote in, hunt around, nothing deleted etc. How many contacts are missing? I don't know. How many did you used to have? I don't know. OK, how do you know what is or isn't missing?... Wait, what did you do to get to this point? Well I started typing an email, and their name didn't come up... Right... Autocomplete issue, used nk2 to export the list and imported into contacts... I thought they'd hit 999, but nope, just 30-40...

Reply

[-]

sgt_Berbatov@reddit

When are we, the people of IT, going to rise up and put these fuckers in the gulags? The time of the IT tech is now my friends. Rise comrades! All we want are the means of going to work and not being blamed for the inadequacies of the idiots!!!

Reply

[-]

thermonuclear_pickle@reddit

And when we run the governments, who’s gonna do our IT?

Reply

[-]

sgt_Berbatov@reddit

I dunno. Zuckerberg?

Reply

[-]

thermonuclear_pickle@reddit

He wouldn’t be interested, he’s already got all our logs.

Reply

[-]

sgt_Berbatov@reddit

He'd know what was wrong though, could actually improve the SLAs if he did it because of that.

Reply

[-]

havoc2k10@reddit

just be firm with your report, show them all the logs and proof that you have seen the email on their mailboxes and that an one of them deleted the mail. Finalized the report and tag it as "No issue found", mail sent and received and read by the recipients.

Reply

[-]

LogicalExtension@reddit

Question - are any of the users using Mac Mail? Several years ago I had a dev claim that they were not getting certain email. I pulled out audit logs, and found the events where it had something along the lines of "Client Delete". So I sent a screenshot of that log entry showing they were deleting the emails. Turned out it was hitting some local Mac Mail spam rule, and rather than it moving it to Spam in Exchange, it was deleting it off Exchange, and moving it to the Mac's local Spam folder.

Reply

[-]

Ochib@reddit

You want the truth? You can’t handle the truth. No truth handler, you. Bah, I derive your truth handling abilities!

Reply

[-]

da_apz@reddit

"The e-mail was lost" has become such a stereotypic thing that people fire away that excuse often without thinking who they're telling that lie. I have also been told a lie of "computers just always keep on crashing, I could work at all today because it just gave out errors" and since that was on us, I had to investigate and found that it was probably the oddest computer ever, as it worked perfectly good for Facebook browsing until the employee got caught when they hadn't done anything that day.

Reply

[-]

EspurrStare@reddit

Can you arrange a quick investigation? Without warning just plug in and watch the mailbox and find the file. With permission but without warning.

Reply

[-]

wrootlt@reddit

Haven't anyone of them claimed email magically got deleted from their inbox? :)

Reply

[-]

WatchOne2032@reddit

Heard it all before. These types of incidents allegedly always cost thousands or millions of pounds. And for some reason there was no way that amount of money was worth the user picking up the phone or re sending the email!

Reply

[-]

AppIdentityGuy@reddit

The number of times this has happened to me....Or some variant of it.

Reply

[-]

emmvee17@reddit

Those are the best conversations. /s But really stick to your facts and only facts. Once you bring feelings into it you lose. But people always want a reason they are not an idiot, but you have facts and a job scope/description to fall back on and show sending this email is not your problem. You don't blame the car manufacturer when you crash a working car into some other car. How people use your working system is outside of your scope, you only care that it works.

Reply