Last words.... | TheaterFire

Systems engineer here. Only a sucky sysadmin speaks losely like that. If you do your due diligence and test your work there should be no problems in being certain

[-]

kenfury@reddit

Arrogance and stupidity all in one sentence, how efficient.

[-]

p3ac3ful-h1pp13@reddit

Efficiency comes with automation which is determined by the type of infrastructure and software stack in your DC or Cloud.

[-]

Own_Back_2038@reddit

I think you are confused about what makes it suck to work with someone

[-]

My core switch isn't properly sending dhcp requests right now. I have it debugging to the log. PC directly connected to the core. All acl removed. Broadcast leaves, confirmed by wireshark, but the core doesn't acknowledge it most times. Sometimes it does sometimes it sends it, but doesn't process the return packet. When it works, I see the full process in the log every time, so I know it's not just skipping some logs due to a buffer issue or something.

I've made about 15 changes that likely should have been made anyway (I inherited this system) and they seemed to work for about 10 minutes each before I got to realizing what the actual problem seems to be. Each time those 15 minutes came around, I told my team so they could test and tell me if it didn't work.

A lot of this job is "in theory", but hey, I'm happy for you if you don't have that experience. But it probably just means you're like my predecessor: intelligent but untrained in best practices, but mostly lucky.

[-]

reilogix@reddit

Hopefully you have been a Systems Engineer longer than you have had this Reddit account but your comment screams lack of experience.

[-]

p3ac3ful-h1pp13@reddit

Been in the field over 15 yrs and seriously? Our work is like scientific method. You start with the most likely hypothesis and test it with the theoretical solution set. You use probabilities based upon logs, metrics and monitoring to narrow down a solution. You test the solution and apply to change management with evidence to your claim. Once the change approval board approves the change, the lower teams will apply based on your documentation. What am I missing here? are you just a shitty sysadmin?

[-]

Tiddzz@reddit

Tell me you've never seen a computer/server break for no reason even after doing everything correctly without telling me.

[-]

p3ac3ful-h1pp13@reddit

Tell me you've never worked in an enterprise level environment without doing so lol.

[-]

saige45@reddit

Reminds me of the time I had to decompile a deployed dependency to prove to the owning team that their deployment/build failed. Sometime we get so stuck on believing our automation that we lose sight of "yep, ish happens"

[-]

p3ac3ful-h1pp13@reddit

Dude you'd never make it in a fortune 500 environment with that kinda sloppy work. No offense. It sounds like just cause you and your teams don't have a cohesive work relation together and debug code properly. Jeeze no shit silicon valley is getting taken over by immigrants lol.

[-]

Competitive_Tree8517@reddit

You're missing all the people that intentionally or unintentionally fuck this up. That's the system on paper. Definitely awesome. But in practice, it's often much messier than that. I'm glad it's nice where you work, though. 🙂

[-]

p3ac3ful-h1pp13@reddit

All code must be peer review friend

[-]

lilrebel17@reddit

Only helpdesk techs speak in absolutes.

[-]

BarnacleKnown@reddit

I was once told I never give absolutes.

I once heard a pm give 110% certain it would be completed by x.

2 weeks after that date their manager was let go.

[-]

Thats-Not-Rice@reddit

It was said you would fix the problem, not replace it with a new one! Bring balance to the network, not leave it in darkness!

[-]

s-17@reddit

We suspect it's resolved now.

We've received some reports that appear to indicate it's resolved now.

[-]

wrootlt@reddit

Word "seems" is in every second sentence of mine.

[-]

Inevitably_Expired@reddit

This, i used "seems" so much i start questioning the existence of the word itself.

[-]

Booshur@reddit

I always say "The issue looks to be resolved".. sounds less unsure.

[-]

alwaysdnsforver@reddit

Nothing has changed

[-]

robberjck@reddit

We are experiencing issues after we patched and rebooted, it has to be patching.

[-]

Icolan@reddit

Seamless.

Where I used to work we had a network engineer who put in a change that he described as seamless, he assured everyone that no one would notice his change.

He made the change, caused a problem with spanning tree and took down the entire network. He rushed back to the datacenter, logged onto a switch or router and stopped the spanning tree storm.

Then he went back to his desk and decided to implement his change exactly the same a second time, with exactly the same results.

After recovering the second time, he was all set to do it a third time when the director stepped in and put a stop to it.

[-]

cats_are_the_devil@reddit

That command doesn't affect storage paths.

[-]

Feisty-Shower3319@reddit

Ouch

[-]

cats_are_the_devil@reddit

It's okay the database has a fresh backup.

[-]

apandaze@reddit

"This isnt work related, but"

[-]

Commercial_Growth343@reddit

this change will be seamless

[-]

ImCaffeinated_Chris@reddit

"what could possibly go wrong?"

[-]

Igot1forya@reddit

I always think about this one "what are you gonna do shoot me?" -Guy who got shot

[-]

TwilightKeystroker@reddit

"All we (I) have to do is "

Signed,

Sales Managers Other/Ignorant /Selfish/Lazy coworkers

[-]

stratospaly@reddit

Non-production change, no outages expected.

[-]

bootlessdipstick@reddit

Don't worry, there's a snapshot.

[-]

hornetmadness79@reddit

Today is a good day to die!

[-]

Ivy1974@reddit

Fix it! What’s wrong? I don’t know…fix it! 🤦

[-]

Bimpster@reddit

let the interns do it

[-]

Micux@reddit

"That was in testing environment, riiight?" ^^'

[-]

Mesquiter@reddit

It's easy.

[-]

jpinoniemi@reddit

Quick question

[-]

Igot1forya@reddit

My provider is Network Solutions...

[-]

BrainWaveCC@reddit

Seamless integration
Fully secure
Fool-proof plan
Full end-to-end encryption
Single pane of glass
Fully tested upgrade
"This will be a temporary fix..."
"This will just be a brief question..."

[-]

Zangrey@reddit

Ah yes, the classic 'temporary fixes/solutions' that end up business critical and horribly permanent. Got plenty of those...

[-]

HorrorFlamingo3213@reddit

theoretically there should not be any impact of this change…

[-]

Ams197624@reddit

"It's OK to have all DC's running on the hyper-v cluster and have the hyper-v hosts administrator accounts managed by LAPS".

Until a power outage shuts down all hyper-v hosts that is...

[-]

Apprehensive_Bat_980@reddit

Good morning, afternoon, evening

[-]

CptBronzeBalls@reddit

“There’s no fucking way this is a DNS issue.”

[-]

Lordgandalf@reddit

I have learned always build in some Leeway it can always got pearshaped and quick sometimes so never say it's done then unless you're 1000% sure it's done then.

[-]

ryand32@reddit

It's definitely not ....

[-]

senolsun@reddit

It was working on local

[-]

Expensive-Rhubarb267@reddit

“Have you got 5 minutes?”

[-]

Parthorax@reddit

Wait a minute this is about last words…what do you do to people who ask that?

[-]

Inevitably_Expired@reddit

likely because that "5 minutes" turns into a 2 hour conversation that would've taken 5 minutes to read on email.

[-]

airinato@reddit

Probably means last words they hear before they self delete to get out of whatever hell they got pulled into.

[-]

BreathDeeply101@reddit

Also see: "Since you are here..."

[-]

Automatic_Mulberry@reddit

"We didn't make any changes." Followed by a creep to "We didn't make any substantive changes."

I think you can guess what the root cause was.

[-]

Inevitably_Expired@reddit

sounds like my network admin... "Hey X we can't reach x.x.x.x have there been any changes lately ?" "no i didn't make any changes" 5min later suddenly starts working again.

[-]

reevesjeremy@reddit

I was in a dentist office waiting room once with my work phone in hand. I received a message from someone in my org asking why he’s suddenly having to reauth every single hour. I pinged my colleague asking if he changed anything in the condition access policies. “No I didn’t change anything.” When I got back to my desk I pull up the audit logs which show that same admin edited a conditional access policy. I asked him about it. “Oh yeah, I did change that one.” He thought because he was viewing something that was labeled test, that the conditional access policies he sees was also test. Nooooo dude good grief. He could have done so much worse, so thank goodness it was just that.

[-]

techvet83@reddit

Related: Changes made without change tickets, so hours and hours can be spent by many people chasing down an issue.

[-]

UNAHTMU@reddit

I haven't submitted a ticket.

[-]

networkn@reddit

The ticket queue is really quiet

[-]

TheGreatNico@reddit

"I need you to make a Visio for ..."
Queue a week of digging through decade(s) old documentation written by people who have long since quit or died that reference departments, buildings, or countries, that no longer exist.

Hmm. I need to go dig up, literally, an old EE to explain this shit. How do you use fiber with an acoustic coupler? TF? Oh, this page is in... some language I don't recognize. Is that a vacuum tube? What does радиоизотопи mean and why is it in red?

[-]

Evernight2025@reddit

IT won't need to be involved

[-]

nnmcln@reddit

Exchange in the cloud is great, let Microsoft manage it.

[-]

i2noob@reddit

it's relatively quiet this morning. must be a good day

[-]

maglax@reddit

"Hey @Me, Good morning"

[-]

zaphod777@reddit

"I assume ..." followed by "I assumed ..." when it eventually goes sideways.

[-]

RegisHighwind@reddit

"What are we gonna do, break it more?"

It broke more.

[-]

Tymanthius@reddit

This shouldn't take long

[-]

Immediate_Client_757@reddit

Fuucccking learned to stop saying that one 🙃

[-]

RegisHighwind@reddit

Rule number one with my team. If we think it's gonna take 1 hour, we tell everyone 2. Under promise and over deliver

[-]

RegisHighwind@reddit

Years ago, we had a guy helping us stand up a new asset management system. He had written an automation script in PowerShell. He launched and said "This is my favorite part" and it hosed almost immediately. It's become a staple saying in the office when someone is working on something sketchy as a way for us to mess with them.

[-]

wrt-wtf-@reddit

"I'll just do my patch when you do yours"

[-]

adx931@reddit

I'll just edit the file on the live server.

[-]

Darth_Malgus_1701@reddit

"So I found this USB stick in the parking lot...."

[-]

steveatari@reddit

I know it's end of day Friday but it should only take a reboot.

[-]

stvdion@reddit

You should be all set

[-]

Pyrostasis@reddit

I know its friday but this is a simple update and its in test, it cant cause a problem trust me.

[-]

intmanofawesome@reddit

It’s always DNS.

[-]

ProfDirector@reddit

The haiku exists for a reason

[-]

Powerful-Cost-8387@reddit

Also "Who did that this way? Lemme just fix this real quick."

[-]

Virtual_Low83@reddit

"wr mem" followed by "reload"

[-]

Powerful-Cost-8387@reddit

"It'll just be a blip. No one will notice."

[-]

Happy_Kale888@reddit

This will only take a minute.

[-]

0RGASMIK@reddit

Should only take an hour. (From Manama to execs)

Was moving DNS from some legacy registrar to Cloudflare.

Of course the legacy registrar didn’t let you change the TTL on the NS so we were basically at their mercy for 24 hours.

Nothing bad happened but I was clenched for 24 hours waiting for it to fully flip over because they had some very critical LOB apps depending on DNS. I’m talking completely shutting down production if they failed and for whatever reason for the first hour of the change it kept going down. I was scared it was going to keep happening until the NS fully migrated but fortunately it was just an anomaly.

[-]

jeffrey_smith@reddit

Did the old DNS hosting have uptime problems? If the records aren't changing during the nameserver change surely there's almost no risk here? Just trying to understand.

[-]

jamesaepp@reddit

/r/ShittySysadmin/comments/1fgxy86/a_surprisingly_unshitty_dns_migration/

[-]

BoredTechyGuy@reddit

That will be easy…

[-]

Subject_Estimate_309@reddit

hey.

[-]

ApricotPenguin@reddit

"It's a real easy fix / change"

[-]

ThrowAwayTheTeaBag@reddit

This got me today. Fuuuuck.

[-]

tkecherson@reddit

When I tell my wife it's a 10 minute fix.

[-]

Krigen89@reddit

Should be 10 minutes
See ya tomorrow chief!

[-]

Jeff-IT@reddit

“Fuck it. What’s the worst that could happen”

[-]

Krigen89@reddit

Ouf, that's my natural style, too. "How bad can it really get?"

Yes sir, it can get pretty fucking bad.

[-]

S3xyflanders@reddit

This change should be quick
This update will be easy
I should be home in half an hour

Don't fucking say anything assume the worse and hope for the best :D

[-]

rswwalker@reddit

This shouldn’t impact production.

[-]

Conscious_Pound5522@reddit

It's security.

[-]

dcnjbwiebe@reddit

Oops!

[-]

Confident_Pop_9292@reddit

It's almost back up...

[-]

sole-it@reddit

Honey I got this small change to finish, and i will be home soon.

[-]

Regular-Nebula6386@reddit

Instead of half a day, can you not do the migration and upgrade in 1 hour?

we can try …

[-]

moderatenerd@reddit

This should take 5 mins

[-]

fagulhas@reddit

When was the last time YOU restart your laptop?

[-]

LaceyAtEvo@reddit

This isn't what you're supposed to do, but this is how we do it

[-]

jdptechnc@reddit

"should be ok'

[-]

Tarirai_Nkomo@reddit

Please try again…

[-]

rwdorman@reddit

Quick question...

There is no such thing.

[-]

Verukins@reddit

"Its a temporary fix"

Nothing is more permanent.

[-]

i-heart-linux@reddit

Hey are you able to pull a fill VM backup from tape?

[-]

jpm0719@reddit

What's the worst thing that could happen, it's already broken.

[-]

ISaidGoodDay42@reddit

Oh great Microsoft released an out of band patch at 3 PM on a Friday. I'll take care of this really quick and be home at my usual time.

[-]

mschuster91@reddit

"Legal has approved it"

[-]

recoveringasshole0@reddit

Simple patch on DNS.

I want to know what this one even means. Who's out here patching DNS? What is DNS written in anyway? Probably QBASIC.

[-]

GrimmReaper1942@reddit

“It worked fine until you….”

[-]

CollegeFootballGood@reddit

Lmaoo I legit had to reboot 5 servers after hours one night.

I sent an email saying I would reboot at 10:00pm eastern time. No impact is expected and the servers should be back online after just 10-20 minutes.

Oh how wrong I was

[-]

"It was approved by management"

[-]

Can you give $AccountingBob Domain Admin?

Will you add this firewall rule this coming Friday?

[-]

cammontenger@reddit

"Nothing else should be affected"

[-]

Hdys@reddit

Last week my coworker: “ We’ll be back in an hour”

Swapping out a codec last week that was end of life… 5 hours later we gave up and called our av vendor who fixed the issue we were having in 30 Seconds

[-]