7:00 PM Network Outage

Posted by wanderdive@reddit | talesfromtechsupport | View on Reddit | 36 comments

Yesterday I read the post about the 7:00 PM WiFi disappearance and it reminded me of this one. It didn’t technically happen to me in person, but it was my team and my boss at the time.

This was maybe some 20 years ago, I was working IT support for an oilfield service company, and was taking care of a large operational site along with 5 remote ones.

One of the remote sites was a training center, and they started to have a weird issue. Exactly at 7:00 PM like clockwork part of the building loses connectivity. We tried looking for any issues remotely but couldn’t find anything wrong, only that the distribution switch for that part of the building in unreachable.

So finally my boss visits the location and at 7:00 PM, poof! He does the troubleshooting and the both the core switch and the distribution switch are up, but no communication between them. Other distribution switches are connected with no issues.

The next day he started to observe anything and everything that happens on the site at 7:00 PM. One of the things he noticed was that this was when it starts to get dark, and the security person starts making his rounds, and turning on all the huge stadium-like lights for the site. And this is the exact moment the connectivity is disrupted.

Turns out they just installed new lights, and the lazy money-saving contractor extended the high power cable through the network ducts that housed the Ethernet cables (later upgraded to fiber optic) connecting the switches, instead of laying new ducts.

Once he had the cables separated everything worked perfectly as before. Mystery solved.

[-]

redhairarcher@reddit

My favorites:

Remote office server unexpected shutdown daily at 5pm. Cause: Power socket was connected to the office light switch.
Irregular los of connection (line of sight radio transmission) between to office locations. Cause: A bridge broke the line of sight when opening.

[-]

KelemvorSparkyfox@reddit

That first one would never happen over here (UK). Lighting and power sockets are on different circuits.

[-]

bhambrewer@reddit

That's the way it's supposed to be, but there's a lot of really stupid people out there.

[-]

redhairarcher@reddit

That's the normal way of doing it here as well, that's why it took quite some time to find out what happened.

[-]

frac6969@reddit

Our story was old DVR always hangs and needed regular rebooting and the new HR guy would turn off our rack UPS instead of just turning off his DVR.

[-]

fresh-dork@reddit

irregular packet loss: network feed goes under nearby train tracks and gets squished. that one was frustrating because we identified the problem and then watched for a year or two as nobody followed up

[-]

TheArmoredKitten@reddit

That's hysterical. Was it distorting a fiber or something? Was it interference from all the moving steel?

[-]

fresh-dork@reddit

i think the former. we'd get high error rates and other problems that i think were from crushing fiber. we also got network saturation at 11a, but that's just the workers using YT at lunch - nobody's on the machines, so who cares.

ontheroadtonull@reddit

My uncle relayed to me a story about a microwave link between two buildings that started randomly going down.

Between the two buildings was an airport runway. It took several trips to the tops of those buildings for someone to notice that the airport had just finished an expansion that allowed them to land 747s. The 747s had a much taller and larger tail than any other aircraft that would land there.

Any time a 747 would pass between the buildings the tail would attenuate the signal enough to bring down the link.

Infinite-Land-232@reddit

A data center manager I knew was having difficulty with power spikes and took this up with the electric company and ended up inspecting the power line leading to his building with an electric company employee, both of them on foot. The log of spikes on the manager's clipboard correlated well with the age of the decomposed carcasses of electrocuted crows. Power line was okay.

That's surprising. Crows normally conduct something akin to autopsies with sudden or unexpected death, for the safety of the murder.

Unfortunately, CROWSHA has not yet discovered voltage or writing, so their report was inconclusive and not well circulated in the affected industries

Empty_Rutabaga_4649@reddit

Upvote for CROWSHA!!

Yuri-theThief@reddit

If there are any fans of TTRPG's here, please know; that in some distant future I will have a kenku organization of CROWSHA.

scyllafren@reddit

My favourite is the 7AM morning wifi down for 15 minutes in one of our office building. Not exact time, +-10 minutes, and only weekdays.

Site visit found out, that the wifi's power adapter wasn't put above the false ceiling, but was plugged in to a DADO (cable tunnel usually waist high on walls), and the cleaning lady needed a power socket for her vacuum... :D

DuckyDoodleDandy@reddit

There was a similar thing at a nursing home where any patient on life support in a certain room died on Friday morning. It turned out that the cleaner was unplugging the life support for ~5 minutes so they could use the outlet to power their cleaning machine (floor cleaner of some kind), and then plugging it back in.

KungenBob@reddit

Manslaughter, I hope….

Absolutely_Cabbage@reddit

Its an urban legend

snopes debunked that one

techslice87@reddit

That is terrifying

This was decades ago, and now such machines have features to prevent it from happening again, but like most safety features, they were created because someone died.

Terrible_Shirt6018@reddit

Yup, that's something solved by incompatibility. Server rack PDU sockets and plugs are usually C13 and C14, wired directly into the UPS. Can't plug a vacuum into that without adaptors.

Do you store wifi devices only in comms rooms? :) you can't use those sockets outside of comms rooms easily, especially if the power plug and adapter are in the same casing as with most devices this type.

APs are usually PoE. No power bricks needed.

Now yes, but look for a Netgear N150, that laughs at your PoE switch :)

Didn't something like that happen to the original DNS server? Like the first one that was just under some guys desk for a week

MikeSchwab63@reddit

I remember reading about a 1 hour outage starting about every 12 hours 25 minutes. It was on a island and at high tide the water was close enough to line of sight to block the signal.

henke37@reddit

A classic.

SteveDallas10@reddit

I was called out to a certain big box store about a network switch that was offline. When I arrive, I get a scissor lift to the network cabinet containing the switch and find that it is on and blinking away.

I call the NOC and the agent pulls up the history. It seems to be unreachable every evening after 11PM store close and reappears every morning around 6AM, before store open. The switch serves the electronics and photo departments. NOC agent is wondering why this was even dispatched for a priority service.

Turns out that the circuit that supplies the TV wall is on a timer and some bright young electrician ran power for that IDF cabinet from the nearest circuit, the wall of TVs, since the IDF is directly over the TV display.

I told the manager that he doesn’t have a network issue; he has a power issue (that may have occurred during the most recent remodel).

This was at the Mall of Warts, of course.

onebitcpu@reddit

In the early 90's we had a customer who would have random server crashes.

They rented construction equipment and their server was in a construction trailer added onto the shop when they needed more room.

During an on site visit they had another server crash just as a road works compaction roller drove past. This particular one could not turn off the vibration as it drove.

3 inches of foam isolation later and the server crashes stopped happening.

blind_ninja_guy@reddit

kind of like the guys who screamed at hard drives to induce higher error spikes as a demo.

That reminds me of my favorite error message ever.

Some time in the 90's I think one of the big software companies in my industry made the cover of the North American industry association magazine. There was a picture of their software on a pc, with the words "General read error on drive C:" on the bottom of the screen.

SafariNZ@reddit

Reminds me of the dropout every Friday at around 3pm on a microwave transmission link.
They eventually sent someone up the tower with binoculars and found a truck parked at the side of the road to have their afternoon break. It happened to be in a gap between hills where they had routed the signal. They ended up having to raise the towers.

panamanRed58@reddit

The best is the original story from this line of tropes. Way back, early in the 60s, Sperry Rand was top dog in this realm. Or at least they thought they were. They installed and configured a massive machine for an international customer which cost many millions, even in the 60's.

Shortly after turning it over to the customer, the customer noted a regular crash of the system and called Sperry back on site. So the hot shot engineer spent considerable time on logs and troubleshooting but still every night the system would fail. The engineer working thru reams of reports, writing to pull all his data together but he was still stumped. He tried and tried... it was getting very late but he was determined to break through the issue. He hadn't even reproduced the error.

As he sat there feeling the pressure of a possible major client rejecting the installation, knowing that his time was costing money as well, he pondered and pondered. Sitting at his desk under a pile of reports in the dimmed office, his elbows on the table and hands holding his weary heard when housekeeping came in. She reached for the light switch and popped the lights on, while pulling her vacuum through the doorway. With a nod the computer expert, in one move she reached down to the wall socket to plug in her gear by swapping out a plug. Crash goes the computer.

Throwaway_Old_Guy@reddit

That is the reason why my former Employer in O&G specified shielded data cables.

Too many things were operated remotely to take a risk.

FanMysterious432@reddit

When I am working from home, the VPN connecting me to my computer in the office stops at 11:24 every morning. No idea why. I can always reconnect immediately and I never lose work, so I am not worrying about it.