The last two posts reminded me of a continuous network outage we had at one of our customers sites. It initially wasn't my problem, but decided to help out because of its stubbornness.

Customer comes in (after like two weeks, because why would you want to speed things up) and says their C&C machines lose Internet in the morning, from startup until anywhere from 15 minutes to 5 hours later. No other devices had this issue either.

Colleague didn't trust the small desktop grade switch it had, and replaced it with a new one, but this didn't solve the issue. We discuss with the vendor for a while, but they don't want to come onsite to troubleshoot with us and they can't remote in while the problem is occurring.

At this point I step in having trusted that my colleague has done the basic troubleshooting steps, which will come back to bite us later. Perhaps the internal nic of the machine is defective so we use a USB nic adapter, unsuccessfully.

I also setup an iperf/pingplotter kit and come across some wierd values. The network will come back online for 6 seconds every minute like clockwork, but this isn't enough for windows (or the application) to realize Internet is back up and running.

Okay, so something is definitely going on with the network. I rack my memories and recall we had an external contractor call us two months before if we had an issue with one of our AP's at this site (the answer was yes), so I called them up and asked what they did that day.

After a lot of back and forth, I learn that we had contracted them to install a switch and two AP's in/near a conference room. Now, normally this isn't a problem, you'd say right?

Wrong. Every day, this company turned off the main breaker to the production machines. And because the contractor pulled a cable from one of the C&C machine switches (instead of the core switches), it would cause the newly installed switch and AP's to lose Internet connectivity and establish a new one via mesh.

The switches and AP's we have are not smart enough to release a mesh connection if a wired connection appears again, so this would make a loop. Disabling mesh instantly fixed the issue, even though it caused a network disruption late in the day for the conference room.

Hours spent fishing for red herrings and talking to managment: 32 ish

Hours spent actually fixing the issue: 0.5

Hours spent trying to talk some common sense in my colleague and myself to check the basics first: infinity + ongoing