One of our stores had a power outage. Not a huge deal, it happens, we can deal. Due to some recent work on the network, however, we didn't have everything in the network closet running on the UPS (yet), and when the power came back on, the SDWAN box - the one single point of failure - didn't boot up correctly. It would route WAN traffic, but not LAN.

Note: When we upgrade our phones to VOIP, we pull cable for each phone, separate from the computer network, so the phones are entirely outside of our network. More expensive, but there are . . . reasons (and good ones). The SDWAN box thus has (or, rather, uses) two LAN ports, one for us, one for the phones.

Our telecom provider had, of course, detected the outage and automatically opened a ticket. They detected that the SDWAN was back up, and closed the ticket. The store was still offline, and was less than 100% efficient at letting me know, but when they did, they told me the phones were still down as well. This was the second store this exact thing had happened at in two weeks, and the first one took an on-site visit by a tech to reconfigure the SDWAN box. This was late on a Friday.

Once I found out, I reopened the ticket, and explained that with the phones down as well, it has to be the SDWAN box that's acting up. They looked at it, saw the SDWAN was showing as online, couldn't see our firewall behind it, and closed it again. "We show you're online in the orchestrator." Sigh.

So I reopened it a second time, made it clear we were still offline, and so are the phones, which are outside of our network, same result, same message, word for word, closed again. This is Saturday afternoon. We've been offline for two days at this point, which is painful, but not a huge deal, but the phones have been down as long, and that is a big deal.

So, OK, we're a six figure a year account (with about two dozen locations now) with a platinum support team, so I emailed our account rep and asked her to escalate the ticket. It was Saturday afternoon by this point, so I didn't actually expect a quick response. I was wrong. Less than five minutes, I get cc'd on the message to the escalation team. And less than two minutes after that, I get an email from the escalation team. Woo-hoo! Things are happening!

The SDWAN is up. I do not see your internal network equipment switches/routers/firewalls).

Yeah, things are happening, but all progress is backwards.

For the fourth time, the phones are not working. The phones are not on our* network. They go through a separate switch that plugs directly into the SDWAN. There is no part of our network that can affect phone service. The SDWAN is not communicating with its own LAN side.

For the fourth time, this is the exact same thing that happened after a power outage at another location last week that required an on-site visit to reconfigure the SDWAN because the configuration got corrupted by the power outage.*

I did get an apology after that, and he was able to get into the box remotely and get it working in less than half a hour (unlike the previous incident).

I guess there are advantages to being a big fish in a medium sized pond, and they're better than any other phone or internet company I've dealt with, but there are times . . .