Outages?
Posted by emteereddit@reddit | sysadmin | View on Reddit | 79 comments
We're having some major internet issues at our site. I also see on downdetector a spike in outages reported for AWS, Lumen, CenturyLink, and others.
Anyone else having problems, or have any info?
CucumberDull6540@reddit
French here, same thing, a lot of online services are down/currently having problems, for example Discord, Google, youtube and reddit.
xendr0me@reddit
Here's the full outage log thus far newest to oldest:
2026-04-28 21:27:33 GMT - Splicing effort has commenced and the NOC is beginning to see alarms clear. Customer circuits will begin to be verified.
2026-04-28 20:30:50 GMT - Fiber is being pulled into the second fiber trailer and will be prepared shortly to begin splicing.
2026-04-28 19:34:19 GMT - Trenching continues with approximately 50 feet remaining, and the existing fiber has been prepared in the second splice trailer. Splicing will begin on the second side as the new fiber is pulled into the trailer.
2026-04-28 19:16:12 GMT - Splicing continues and is delayed on the first side of the fiber span due to old cable that needs to be handled carefully.
2026-04-28 18:37:23 GMT - The fiber cable is in the splice trailer and prepared for splicing. Trenching continues on the opposite side of the fiber span and is approximately 50% complete. Once trenching is complete, the new fiber cable will be placed, and a new handhole will be installed to replace the damaged one.
2026-04-28 17:45:51 GMT - One side of the fiber has been exposed and is being staged in the splice trailer in preparation for splicing. Crews are actively pulling the replacement fiber into place.
2026-04-28 17:24:10 GMT - Excavations continue to expose all the damage on the fiber.
2026-04-28 16:36:53 GMT - Excavations are underway to expose the damaged fiber.
2026-04-28 16:28:15 GMT - Construction and Splice teams have arrived onsite.
2026-04-28 15:40:34 GMT - Construction and Splice teams are en route with the estimated time of arrival at 16:00 GMT.
2026-04-28 15:04:00 GMT - A pipe being moved for the local utility company has been further isolated at the cause of the fiber damage.
2026-04-28 14:53:54 GMT - Multiple third-party construction activities have been identified, and a boring machine is suspected to be the cause of the fiber damage.
2026-04-28 14:33:27 GMT - The damage has been isolated, and a 144count fiber replacement cable, two handholes, and additional crews are being mobilized to the failure location to begin repairs. The current estimated arrival time is 16:00 GMT, traffic permitting.
2026-04-28 14:02:30 GMT - Teams have isolated a damaged 84-count fiber cable and continue to further pinpoint the damage.
2026-04-28 13:17:31 GMT - Field Operation teams are light level testing the fiber to isolate the damage.
2026-04-28 12:29:39 GMT - Field Operation teams are onsite to further isolate the failure.
2026-04-28 11:41:44 GMT - The Lumen Network Operations Center (NOC) has been engaged in reference to a suspected fiber issue and technicians have begun initial investigations as well as remote troubleshooting efforts. Lumen Field Operations have also been engaged to dispatch to assist with further isolation and manual troubleshooting efforts.
2026-04-28 10:26:43 GMT - On April 28, 2026, at 10:10 GMT, Lumen discovered service impact in Leesburg, FL. Your customer trouble ticket has been related to a network event; all network events are automatically escalated to Lumen leadership.
The Network Operations Center (NOC) is investigating to determine the cause. Please be advised that live updates for this event will be relayed at a minimum hourly unless otherwise stated or as relevant details evolve.
Eclias@reddit
Are we thinking the damaged fiber in Florida is related to the outages in Portland and Seattle?
spokale@reddit
That is what I was wondering, how the heck did that cause connectivity issues in Washington?
xendr0me@reddit
Anything is possible, they may have at one time relocated a subnet of client WAN IPs from Florida and never updated the regional DNS servers it gives out with DHCP enabled clients, so those clients were attempting to get the DNS servers located at the end of that cut fiber if they were not using their own static 3rd party DNS servers.
Just an example, and it could have been a number of things, but yeah, possible.
starclaws@reddit
I certainly hope the 3rd party digging company gets some fines and/or penalties for the damage they caused. Appears to be a huge outage.
Used-Opinion-8819@reddit
You are amazing.
ballgame1881@reddit
Happen to have a link to this?
xendr0me@reddit
Sorry no, updates are via e-mail and they don't post status publicly, looks like it's all behind ControlCenter.
Halfcore@reddit
doo dood o doo doo
Capt91@reddit
Anyone know if AZ is affected?
Halfcore@reddit
It is.
Global_Librarian1012@reddit
When this issue started we noticed many of our clients using CenturyLink network were not getting any packets sent to them from our servers in AWS us-west-2. We even had traffic mirroring enabled on the EC2 instances which showed the packets going out the EC2 network interfaces.
STORMBORN_12@reddit
Utah Lumen here seeing random website unavailability and 503 pages. Nothing major as most things are working but some pages that worked this morning are getting a 503 and then some that errored this morning became reachable later. Spotty, but enough to be hearing about it every other hour or so
j8048188@reddit
Seeing outages with clients in both New Mexico and Seattle. Not full outages, but random routes working while others fail.
j8048188@reddit
Stuff returning to normal for me.
JuggernautFar2612@reddit
We have a Lumen fiber circuit, and we're having problems routing to multiple services, but are fine connecting to the rest. Put in a Routing Issue ticket with Lumen about 20 minutes ago, but nothing yet.
JuggernautFar2612@reddit
All routing issues just resolved for us.
im_thatoneguy@reddit
Same
taemyks@reddit
Yup
emteereddit@reddit (OP)
Yep, same here. I can get to some sites and services totally fine, others are totally down or intermittent.
bisque1123@reddit
Circuts are coming back online
Maximus5684@reddit
https://downdetector.com/
It's nearly all services and providers at the same time.
Best-Impression2077@reddit
Same in Germany
im_thatoneguy@reddit
Lumen is one of the few Tier 1 backbone providers. So, them having issues can literally cripple the internet until the routing tables update to avoid them.
Kaligirlsam@reddit
Scarey
Break2FixIT@reddit
Who told the webserver guy to reboot the website?
Ill_Return_8627@reddit
A seemingly random assortment of sites and services are unreachable from a Lumen connection in Aurora, CO. Those sites and services are up and available from elsewhere in the country, via a different ISP.
I can't access duckduckgo.com 😫
Ill_Return_8627@reddit
The symptoms have ceased. Things appear to be working as expected.
Tye595@reddit
My work is down in PDX, phones along with it.
Can still use a RDP to connect to a terminal though which is weird.
Chopsticks613@reddit
Seattle area here, similar issues. Hope the phones don't blow up too much...
books_cats_please@reddit
Also Seattle area. Some sites load with no problems, others load very slowly, and some just won't load at all. We keep testing and thinking, "Oh the sites are loading fast it's fixed!" Only try another site and realize the problem is still ongoing.
Maleficent_Ad6318@reddit
Sam's issue. Also Seattle area
taemyks@reddit
Ours won't. All the SIP service is down
Available-Shine-5337@reddit
Enterprise lab down in downtown Seattle since around noon. We are CenturyLink, absolutely no traffic at all.
FriskyDuck@reddit
MN - no Lumen SIP outages
emteereddit@reddit (OP)
Appreciate the confirmation everyone!
We are Lumen/CenturyLink also.
I always feel bad that there's nothing I can do to fix it, but also super glad that it's not my problem to fix!
cohortq@reddit
I mean this is essentially why O365 great compared to Exchange. Not really your fault, just point the finger wildly at clouds and scream Microsoft.
brink668@reddit
Lumen has a fiber cut in PA from my source
Kaligirlsam@reddit
One in Florida also
Kaligirlsam@reddit
Its out in Lake, Marion and Citrus counties with multiple providers.
olydrh@reddit
Lumen fiber in PNW no issues today (yet).
spokale@reddit
You too? It seemed weird - we started with Qwest which became CenturyLink which became Lumen, to my knowledge we were never Level 3. It makes me wonder if this was a peering whoopsie
robrie@reddit
Down in Kent, WA
Opposite_Cycle_2055@reddit
Down in Portland
VERSACE_COCKRING@reddit
Portland, Oregon area. We have a routing black hole between Lumens Seattle edge and Denver edge sites.
skibumstevo@reddit
Ditto in Vancouver, WA
crunkdad@reddit
yes, we just had to change our dns forwarders from google to cloudflare
_Lowd@reddit
this works to get up and running again?
crunkdad@reddit
it fixed our onsite web traffic issues, remote vpn users struggling still
hopmastery@reddit
Down here too!
xendr0me@reddit
Anyone else hear anything back? I've not heard a peep from the PRI side of the house.
xendr0me@reddit
We currently have a PRI outage on Lumen in the reported map area of Florida on downdetector. Started around 2:50PM EST. Lumen, CenturyLink and Quantum Fiber are all showing a similar spike.
Luneward@reddit
Considering that's all the same company, not really a surprise. They'd all use the same central infrastructures.
xendr0me@reddit
Exactly why I posted all three names, as CenturyLink/Quantum Fiber are subsidiaries of Lumen
Not_So_Invisible_Man@reddit
Quantum was sold off to ATT, they may still be using their last mile infra tho.
xendr0me@reddit
Yeah that is accurate as of Feb 2026. I doubt they've had time to migrate traffic/customers fully so that's likely the case.
taemyks@reddit
Yeah, all SIP service is down
ManageExpectations@reddit
Down in Wyoming. None of the phone numbers work.
FLGuitar@reddit
I found this on google. I am going to take a wild guess where Lumen/CenturyLink/Quantum runn thier operations from. I was even having trouble getting the phone system to work properly calling in for support. Website is not working right either.
As of April 27, 2026, AWS is experiencing a localized outage in the ME-SOUTH-1 Region, specifically impacting the mes1-az2 Availability Zone. Power and connectivity restoration is ongoing; users with affected resources should move to unaffected zones or regions. No global AWS outages are reported.
OutrageousPenalty775@reddit
Using Lumen fiber in Los Angeles, our network went down briefly for about 4 minutes this morning around 9 AM.
RepulsiveDuck331@reddit
We are impacted too, seemed like Ninja RMM outage initially however Ninja is running clean per their staus page. Went on to look for common elements across multiple clients however did not find much there. What made most sense is DNS related outage at larger scale and that's what it seems like based on Cloudflare status page -
https://www.cloudflarestatus.com/
eufemiapiccio77@reddit
Why does this sound like a vibe coded thing. Cloudflare is investigating duplicated values in the X-Forwarded-For header. Origins depending on this header may exhibit unexpected behavior.
moonrakervenice@reddit
I don't believe that Cloudflare issue is related, that would not cause widescale disruption like this.
RepulsiveDuck331@reddit
yes makes sense, this seems more like ISP or cloud providers related issue
eufemiapiccio77@reddit
Ah that makes sense I was having problems using DeepSeek and a few other sites
bisque1123@reddit
MSP here, at least 25 sites impacted on Lumen circuts
moonrakervenice@reddit
Lumen definitely having an outage: https://statusgator.com/services/lumen
Dhk3rd@reddit
Lumen = CenturyLink
Luneward@reddit
It's the same company. One down usually means the other is down for a fiber cut. Usually only one goes down if you're looking at commercial or residential equipment failure in their central offices.
CeC-P@reddit
I'm seeing very, very unusually bad bandwidth throughput and ping times at some client sites.
JuggernautFar2612@reddit
We have a Lumen circuit here, and we can route to some things and not others. I put in a ticket about 20 minutes ago for Routing Issues, but am still waiting on a response.
Eclias@reddit
Just had to manually flip a client over to their backup Comcast connection. Lumen was still "working" enough that the auto failover didn't kick in like normal.
KillingTime1212@reddit
Our Lumen fiber bounced a few minutes ago.
Evening_Plan_2302@reddit
https://www.cloudflarestatus.com/
Nothing from AWS yet tho
xendr0me@reddit
Yeah I don't believe it's AWS or GC, it's likely a fiber cut on the Lumen network. I'm waiting on a call back for our ticket and will provide an update if they have anything insightful.
RyanLewis2010@reddit
2 Lumen Sites down here with DIA fiber.
norsewrath91@reddit
Yes we are seeing a lot of issues with our customers
emteereddit@reddit (OP)
https://imgur.com/QBXoQ3Y