Power Outage Emergency Plan?
Posted by DeluxiusNL@reddit | sysadmin | View on Reddit | 59 comments
I'm sure most of you already have UPS units in place to handle short power outages. However, the 24-hour power outage that occurred in Spain last year has prompted European authorities to issue warnings that such events are likely to happen again—and potentially last even longer.
When you think about it, there’s a useful way to look at the problem through a matrix with three dimensions:
- Duration of the outage (Powerdip, 4 hours, 24 hours, 72 hours, longer)
- Scope of the outage (within your building, across your city, your state, or even the entire country)
- Impact Type – What areas are affected (e.g., IT systems, safety, operations, logistics, customer service)
Given this reality, have you considered developing a plan to cope with extended power outages?
Ziegelphilie@reddit
Boss says they're not needed, fuck my life
rcp9ty@reddit
UPS for the first couple seconds for little blips. UPS for power outages for up to 5 minutes. Natural gas generator for the rest. Also, if you're looking for something that's long term like on for a long time instead of a backup generator you want a prime generator.
phalangepatella@reddit
Of course they are. And if you know that, you know the infrastructure it would take to even entertain the idea.
Our burn rate for a down production day or two per year comes nowhere close to what it would cost to fuel a 2MW generator, let alone the cost of install and upkeep.
phalangepatella@reddit
We have UPS and generator backup. We can keep the entire network infrastructure running as long as there is natural gas supply. We can also power most of the offices and the entire Engineering department.
However, we’re a manufacturing facility, and none of the ~60 welders, 100 hp, 75hp and 25hp compressors, or any of the shops at all have backup. The shop has 1600 amp, 600 volt power so I’m not sure if we could generate enough.
databeestjenl@reddit
2MW generators are a thing.
wild-hectare@reddit
purpose built tier 3 data center, tier 1 regional colos & public cloud recovery plans
if your business is technology dependant, continuity planning is not optional
Darkace911@reddit
We have one due to very bad power input to the main building. Basically, if an outage goes over 60 minutes power the clusters down until power returns.
NoDowt_Jay@reddit
We’ve once had our building running on UPS + Generator for about 3 weeks. Building manager had to keep topping up the diesel.
Dal90@reddit
VP of Infrastructure was told the data center UPS batteries needed replacement.
He spread it out over two years budgets...
Then one day Facilities did their normal 10am quarterly generator test and a third of the data center failed and we ran on the diesel powered generator (for the entire building -- ~400 employees in that one) for three weeks until the new batteries were installed and there was a weekend time frame to fail back to utility power. (Third of the data center because few people ever paid attention if redundant power supplies were each plugged into different circuits fed by different UPSs)
MrJingleJangle@reddit
Was that data centre in Croydon, UK perchance? Judging by the weeks of black smoke from the roof, this building experienced something very similar…
admlshake@reddit
We put in a generator about 9 years ago, our CIO really pushed for it when he came on board and was shocked to find we didn't have one. The biggest critic of it, argued that we had been running this place with out one for over a decade and three wasn't any reason to spend the 30-40k on something we'd never use. Even though we had constant power issues and lost power a few times a year (usually for a few hours). 2 weeks after the generator was installed, we had a massive storm blow through and knock power out to the whole area. Then because our luck is just shit, right before they got out power back on, a semi driver hit something and caused a bunch of equipment at the substation near us to fail. But our generator (natural gas powered thankfully) chugged along and our building stayed open and our services up. That critic never spoke of it again.
BinaryWanderer@reddit
Same. And its first test was the day it was installed. Someone blasted a power pole down the street right as we weee bringing our gear back online after installation.
“Who’s fucking testing the generator again!?”
… oh. OH! Hahaha, it works!
Tymanthius@reddit
That's actually funny as hell.
Yea, many places don't need a genny. But those that do, the ONE time you use it a year/decade pays for it.
BinaryWanderer@reddit
Yeah it was for a school which normally you wouldn’t. But they started offering online classes and evening remote studies hosting it internally. It became profitable and the budget for power improvements came shortly after a six hour power outage.
Many internal pissed off people realized they had the power (pun!) to prevent this had they approved the capital expenditure for a generator.
SperatiParati@reddit
UPS covers the switch-over, then we're onto diesel generators.
We have an on-site fuel bunker, and mobile bowsers to refill the generators, but for an extended widespread outage, we'll be doing graceful shutdowns of IT kit, as there is an expectation that we won't be able to refill the bunker.
If the outage is limited to us (e.g. our private HV network has popped a transformer), we'd probably shut down HPC clusters once their generators are running low, but refuel the generators running campus/corporate datacentres.
If the outage is wider than just us, we'd expect that fuel will be rationed, and hospitals and local government will get priority above universities, so we'd look to shut down all IT at some point to keep long running research going.
IT is just one small component of the business continuity planning here. We have resident students - once fire alarm and emergency lighting backup batteries have drained there's an immediate H&S risk. We've had a building lose power for 48hrs before (external cable fault which kept tripping breakers) and we evacuated the students to a sports hall.
We can do that for one, maybe two buildings. If we lost the entire campus, our Estates and H&S colleagues will have issues to plan for.
I can be reasonably confident in a managed shut down and start up when Estates or Gold Command tell me I'm 1hr from running out of diesel, and not a priority for a refill.
vlaircoyant@reddit
Sounds like a plan.
davidm2232@reddit
TIL that bowsers are not just from Super Mario. Neat!
SperatiParati@reddit
Yeah, although I guess both could breath fire....
Not sure of the exact model our Estates team use, but they look close to these:
https://parts.clarkemachinery.ie/product/cashels-1360-litre-fuel-bowser/
gmitch64@reddit
We have a redundant UPS for our data center that will last for 15 mins currently. Our generator fires up and kicks in in 12 seconds. We test it twice a year.
PoisonWaffle3@reddit
All of our sites (datacenters, headends, and offices) have UPSes for all equipment, plus generators that power the entire site.
We do monthly run tests on all generators, and quarterly load tests (where we actually fail the site over to generator power instead of grid power for 30 minutes).
We have a vendor that manages the diesel fuel at each site.
In the almost 8 years I've been here I've seen only one impacting loss of power, and that was in a site that was being decommissioned and the site monitoring system (ATS trigger, generator run sensor, fuel level sensor, etc) was unhooked prematurely. There was an extended power outage and we had no idea that the generator was even running until it ran out of diesel and the site went down. Luckily there weren't too many things left there.
leftplayer@reddit
I’m not in such a position anymore, but if it were up to me, my backup plan would be solar PV inverters + batteries.
Predictably recoup the investment even if you never have an outages (the money guys love that) and if you do have an outage it’s an instant switchover so your UPSs can be smaller (therefore cheaper), and your backup is “fuelled up” every day.
reol7x@reddit
If power is out block or city-wide for several days, our plan is to head on down to the Winchester till it blows over.
Sure, we could keep everything together on our generator, in theory as long as the gas is flowing in the pipes.
How long will the Internet be up? Once that's down, there's zero point for us running on on prem infrastructure.
theoreoman@reddit
For us it was simple, if there's a power outage, no one is working, so we don't need any equipment
Coldsmoke888@reddit
My sites all have diesel generators that power the critical circuits like server rooms and IDF closets. UPS backups in all of those to float outages for an hour or two depending on load.
Lose the generator and everyone goes home for the day. ;)
SUPERDAN42@reddit
Generator + Full UPS for 30 mins. Generator under contract and maintenance / Tested regularly.
natefrogg1@reddit
We had a couple Tesla Powerwalls installed. One issue we had was that the wiring in that building has sprawled all over the place over time, on the same sub circuit that my infrastructure is on there are a ton of industrial sewing machines. If power cuts out we have ~40 circuits that need to be manually turned off or else the sewers and their machines will exhaust the powerwall in about 4 hours. Without that extra load, infrastructure can run for over a week.
zombieblackbird@reddit
UPS/battery is short term, giving the generators time to spin up or commercial.power time to recover. Most of my sites have natural gas generators, some use diesel.
fdeyso@reddit
The onsite diesel generator should last for a while.
dracotrapnet@reddit
Our power outage plan is in response to a business plan set up by the execs. The business is primarily a welding fabrication shop with large electric cranes, air compressors, and other monumentally large 3 phase motors. Lighting and heavy lift equipment is essential for safety. Nothing can be done if that is down. After 45 min of a power outage with no estimated time to restore, a plant manager will dismiss workers and have them clock out and go home. Next shift will be notified if they will be unable to start work at the beginning of their shift or if they are to delay arrival and short shift hours.
Far as compute goes, all the servers are in a COLO.
The COLO does their best. They failed last year during a hurricane - they had 2 of 3 generators fail and they cut power to all customer equipment around 2 pm Monday and restored Wednesday at midnight. 5 am I was up to check if I had power/internet, I had internet, started a generator and VPN into work to start up servers at work. We had one PDU fail badly which resulted in strange bank 1 of 2 cycling at random intervals until it could be replaced the next week - lucky I had a spare and lucky everything on that PDU had dual power supplies. Meanwhile I had a 5 day power outage at my house, lost cable tv/internet and cellular before the storm landed, My fiber internet dropped Tuesday 2 pm and returned Wednesday. I ran a generator all week and played NOC/last helpdesk with internet from the couch in front of a fan with 89F indoor temp vs the 98-105 F outdoor temps.
On site networks, we have UPS but they last anywhere from 15-45 min at best with the exception of a few that have hours of storage. The concern becomes no cooling capacity is on any backup power system so heat becomes a problem after a while. It's best to just gracefully run out of power and shut off. Camera systems last as long as UPS across the network does. Door systems will operate up to 24 hours on their own batteries. Some doors may need to be manually locked for long power outages.
One site had a long outage and decided to get a rental trailer 3 phase generator they had on site hooked up to the main building's power (read main building as collection of office trailers stitched together wildly). A completely manual swap over. IT had no idea power was out for a long time and running on a generator as we had nobody on site. We just thought power was really unstable. One UPS was angry, very angry. It refused to operate on the power supplied. It wasn't until someone called and complained the network wasn't working. Yea, power wasn't supplied properly to all the UPS, just one and it was continuously going to battery untill it died. 3 hours later power was restored to the area and they undid their manual generator tie in.
DestinyForNone@reddit
Well... We have a UPS that will last us about 30 minutes on battery.
We have two sources of backup power. An onsite solar farm, and a dedicated natural gas generator.
Beyond that, we'd probably truck in a generator to keep the servers up.
jtbis@reddit
We have diesel or natural gas generators that can keep critical equipment going for a couple weeks.
The problem is powering up a building to the point of employees working in it is absurdly expensive. A generator capable of fully powering even a small office building costs several million dollars.
Meecht@reddit
Our HQ has a natural gas generator that tests every week, and enough gas in the tank for a few hours. However, the generator at our secondary site has a direct connection to a natural gas supply line, so that site can run so long as that pipe stays intact.
wideace99@reddit
Using large generators (2 pcs) each one on a trailer, both connected to an ATS of the datacenter. Those generators can run on gasoline, LPG or Natural gas. Since the place has already the heating based on the natural gas grid, same used for the generator with unlimited natural gas without the hassle to refuel.
Towing one of the trailers with a family car for yearly maintenance shop is easy & cheap, while the other one remains for emergency.
Stryker1-1@reddit
This is honestly why I put critical infrastructure in the cloud. I don't have to worry about UPSs, generators, fuel delivery etc.
Power goes out I get an email telling me new provisioning is suspended and the site has x hours or days of diesel with a delivery scheduled for x hours or days.
punkwalrus@reddit
I worked for a data center that had a generator backup in the parking garage. During a site inspection, they noticed that "the tags have not been removed." That meant the generator had been installed and set up, but not actually run and tested, and still had "remove before starting" tags in various places. For maybe 7 years? Needless to say, when they tested it the first time, it did not run. We had to pay for massive repairs because it just sat idle since it was put in and never actually spun up and tested.
MrJacks0n@reddit
Generators large enough to handle almost the entire building with UPS for a few minutes while the generator gets up to speed and quick blips.
j2thebees@reddit
As someone else said, long enough for graceful shutdowns. My primary gig is in an equipment fab. If you can’t run lasers, welders, etc. everyone needs to go home.
We keep the UPSs in good shape for about 2 hour outage. In the 8 years I’ve worked in this spot, only once was there a 3.5-4hr outage, with minor blips occurring probably 1-2x annually.
LeeRyman@reddit
I maintain the equipment at a volunteer marine rescue base.
We have: * ops room (three radio operator stations), phones and the essential network infra on 8hrs of UPS. * search and rescue command centre (one radio operator station) on 1.5hr of UPS. * backup radios on 8 days of battery. * the entire facility on an Automatic Transfer Switch fed from a 3-phase generator with remote start. (Gets tested on-load automatically every first Monday of the month) * redundant internet connections, one via a wifi link and its own UPS and different ISP for path diversity. * backup 4G modem with antennas mounted high. * a load-shedding and refueling plan. * prtg monitoring with push notifications to key individuals.
The entire area lost power, cellular and broadband in Feb for almost four days due to a low pressure system. We were the only place (apart from the local hospital) with power and comms. Whilst no one was out in the mess that time, we've had other low pressure systems take out the region for similar durations and been able to receive MAYDAYs and conduct rescues. This all through community donations and too many hours by a few people.
About the only thing I wish we had was a diesel tank on a bunded stand and better diagnostics from the ATS - working on that.
(It is a deficiency of the National Broadband Network distribution hubs and cell towers in Australia that they only have, say, 4 or so hours of battery. If power doesn't come back by then it suddenly becomes very hard to call emergency services, let alone advise family know you are okay. With worsening climate and weather, and the loss of POTS, the role of emergency monitors on CB and amateur bands will probably become more important.)
davidm2232@reddit
Our main sites all have had generators. At my current job, originally, just the MDF had generator backup. The UPS's in each IDF only had a 45 minute runtime. With all the power issues we have had over the years, we finally ran a line to each IDF for generator power. That was over a year ago and the power has not gone out since. Either way, a generator is a necessity imo.
tankerkiller125real@reddit
Our UPS lasts around 40 minutes, which is enough time to switch everything over to a generator we have which should be able to run everything for around 10 hours on a full tank.
CeC-P@reddit
We just had a meeting about this. All non-important sites we attempt to safe shutdown (but powering your modem doesn't guarantee internet for equipment remote control) and then for primary, we test a generator monthly and know approx how many KWH each gallon of gas is so have a can of 91.
Otherwise we send out a company-wide email from our phones about the outage warning of primary non-cloud servers and services and then shut it down.
jamesaepp@reddit
We're in two Equinix datacenters which are on separate power grids.
Funny enough I was thinking about this risk but for a different reason. This is anecdotal.
I live in Manitoba, Canada. The bulk of our generation comes up north from hydroelectric dams. That means there are a lot of transmission lines that travel north-south to get the supply to the demand.
But very recently we had a large number of ""wild"" fires in the north and I was doing some amateur cartography and ... yeah ... those fires were getting relatively close to the transmission lines.
BinaryWanderer@reddit
Those are fun conversations to have with the resiliency officers.
Hey Bob, I know you’ve been polishing up our disaster plan and accounted for everything… I just noticed this one new thing….
(Death Vader: nooooooooooooooo)
jamesaepp@reddit
Honestly because it's an anecdote and I'm an amateur, I don't put much weight in it.
As politicized and mediocre as our grid is, as long as no one else is making a stink about it, I figure that's a good "calm down, you're too ignorant to make a conclusion" moment.
ncc74656m@reddit
I mean it's obviously a question of what your tolerance is. We are a mid-ish sized NFP where literally everything is now cloud based. I nuked our one very old very badly set up server when I came aboard. Now we are functionally just keeping up the internet access and cameras. We also have a pretty robust WFH policy, so everyone could just... go home.
That said, I've worked for major hospital networks and large orgs with critical uptime requirements that had to have a lot of dirt side servers and appliances. For those, not only did we push generators with heavily redundant UPSs with ample extra capacity, but we also had annual generator testing because obviously.
If you can't have any outages, gas is the way to go, of course, but we had diesel at one venue and damn near ran out more than once. It once took some very angry emergency calls to the diesel companies to remind them that our lawsuits would ruin not only the company but everyone involved with running them, lol.
DheeradjS@reddit
We have UPSs that handle about 20 minutes and Generators that can keep us going for 4 hours.
Really, they are there to give us time for gracefull shutdowns. If te powergrid is actually down like Spain had, we're not going to bother with much more than that.
It'll suck for us and our customers, but seeing as nothing will impact personal or public safety, we got bigger issues. Like the damned Powergrid.
BinaryWanderer@reddit
I worked for a grocery store that lost hundreds of thousands of dollars in frozen and refrigerated foods because of a four hour outage.
Insurance company actually paid to install a generator for the cooling equipment after that.
LeaveMickeyOutOfThis@reddit
Standby generators, with contracted fuel provision via two independent sources (in case there is a rush on fuel), and a cell plan with higher priority for ops center staff (in case cell towers need to limit traffic due to disaster conditions).
NightMgr@reddit
The National Guard is charged with bringing us diesel, food, water…
Hospital/ trauma center for region.
angryPenguinator@reddit
Last year we put a plan in place to use a natural gas generator to keep the data closet humming along so our users can work remote - we have about 2 hours of UPS (more if we shut down some switches) before it kicks in.
Natural gas means we can basically run forever on them, since it is tied into the natural gas main.
TopGlad4560@reddit
I’d recommend mapping out critical systems first (things like network, access control, HVAC for server rooms) and identifying what can run on extended battery vs what needs generator support. For longer outages, consider agreements with co-location providers or cloud backups you can activate temporarily. Also worth testing what actually happens after 4, 24, and 72 hours of downtime. Most orgs assume the UPS buys enough time, but things get messy fast beyond a few hours.
dirtymatt@reddit
UPS and a generator that covers critical IT and life safety systems. If the power is out longer than that can handle, we probably have bigger issues that are way more important than anything we run. No one will die if our stuff goes dark and most of our users would be impacted by the power outage too.
gumbrilla@reddit
We're fully cloudy. So server infra is not an issue, our head office is out of power today, late notice planned work ironically, so everyone just stays home. No issue.
For a national power outage however, we are big in the shit I think,
I've not thought about it. As a sysadmin, I'd be getting into a car and going to somewhere.. likely Germany (I'm in the NL). Given EU grid and that's massive, if that all went down and - I'm in the NL, maybe the UK, assuming transport is working across the channel, otherwise the Nordics, they have their own grid and a roadlink, but it would likely suck, and actually I'd be looking after family first..
ntengineer@reddit
Multiple UPS's fed from big generator. Contract to refill diesel tank every 8 hours
Gold-Antelope-4078@reddit
Shotgun shells and canned beans.
GamerLymx@reddit
the shutdown lasted almost 12h. Spain opted to not power the trains for another 12h.
if you don't have a generator like in my case, you need to make a tiers of machines and services to shutdown. in my case it's:
1° research, development, virtual desktops and everything that uses GPU.
2° non essential virtual machines
3° Redundant machines like secundary dns, dhcp, ldap etc.
4°everything else
hkeycurrentuser@reddit
We have critical infrastructure in a tier3 co-location datacentre. The rest will be annoying to be down but not company ending.
We could do more of course but it's a risk/cost/reward balance you gave to play.
swissthoemu@reddit
We did when the Ukraine was attacked by the Orks and electricity prices went bonkers. Our most important locations are independent for 8-10 hours now, after that we just shut down and wait. We’re not critical for public infrastructure though.
TehH4rRy@reddit
We have generators on site. UPS covers us for the switch over from mains to generator. As long as we have plenty of fuel we can run for ages.