Texas Low allows Disconnecting Datacenters Power from Grid during Crisis

[-]

ActualModerateHusker@reddit

Iowa has been forcing people to stop watering their lawns because the water is getting used up by new data centers for cooling.

At least know if you need drinking water in an emergency you may find a large supply at a nearby data center

[-]

You DO NOT want to drink that water. I was on a project building similar equipment cooling systems, it is not safe to drink at all. Lots of heavy metals and other toxic crap. Maybe with a really good filter in an extreme emergency.

[-]

ActualModerateHusker@reddit

So not only are they using municipal water they are also ruining it? These data centers don't seem any better than petro chemical plants

[-]

BBQandBitcoin@reddit

Well… let’s see.. yes, these data centers run municipal water across their gigantic radiators, then that contract out disposal wastewater tankers to transport to waster treatment facilities, once “treated” water goes back out into your local streams, tributaries, rivers, etc. (your watershed).

The byproducts on those data centers are definitely hazardous especially if the system is leaking.

refrigerants & biocides are environmentally hazardous

[-]

SeigneurMoutonDeux@reddit

Fun fact: Treated water doesn't have to be potable if it's not being reintroduced into a drinking system.

Rivers and lakes aren't considered drinking systems.

[-]

HomoExtinctisus@reddit

Rivers and lakes aren't considered drinking systems.

Not anymore anyways.

[-]

SeigneurMoutonDeux@reddit

I should have qualified my statement with "legally" to make it more clear.

"Legally, in the context of reclaimed water, rivers and lakes aren't considered drinking systems, and therefore the EPA has approved polluting streams and rivers in the US."

[-]

BBQandBitcoin@reddit

Fun Fact: There’s an abundance of aquatic life (many that are edible) and agriculture connected to said watersheds that are affected by these chemicals.

Companies, in general, should be placed on very strict regulations and guidelines on what they put into our air and water (potable or not) it’s all connected.

So pipe down there, “Mr. Fun Fact”

[-]

SeigneurMoutonDeux@reddit

So pipe down there, “Mr. Fun Fact”

I was supporting your post by adding that the water doesn't need to be potable according to law.

Do you just have your jimmies rustled or do you need to lower the aggressiveness setting on your bot?

[-]

melympia@reddit

He doesn't seem to be criticizing the person he replied to, but the regulation itself. Which... seems to be pretty much your point, too.

So, why are you attacking him?

[-]

ActualModerateHusker@reddit

Calling BS. They use too much water to use tankers and possibly too much to significantly treat. It's like 10% of the water in the city of des moines. It wouldn't be economical at the least to transport via trucks.

[-]

Nanyea@reddit

A lot of the bigger ones have their own on-site treatment plants, then dump it back into the watershed

[-]

BBQandBitcoin@reddit

I’m inside. So I know, witnessed it.

3 tankers 3-4 trips a day… you the numbers.

[-]

Aoushaa@reddit

Source?

[-]

BBQandBitcoin@reddit

I got inside details. I don’t know exactly how much water these data centers are putting out, but state & federal authorities are supposed to monitor operations. A contract to keep them at bay would be enough.. but yeah 3x tankers at 5,000 gallons, 3 trips a day…

Isn’t BS..

[-]

RagingNoper@reddit

I work in an industry where small plants regularly dispose of that much water or more daily, and we have quite a few plants. It's really not a lot when you're talking at an industrial scale.

[-]

BBQandBitcoin@reddit

I can agree. Don’t know how far along this particular project is at or what scaling, but witnessed it

[-]

Beardth_Degree@reddit

This isn’t true of all DCs, I know for a fact the larger ones have water treatment plants on site and return water cleaner than they receive it, often becoming a water supplier for their area.

[-]

ActualModerateHusker@reddit

Lol why not just re use the water then? All I know is from context clues the water being used in Iowa isn't getting re used by anyone.

[-]

Beardth_Degree@reddit

Mineral buildup in the water from evaporation of the water in the coolers needs to be diluted back down. It’s easier/cheaper to purify the water with higher mineral content than remove the excess minerals that would cause issues with the cooling equipment. What’s safe for humans and potable water isn’t as suitable for cooling systems.

[-]

socialmedia-username@reddit

Potable water is generally toxic to aquatic organisms as well due to heavy metals.

[-]

MassholeLiberal56@reddit

Ah, yet another convenient externality being swept under the rug for the taxpayer to foot the bill.

[-]

Beardth_Degree@reddit

Please explain the cost on the taxpayer for this?

[-]

MassholeLiberal56@reddit

Really? So they use up gobs of the local water — always with a sweetheart discount btw — and then dump it back into the system for the taxpayer to pay for? What’s not to understand?

[-]

Beardth_Degree@reddit

It’s not “dumped” back into the system, the water is then offered as a source after it has been cleaned, to a state better than it arrived, and there is so much surplus that they then also have to pipe it to local bodies of water.

There’s a very real expense involved and the “sweetheart” deal is offered to anyone consuming in bulk, it’s not on the tax payers as the taxes paid by these companies is far higher than anything else that would occupy the same land. You should look at the local communities surrounding these areas before and after they show up and then talk about how it’s in the taxpayers back.

[-]

MassholeLiberal56@reddit

Any bulk use of water by industry should include the external costs to return the water to a pure state. Otherwise that venture should not exist.

[-]

HomoExtinctisus@reddit

Your comments don't actually convey a lot of meaning even if they are 100% honest. They leave so much to question that I wonder about the motivation for making these type of assurances.

Mineral buildup in the water from evaporation of the water in the coolers needs to be diluted back down. It’s easier/cheaper to purify the water with higher mineral content than remove the excess minerals that would cause issues with the cooling equipment. What’s safe for humans and potable water isn’t as suitable for cooling systems.

This implies the released water does have some type of additional material(s) in it. What are they and how much? If it's so clean, why isn't it potable?

Also from your previous comment, what is "cleaner than they receive it"? As in they distill it, add some PFAS types EPA and local government don't test for, run it through the system where it picks up more contaminates not tested for, then the water is released back into the local environment "cleaner than when received"?

[-]

-UltraAverageJoe-@reddit

The water is cleaned by our tax dollars! (Hopefully cleaned).

[-]

MonkeeFrog@reddit

They can't clean the forever chemicals

[-]

HomoExtinctisus@reddit

You can clean/destroy PFAS type chemicals in this type of situation.

[-]

Suspicious-Tip-8199@reddit

It is no cleaned

[-]

Playful_Possible_379@reddit

You voted for this

[-]

Careless_Acadia2420@reddit

At least petro chemical plants produced something useful.

[-]

Difficult_Source8136@reddit

If you really believed that you wouldn't be using the internet right now

[-]

Careless_Acadia2420@reddit

Ai is not the internet.

[-]

MothashipQ@reddit

[-]

Difficult_Source8136@reddit

Nah this isn't that. This is more like taking a puff of a cigarette and complaining that its bad for you.

[-]

MovinOnUp2TheMoon@reddit

Lots of heavy metals and other toxic crap.

Can you explain this in more depth, or provide a source?

I am curious, no disrespect, but what? and why?

thanks

[-]

-UltraAverageJoe-@reddit

The pipes and cooling systems used for this are metal, usually stainless steel. The components they cool have all kinds of heavy metals in them that leach into the water over time. There’s likely some PEL (permissible exposure limit) for it but over long periods it adds up and depending on the state the center is in the PEL may or may not be as low as it probably should be.

If they use coolant that’s another source of “toxic crap”. It should be cleaned out of the supply before it is drained to sewage but it’s a risk that goes up exponentially with the size of the cooling system — more coolant, more points of failure/leakage.

[-]

keyboardwari0r69@reddit

Bro this sounds like bullshit.

Guess what else is made of stainless steel? Virtually everything designed to process food and drinks. Specifically because stainless is considered food safe and inert.

[-]

-UltraAverageJoe-@reddit

There are different grades of stainless.
I didn’t say the stainless was the problem.

[-]

MovinOnUp2TheMoon@reddit

Thank you for helping me learn more about this!

[-]

Jetfire911@reddit

Also, it's 30% glycol.

[-]

Pretend-Policy832@reddit

Ugh, fuck the data centers. This sounds like an impending disaster. We need are water. Humans are idiots and have no long term thinking skills.

[-]

iridescent-shimmer@reddit

Some use water, some don't. I wouldn't recommend drinking the vegetable oil solutions either though 😂

[-]

RunMysterious6380@reddit

Yep, and it contaminates the ground water when they release it back into the environment.

[-]

QHCprints@reddit

Like drinking pure, concentrated cancer!

[-]

LetsDOOT_THIS@reddit

Isn't the water in a closed loop and thus only requiring a finite amount upon initial construction?

[-]

iridescent-shimmer@reddit

Coolant degrades over time. But, they use all kinds of proprietary mixtures now, it's not even just water sometimes.

[-]

LetsDOOT_THIS@reddit

oh yeah that'd do it

[-]

Aurora1717@reddit

Where did you see that? I live in Iowa and we've been reducing water usage and because of the high nitrate levels in the rivers where our drinking water is pulled from.

[-]

TobleroneThirdLeg@reddit

Why does potable water need to be used for cooling? Can’t filtered non-potable water work?

[-]

ActualModerateHusker@reddit

Likely but it would cost money to filter it. Remember in flint Michigan the car plant got the good water because the bad stuff would also hurt their equipment.

[-]

TobleroneThirdLeg@reddit

Couldn’t they have made slightly more expensive but much more resilient of a machine? Oh wait. I see where I went wrong here

[-]

Robertsipad@reddit

More likely they need something similar to distilled water, like you would put in your car’s radiator. Otherwise you risk clogging or corroding the equipment.

[-]

flyingace1234@reddit

And here I thought that one Family Guy episode where the Walmart stand-in caused power outages was hyperbole.

[-]

QHCprints@reddit

I wouldn't call it "drinking" water just yet... but maybe after a few boils and through filtration. Has to be some nasty shit inside those cooling systems.

[-]

herbmaster47@reddit

I'll be damned. A common sense law in Texas?

[-]

QHCprints@reddit

I 100% see something happening like they power down the colo that has the power company's systems on it and cause a blackout they can't bootstrap from easily. Anyone who says this couldn't happen has never worked in IT.

[-]

MrPatch@reddit

If a power company is single location with no failover I'd be surprised, it seems like the kind of thing that'd be regulated for core infrastructure.

[-]

QHCprints@reddit

Regulation in Texas? That’s not how they roll.

[-]

throwAwayWd73@reddit

That's exactly why they have their own interconnection and don't transfer appreciable amounts of power to the other ones so they can remain independent. Which prevents them from having Federal oversight like the Eastern and western interconnection are subject to

[-]

throwAwayWd73@reddit

In theory, there are redundancies.

I've also seen some shit in my time as a transmission operator. There are some things that they found out at the wrong time were a single point of failure. For instance, when you have a primary and backup and one of them has failed and you haven't replaced it yet when the other one ends up failing.

Iet me link a NERC lessons learned

https://www.nerc.com/pa/rrm/ea/Lessons%20Learned%20Document%20Library/LL20250301_Loss_of_SCADA_EMS_Monitoring_Control_GPS_Clock_Failure.pdf

The above is loss of control and monitoring abilities for that affected company.

[-]

iffywizard2@reddit

This guy does the IT. Need a shotgun next to the dot matrix in case it makes a noise.....

[-]

Protahgonist@reddit

This guy works in Texas IT

[-]

QHCprints@reddit

Shhhh... I think the toaster laughed at your joke.

[-]

iffywizard2@reddit

Boom

[-]

kingofthesofas@reddit

These are big cloud data centers like AWS, AZURE, GCP etc. Likely if those apps are in the cloud and designed right they have regional redundancy. Also the data centers wouldn't power down they would just switch to the on-site generators and burn a fuck ton of diesel fuel and keep running (maybe turning off some stuff that can be shifted to other regions).

[-]

BBQandBitcoin@reddit

Lol 😂 facts

Hopefully TCEQ comes down hard 🦬🪶

[-]

Timmy98789@reddit

The grid has to still be up and stable for this to even matter.

[-]

herbmaster47@reddit

Well yeah, but if it wasn't it wouldn't matter anyway.

I'll give credit where credits due, even if it works out all fucked up.

[-]

Timmy98789@reddit

It's Texas, more likely to be lip service and false comforts.

[-]

GuiltyYams@reddit

It does seem so:

The law’s intent is “to make sure [large loads] pose as little reliability risk to the system as possible and [are] not drinking the milkshake of all other Texas power customers,” NRG Vice President of Regulatory Affairs Travis Kavulla said in an interview.

[-]

pathf1nder00@reddit

LOL...bet Google and Amazon hate that!

[-]

OutrageousOcelot13@reddit

Good law.

But please people, understand the datacenters won't actually power off during an event like this. Any datacenter worth building has multiple, redundant diesel generators and contracts with fuel companies to maintain a supply at all times.

Hospitals, 911 call centers, airports- all of these places have servers in datacenters. They don't just turn off. They'll switch to alternate fuel for the duration of the utility power outage.

[-]

Bob4Not@reddit (OP)

lol pardon the misspelling in the title. I shared this because the risk to consider is if you use any devices or infrastructure that could depend on cloud servers. This raises the likelyhood of internet resources going offline in a peak grid usage scenario.

There have been stories about how Smart Thermostats and Smart Locks stopped working when their cloud services went offline.

Cloud services should never be isolated to one state, I don’t expect a brownout to affect any of our critical preps, but I wanted to raise the issue.

[-]

QHCprints@reddit

Yea, the people cheering this on as good have no clue how interconnected things are. Take down the wrong data center unexpectedly and any number of "very bad things" could happen. They'll be grabbing the pitchforks when they can't get admitted to a hospital or pharmacies can't fill prescriptions.

[-]

PurpleCableNetworker@reddit

That means it’s in the data centers to have their act together to prep for this kind of scenario. If a provider can’t handle a basic power outage they shouldn’t be a cloud provider and should go out of business.

[-]

MrPatch@reddit

It's not just on the DC to have their shit together, they should absolutely have planned this scenario and have appropriate processes in place to manage of course but anything critical that is co-located into the DC in question also needs their own continuity strategy, some presence in a second DC where they can failover to.

If it's one of the big cloud providers though they'll have multiple geographically separate redundant physical DCs in an availability zone that are effectively capable of seamlessly running everything in case of the loss of an entire DC and then you can very easily build your applications to run multi-AZ for further redundancy and if you're a critical infrastructure you'll absolutely be expected to be running in multiple geographically diverse regions for this exact kind of thing.

We're in Dublin, London and Frankfurt for our cloud based LOB apps, the stuff in our own DCs are geographically separated and everything running there should come up within 4 - 24 hours of a catastrophic loss of any one DC.

The days of 'the server/data centre is offline!' taking down a whole system or organisation is well in the past for all but the tinnyest of tinpot organisations.

[-]

QHCprints@reddit

Calling it a “basic power outage” seems pretty dismissive. You and I both know there are a lot of calculations needed before making broad claims. We also both know that an incredibly large number of companies have poorly tested disaster continuity plans and that’s putting it nicely. I’m glad things are perfect in your ivory tower but after 20 years consulting I’ve seen enough train wrecks that wouldn’t survive a massive blackout.

[-]

PurpleCableNetworker@reddit

Well, a power outage is a power outage. It doesn’t matter if it’s caused by a drunk driver or power getting shut off because the grid is unstable.

A data center should be able to operate for an extended period of time by itself (as long as the network connections stay up that is). If the data center can’t then it’s being done wrong. You and I both know that.

I’m not saying data centers do things right. Being in IT nearly 20 years I know that “doing things right” is a rarity - but my point still stands: If data centers can’t handle power outages - regardless of cause - they shouldn’t be around. Power is a pretty simple thing when it comes to large systems: either you can use it or you can’t (understanding you can have various issues with power delivery, not just black outs, hence the wording if my response).

Honestly I feel bad for the consultants that get called into those messes. Cause if a mess didn’t exist then you wouldn’t have a steady pay check. Lol.

[-]

QHCprints@reddit

I didn’t mean the cause of the outage but rather the duration and expectations while on secondary power.

Disaster recovery and continuity are only as good as how recently the plan was tested. I’ve found very few companies that do full, regular tests. They’re out there for sure, but most are more in the “looks good on paper” category.

There’s just a lot of dominoes interconnected that can have a cascade effect. Healthcare tends to have a lot of external dependencies in their applications that aren’t apparent until it’s an issue. Yes, that is 100% on that healthcare systems IT staff but that doesn’t help the patients that can’t get prescriptions.

I’m just not hopeful but that’s par for the course.

Story time coming to your inbox.

[-]

PurpleCableNetworker@reddit

Ah - gotcha. The expectations while on secondary power can indeed be - well - “interesting”. 🤣

Thanks for the DM. I’ll reply shortly.

[-]

kingofthesofas@reddit

Tagging onto this post they likely will not shut down the data center. Those data centers all have big generators that can keep the data center running for days if not weeks on diesel fuel. They may shift load over to other regions but the odds of this making cloud services go down is very low. The air quality near the data centers might suck though.

This is actually the intent of the bill because data centers have their own generators in the event of a power shortage they could keep opperating on their own generators and stop or reduce power draw from the grid. There is very little chance this results in an outage of anything, it probably actually increases grid resilience because the power gets built out to support the data centers and then they can turn it off if they need it during an incident.

[-]

PurpleCableNetworker@reddit

IT guy of ~20 years here. I’m glad to see this bill. Any data center not prepped to handle a power outage properly shouldn’t exist. Power issues are notorious for causing issues with systems, thus extra care needs to be taken when designing data centers. Any of the basic management and security courses drill it into your head that backup power capable of running everything at full load, including cooling, is a must.

Even in my very small data center we have 2 generators - one of them piped direct into natural gas. Battery back up to handle the load during cutover and twin AC’s that are in a lag/lead configuration. A generator, battery backup, and lag/lead ac’s are bare minimum for any real data center.

[-]

AnomalyNexus@reddit

Isn't this just normal?

The list of things you absolutely can't legally disconnect is usually quite short - hospitals, high speed rail etc.

Beyond that regulators usually have a mandate to do whatever is needed the prevent a cascading grid collapse

[-]

WhiskyTequilaFinance@reddit

That's surprisingly smart and reasonable. ...what am I missing?

[-]

Hot-Profession4091@reddit

Texas is still not going to properly invest in its grid.

[-]

CowSightings@reddit

Datacenters commonly have embedded liquid fuel generators to keep them running in incidents such as this. Sorry to say - but cutting the mains from them will likely only increase the environmental impact with less efficient off grid power. It will of course make them more expensive to operate but since we subsidize fuel so much it won’t likely deter any of the coming buildup.

[-]