Moving from Data Center to CEO Basement

[-]

InspectorGadget76@reddit

A big one missing from this list is security. How are you going to control physical access? Security systems? Video camera? Access control systems? Security guard? Environmental monitoring? Temp, humidity, leak detection?

Reply

[-]

Alex_2259@reddit

Ofc they won't consider any of that, but if the project has an infinite budget you can probably, and I mean probably, build a professional, small corporate data center in a residential setting. At least in the USA, you can do that! You can probably pay ISPs enough to horizontal drill to some suburban house, a ridiculous amount to get your dedicated circuits. You're paying the fuck you numbers, because they'll get 0 ROI so you're covering it yourself. You can probably pay off your power company enough to give your home enough power to do that too! Ofc they all have to pull permits, which could stall it. Not a problem, if you got the money. You can lobby local and state politicians to simply change the laws to allow what you're doing. If that fails, go to national politicians. If that fails, use your infinite budget to hire enough international mercenaries to stage a coup, or ignite a civil war via hyper inflation and mass disinformation. Just so you can build a data center in 173 Incompetence Lane, fucking Shittown Iowa. Alternative to that, you could use a co-location facility for a few thousand a month, that provides everything you need at scale already. Or a cloud provider, depending on what you're trying to do. But WTF is the fun in that?

Reply

[-]

FinancialBottle3045@reddit

Fire alarms. And suppression.

Reply

[-]

garaks_tailor@reddit

Please. Please keep us updated. This is one of the 3 dumbest IT stories I have ever heard

Reply

[-]

Alex_2259@reddit

This is up there, completely insane. It's fun to think about in a parallel universe where it's somehow magically a serious project, where they want all the bells and whistles of an enterprise smaller data center environment in a residential location, and are willing to arrogantly pay for it despite literally every possible viable alternative being cheaper by hundreds of thousands, if not several mil. All US perspective here, countries vary but I imagine similar issues. I mean, you're then basically just building the dream home lab. Bribing two enterprise grade dedicated ISPs to pull permits and drill deep into some suburban, residential area with 0 ROI except your one contract for your path diversity. You could blow hundreds of thousands, if not millions just to pay their "we don't want this" price with a dogshit contract to boot. Unless you're lucky enough to win the Powerball twice. Somehow deal with that residential power service, which very well may involve infinite hoops and bribing your power company a fucking ridiculous amount of money to give your suburban house some serious infrastructure. A serious sine-wave backup generator to add on top. That isn't cheap. Build walls, building security, access control, and all the perhphial shit a data center has that conveniently a co-location facility or even the cloud depending on what you're doing takes care of; alongside everything else here. After that, you're golden. But you didn't yet consider geo redundancy and DR. So find your buddy a few states away, and bust out another 5mil for his house too I suppose. And consider all the problems with that, make sure fixing it is also in the budget.

Reply

[-]

iCapn@reddit

> the ISP needing a circuit that will never go down in a neighborhood I think getting a quote on this would make them change their mind. There's a reason why high-9 availability is expensive. And using a residential connection for hosting servers, especially for a business, is likely a breach of the ISP's TOS.

Reply

[-]

Alex_2259@reddit

This is just such an insane idea I doubt they even give a shit. I know the Reddit answer for everything is to fuck off from whatever it is, but this is so asylum ass shit that's the only real answer here. For essential uses you probably want dedicated fiber. That only is ran to business districts, not residential usually, and that's only a single non redundant ISP we talked about. They have to horizontal drill from wherever their business infra is to a residential house. Unless they're very lucky, the ISP will demand hundreds of thousands with also possibly an unfavorable contract to do that. Many ISPs that do residential offer business services in the same infra. As shit as residential service but you get static IPs and TOS says you can use it for business. Not good enough, but it's cheap. Most sane show use that type of broadband service as just a redundant line. Then there's the electricity. You want redundant over provisioned circuits for a data center. 30A or 20A at the lower end is common for a smaller one. So you have 40-60 Amps dedicated. That's another 5k to impossible to upgrade the home electricity. We're not done. HVAC. Call it 20 Amps, dedicated. Redundant call it 40 amps. Could be up to 80 amps. Homes have 100-200 amps of power. Put up walls in this basement to enclose it too, another few thousand. Also the owner's home becomes a corporate site, so IT may need to invade their fucking house on occasion to do all sorts of activities. Sure in a tech startup maybe you have humble starts, but the owner isn't also the technician so this isn't that. And it's a reduction. That's fucking weird as hell. I would be sending CYAs in writing to my manager and stakeholders if I am given written authorization to send this to them on this, and executing it when all my CYAs are overruled to spec, while I apply for jobs in the background. Not because the company sucks, even if it does, but if they're cutting corners this hard I would be worried about inevitably insolvency.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

How can i find this out? It would end the entire project easily!

Reply

[-]

chibihost@reddit

Find out what ISP(s) service his area and contact them for quotes on business class symmetrical options (I'm out of the game a bit but 250mb up and down, 500mb up and down would have been my starting point ages ago). Ask about fiber options, where the last mile runs and what it takes to get your own line run. If he's close to a main fiber run with no last mile this could easily go into $20k+ just for an install. Many residential areas are only serviced by one ISP, but if there are options repeat for each. If there is only one, you have a single point of failure for the eventual time when a utility cuts the fiber line on accident.

Reply

[-]

samspock@reddit

Fiber seeking backhoes are everywhere!

Reply

[-]

fresh-dork@reddit

that's why my daypack always has a foot of fiber in it. in case i get lost

Reply

[-]

FenixSoars@reddit

North American species are the fjnest

Reply

[-]

ohyeahwell@reddit

Comcast has wanted $50 - 80k for last mile at a few of our sites. One site they only wanted $4k so we went for it.

Reply

[-]

tankerkiller125real@reddit

Get a quote for enterprise DIA to the home. I promise you're looking at around 10-20K on install alone IF the CEO lives near one of one of the distribution points, and then very expensive monthly cost. If the CEO doesn't live near a distribution point those.costs can go way, way up.

Reply

[-]

5SpeedFun@reddit

I actually did this at one time. 6 years at $500/months with cancellation fee being remainder after amount owed for 50/50 dia. They needed that to break even on the service and $30k fiber build out.

Reply

[-]

Frothyleet@reddit

Yep. I'm guessing CEO lives in a suburban McMansion, which will be nowhere near an ISP point of presence, so DIA building out could easily be hundreds of thousands of dollars, if they can even find an ISP willing to do it. Two ISPs, really, because you need redudancy that doesn't run on the same fiber as the other provider :)

Reply

[-]

1n5aN1aC@reddit

Technology-wise, you need a connection WITH a dedicated External IP, WITHOUT CG-NAT, and need it to be fast and reliable. Legality-wise, pretty much all home internet contracts have clauses that say you cannot use it for server-hosting or business purposes. So you'll at a minimum probably have to upgrade to some kind of business plan.

Reply

[-]

Mailstorm@reddit

Ask Mr ceo man for his agreement from the isp. And if they come back and say he'll just get business internet, look at that and see if there's anything that would prevent you from moving

Reply

[-]

BabyLinuxAdmin@reddit (OP)

Awesome thank you sir!

Reply

[-]

garaks_tailor@reddit

Find out who your current isp is. Contact your account manager at the isp. Explain the project to them and the new address. They should be able to give you an initial estimate pretty quickly

Reply

[-]

Angy_Fox13@reddit

Depends where this is. I could get 2.5 gig commercial fibre optic right to my house and im not in a place where a super-rich CEO would live. The line is already there all the way to my house. Backup could be commercial Starlink, a 5G connection, or a business DSL line. Most places do not have fibre lines at all though I'm fortunate they upgraded my neighbourhood.

Reply

[-]

SteveSyfuhs@reddit

This is not really something that can be answered here. What is the state of the basement? Is it finished? Is it full height? Is there a dedicated place that can take power, data, cooling, and also not be a nuisance to the people living in the home because hey 20 servers plus cooling are really pretty loud? The electric bill will not only be high, it will likely require an entire new circuit or three added. What about insurance and liability? Security for the information on the machines? Are they in a flood zone? Are they in an area with regular power outages? Do they have a backup generator and will it handle a sudden increase of a full server rack including network and UPS? What happens when one of the dingus' kids asks "hey what does this do?" and yanks out a hard drive? Do you have access to it in the middle of the night when things go bang? What about when they go on vacation?

Reply

[-]

BabyLinuxAdmin@reddit (OP)

In terms of insurance and liability im hearing its best we look at his home insurance but is there anything else? These servers are internal so at least nothing prod facing but will be used for dev

Reply

[-]

17549@reddit

Is this in the US? All the work will need to be approved by city inspector, and possibly fire inspector, and I'm guessing the home will not be zoned for business. Step 1 is to find out if the zoning is permitted. If not, you're being asked to help with something illegal.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

Boom! im trying to find a zoning map now which may help my case even further as to why its not possible. Any great resources you know of?

Reply

[-]

17549@reddit

TBH, I just googled "Zoning Spokane" when I needed to previously. Spokane has a ArcGIS map where you can click on any parcel and get the zone. Might also try searching "Parcel Map {location}" - sometimes there will only be a county one, but that should work too. I think you could probably just call city/county office too. You shouldn't have to give any personal info. You might not even have to be super specific about the address - just say you're curious about if you can run a business out of your garage and if that's allowed and/or what permitting is required.

Reply

[-]

betsys@reddit

Home insurance will absolutely not cover any of this. Need to think about both the raw cost of the hardware AND the cost of interruption of business. And as others have mentioned, zoning.

Reply

[-]

Marathon2021@reddit

> im hearing its best we look at his home insurance Run with that. Play dumb about it too. "Ok, can we get the name of their homeowner's insurance carrier, policy #, agent that they work with - that kind of thing?" Get the info. Then ... call them. "Yes, my boss CEO so-and-so wants to outfit their basement home with some servers for work. I'm calling you to see if you can give me an estimate of the adjustment to their policy costs annually..." etc. etc. What you want to hope to hear from the insurance company is for them to say (politely) "fuck no we're not covering that" and that might just kill this idea in the crib for you. Make sure you have a lot of specific. Inventory all the physical servers, storage, network gear. Go to the manufacturer's tech specs page for each device, find the maximum power rating/draw information. Add it ALL up. You'll probably come to 10-20 kilowatts of sustained draw. Then you need to double that for HVAC. I don't think most residential policies are calculated for such massive sustained power draw.

Reply

[-]

Grrl_geek@reddit

Home insurance may not cover a business loss/claim. What a horribly bad idea this all is. Do you have your resume ready??

Reply

[-]

joetron2030@reddit

It wouldn't surprise me if he expenses some or all of his home power bill to the business, too, because I highly doubt he's going to do the appropriate calculations to separate out business power usage vs. residential. Good way to skim some money out of the company coffers.

Reply

[-]

cspotme2@reddit

Good time to virtualize! How much are you saving monthly by doing this? Assuming the ceo/owner has weighed the risks and cost savings... My be worth it for a lot of saving. Also depends on your company's usage of the env... You should definitely have some type of auto switching generator long with the ups.

Reply

[-]

HankMardukasNY@reddit

Congrats to your company, this is quite possibly the dumbest idea I’ve ever heard on this sub

Reply

[-]

FenixSoars@reddit

Fuck, please record this entire disaster and make a YouTube channel. I’d watch. Holy shit.

Reply

[-]

Immortal_Tuttle@reddit

This! Vlog of this will be shown for centuries. "Basements and dragons: servers tale"

Reply

[-]

436643346565@reddit

And to top it off, the basement is in an area with a high flood possiblity, as always.

Reply

[-]

Princess_Fluffypants@reddit

I hope OP keeps a regular blog so we can follow along on this train wreck.

Reply

[-]

WWGHIAFTC@reddit

Live updates on this one. I'll subscribe.

Reply

[-]

Recalcitrant-wino@reddit

Seconded.

Reply

[-]

Det_23324@reddit

Top 3 for sure.

Reply

[-]

Prestigious_Wall529@reddit

I know of someone who did this. Approx 75 machines on concurrently. The police raided their house thinking it was a grow house.

Reply

[-]

natefrogg1@reddit

Totally different power usage signatures though, that’s a trip

Reply

[-]

SirEDCaLot@reddit

Wow. I'd just lay out some project requirements. For example- updating basement to ensure it won't flood (structural and drainage work), redundant power / generator separate from rest of home, redundant dedicated fiber, redundant cooling, security (have to keep servers secure from family members AND allow access for tech staff in emergency if CEO is on vacation), insurance, etc. I'd then propose a competing plan where you move everything to a couple of giant VM hosts and throw them in a colo. Chances are it'll end up cheaper.

Reply

[-]

No-Term-1979@reddit

-Journeyman electrician Doing some looking at power for servers. It looks like 10kw is about normal for a server. That's 42A per server for a 240v load split phase. It looks like it will take a 3 ton cooling system per server. That's another 50-60A? -20% load capacity for continuous loads means you will need about 120A supply for every server. 4 servers is 480 amps total. The house is going to need 200-400A. That's covering some redundancy and ancillary equipment. But wow is that a lot of power for a residential service. The POCO is going to tell you to pound sand, unless you pay for the upgrades on their side. That's going to be $20-30k on the cheap side.

Reply

[-]

betsys@reddit

Depends on what sort of servers and what you're doing with them. Something like an HP Proliant draws 2-300 Watts, and VMWare for example wants you to run them on high. Still way less power than a fully loaded home gaming PC. For a storage server , depends what kind and how many disks are in there. Network gear doesn't use much. If they're still using tape, that'll add something while in use. 20 servers are going to kick out some heat, so you'll need extra A/C and power, and a properly sized UPS. If there isn't already, it's worth considering whether it's possible to reduce server count - but that's a tiny bit inside an overall insane picture.

Reply

[-]

natefrogg1@reddit

10kw for a server? Dang that’s some serious processing I’m guessing! We’re pulling 10A each from 2 120v circuits, so 20A total, that is for 5 physical servers and network equipment, add another 5A for the crappy air conditioning that actually is good enough as there just isn’t that much heat generated. Anyways I feel like this stuff can vary wildly depending on what’s going on but 10kw for on server sounds like quite a bit

Reply

[-]

Nnyan@reddit

10kw per server? 42A circuit for server?? 3 ton AC per server??? Where are you getting this nonsense from? This is a small 100 person company. Do you think they have a rack full of SR670v2’s???

Reply

[-]

zcubed@reddit

I really hope this is the last gasp of Network Solutions! It all ends in a house fire!

Reply

[-]

Mr-RS182@reddit

OP please give us regularly updates on this train wreck of a project

Reply

[-]

Reinmeika@reddit

Please document this for future reference. I wish we could give you advice, but you’re about to be thrown into a circus, hope you have your clown makeup on hand. You’ve got the gist of it. Other things is air gapping and security. Not just cooling, but do you have redundancy if/when his house loses power? Not just the circuit, but actual power too. Further, 20 servers? Can his house handle that? You’d need at least a generator if it fries his electrical. Security is an obvious concern too. Not just VPN - actual, physical security. Anyone gets word this dumbass has 20 servers worth of data and it’s happy hunting at that point. Godspeed friend, dust off that resume.

Reply

[-]

Dctootall@reddit

Well if he insists, something’s I would put on the requirement list: (hopefully maybe it will give him pause when he starts to see the issues) 1. Primary high speed network connection, And redundant circuit using a 2nd provider to ensure no single point of failure. (Ie…. Telco isp and cable co ISP for example…. BUT…. Make sure it’s a true business class service with a quality SLA. ) 2. Networking equipment that can handle failover and WAN vpn connectivity. 3. Cooling to ensure servers don’t overheat. 4. Physical racks for the servers. 5. Commercial backup generators/UPS systems to ensure “clean” uninterrupted power even if residential power is impacted. 6. Means for IT to access equipment 24/7 without needing to coordinate with the CEO to allow for maintenance and outage resolution (house key? Alarm codes? Etc) 7. Commercial insurance for the hardware. That much hardware is not going to be covered by standard homeowners insurance. It may even cause issues with his current coverage adding the equiptment into his residence, so he needs to make sure that he doesn’t void his coverage putting those servers in. 8. Additional monitoring, including environmental, To allow for remote health checks of your new data center. 9. A plan for “offsite backups” to ensure data isn’t lost in case of disaster. Those are just a few things I can think of off the top of my head. May also want to throw in a new CEO or IT Manager as well as I feel one or both won’t survive long if this plan is enacted, so it’d be a good idea to include the costs of filling those positions into the budget and planning.

Reply

[-]

par_texx@reddit

You forgot power feed upgrades. a 100 or 200 AMP main breaker isn't going to cut it anymore. So he'll have to upgrade his feed to 400 AMP.

Reply

[-]

Dctootall@reddit

Good point. Honestly, an isolated power scheme would be better so that billing can be separated, as well as to prevent a hair dryer or microwave inducing dirty power or a fridge causing a circuit breaker to trip knowing out power in the new “data center”

Reply

[-]

Marathon2021@reddit

Agreed, doing this in some sort of "tiny house" / shed type of thing ... probably make the most sense out of this least sensible plan overall. The 24x7x365 HVAC noise alone to offset the heat shedding off of 2 racks of servers ... would drive anyone nuts.

Reply

[-]

KadahCoba@reddit

At this point you've redesigned the mini co-loc shipping container. Might be cheaper to just use an actual co-loc instead. :V

Reply

[-]

fresh-dork@reddit

'tiny bunker'. like a large shed with unusually thick concrete walls and a security door :)

Reply

[-]

Dctootall@reddit

Don’t joke…. I’ve been to cable headends which were nothing more than a Home Depot shed packed full of equipment. One that comes to mind consisted of a gate at the bottom of the dirt road going up the mountain, another fence with locks around the shed compound (which included the generator and some dishes), and then a padlock on the door to the shed. Admittedly, there was only about 1 or 2 older servers in there along with a bunch of hardened rf equipment, But the concept is the same

Reply

[-]

schlock_@reddit

Arg, you beat me to it. What happens when someone turns on the dryer! LOL

Reply

[-]

garaks_tailor@reddit

If their UPS is like ours they'll need elevator access just to install it.

Reply

[-]

WMDeception@reddit

This man datacenters, I'd add a generator to the mix in case the uptime is mission critical enough to justify. Also, redundant cooling, not just one single point. This is not a good idea. Better to just get into an actual DC with cooling and power all sorted out for you.

Reply

[-]

betsys@reddit

I'd be willing to bet that by the time this is all priced out, a rack in a data center will be CHEAPER.

Reply

[-]

ConfidentAmbition601@reddit

There may be legal issues to be addressed too. Check the local zoning codes. Running a commercial data center in the basement of a residential building may cause problems with zoning.

Reply

[-]

sjclynn@reddit

It will need a business license as well. The application will ask some questions that will be difficult to answer truthfully. There are reasons that data centers go into buildings in commercial zones.

Reply

[-]

BlkCrowe@reddit

This is a great response. Op already knows this is stupid, so your list is probably exactly what he/she was looking for. I would add: #4-racks secured to wall or floor with adequate front and rear locks that only IT has access to. #6-draw up an sla/ola so there’s no misunderstanding when that time comes #8-monitoring should include security camera pointing at rack with remote viewing plus offsite storage of footage. #9-absolutely 3-2-1 backups Get ALL this in writing. Good luck!

Reply

[-]

Grrl_geek@reddit

Security, both physical and how to secure your data.

Reply

[-]

alpha417@reddit

Mr Wiggles, the diabetic 13yo Bichon Frisé, will let you know when someone is there.

Reply

[-]

TheFriendshipMachine@reddit

This whole thread was getting a good chuckle out of me but this comment has me laughing hard. It just perfectly encapsulates the absurdity of the situation poor OP is in.

Reply

[-]

sjclynn@reddit

#6 should be a separate outside entrance. If I were an admin, I am NOT traipsing through the CEO's laundry room at three in the morning.

Reply

[-]

AnomalyNexus@reddit

> A plan for “offsite backups” to ensure data isn’t lost in case of disaster. At CEO's holiday home of course!

Reply

[-]

cookerz30@reddit

#6 is probably the biggest hurdle, considering their HOA would be pretty upset about all this.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

I don't personally own a home yet so i will def have to do research on this. Likely have to get in contact with his agent

Reply

[-]

silentstorm2008@reddit

That sounds a like a risk management job to research...not a sysadmin

Reply

[-]

garaks_tailor@reddit

The CEO is going to be so fucking pissed once they realize how many power lines and cables are going to need to be run

Reply

[-]

Xenolog1@reddit

In case of disaster - how long can the company survive without the data center? An off-site backup prevents data loss and is of course essential. But there should be also a plan how and how fast you can resume operations in a new data center, no matter how improvised it may be until a proper data center can be built to replace the destroyed one. Ideally, the procedures to bring up the emergency data center online should be tested regularly.

Reply

[-]

trisanachandler@reddit

Don't forget about possible flooding.

Reply

[-]

reedacus25@reddit

My first thought as well. Not only because I live in a flood prone region, but also water is beholden to gravity, so if the toilet or bathtub or sink, et al, overflows, pipe bursts, you name it.

Reply

[-]

llDemonll@reddit

#1 should specify two separate physical connections taking different pathways out of the house, not just two Comcast circuits or something similar.

Reply

[-]

PlsChgMe@reddit

Well that seals the deal, OP, I'd first ask the CEO if the HOA approves of this. That will be the end of it, and NO ONE can beat the HOA, so they'll be forced to rethink the idea.

Reply

[-]

bitslammer@reddit

> so he needs to make sure that he doesn’t void his coverage putting those servers in. Oh it will, at least as far as his homeowners goes. If it doesn't his premium is in for a fat hike.

Reply

[-]

garaks_tailor@reddit

Commercial fire suppression too. I know our hospitals insurance and cyber insurance both sent inspectors out to verify our server rooms fire suppression

Reply

[-]

PowerShellGenius@reddit

Make sure they are zoned for business, and will install proper fire suppression for a server room and get it inspected. Odds are, even if it is zoned for business, the costs of doing this legally will change their mind. If it's not zoned for business, it's flat-out not an option.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

So i know they will ask for example... "But what about people who run servers from their home.. how are they allowed to but we can't" Any good articles i can read to better understand this?

Reply

[-]

Major-Astronomer7529@reddit

This would be questions for the city/county/township or the CEO's home. Considerations: 1. Local zoning laws 2. HOA rules 3. Building codes and safety regulations (to do the build out will require permits) 4. Noise & Heat 5. ISP restrictions and SLAs 6. Power Availability & Consumption

Reply

[-]

betsys@reddit

This is madness. I hope you are job-hunting! If you decide to stay and participate in this, be sure you leave a trail in writing of all of the warnings so when this fails you will be protected, and save timestamped copies in an offsite location. They should also price scaling down to one datacenter with cloud backup, but as others note there is a LOT wrong with this picture, possibly something shady. If the business is large enough to have a CFO, you may have an ally there. There are a lot of potential business costs. For starters, can this arrangement be insured? Not just for the material value of the hardware, but for interruptions? And what happens if the owner passes away? The value of the business is tied to the ownership of their house. Any chance of getting the CEO to go for an external opinion on this plan? If the IT manager is the one managing the project, and you're junior, be VERY careful with how you phrase things. Stick to raw facts and don't summarize. (for example, 'the total wattage of the servers is X" not "we need size X power supply for the servers" - so when they forget to include the power draw of the A/C units, they can't come back and say 'Junior said X would be sufficient". And as mentioned, save copies above. The odds are very low that either you or the IT manager will be working at this company when this is over. PS if the IT manager is a blood relative/college buddy of the CEO - just drop everything and run.

Reply

[-]

korazy@reddit

I love this because it's not my problem. That's awesome.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

You're horrible! LMFAOOO

Reply

[-]

Princess_Fluffypants@reddit

Please please PLEASE keep a regular diary/blog going about this. We all want to follow along to watch this slow moving trainwreck.

Reply

[-]

5SpeedFun@reddit

Stardate 20241220.0800 got a call from multiple users that they couldn’t do Q4 accounting work as vpn/servers were down. CEO’s Kid plugged in the Christmas tree lights again and took down the data center.

Reply

[-]

Princess_Fluffypants@reddit

I have, in complete seriousness, taken a "building outage" ticket because a cleaning lady plugged a vacuum cleaner into a UPS. And this was in a proper commercial building. The logs of how quickly OP's company will fall apart would be a tale for the ages.

Reply

[-]

WhiskyTequilaFinance@reddit

Please! I want a subscription to this!! This guy thinks servers work like big laptops. He's in for an ugly surprise.

Reply

[-]

MrHaxx1@reddit

For the love of God, please keep us updated

Reply

[-]

PlsChgMe@reddit

We were all thinking it though.

Reply

[-]

korazy@reddit

On a more professional response. Your IT manager will have communicated the pros and cons of this change on day to day operations and disaster recovery. They decided the pros checked the boxes that make it worth doing. I don't know your business but 20 servers in 4 data centres for 100 people sounds like decisions were made excessively over scaling IT at a huge cost and now they are excessively under scaling.

Reply

[-]

TheLostITGuy@reddit

GOAT

Reply

[-]

Mike312@reddit

It's not *our* problem....until we find out OPs company is one we use for data services.

Reply

[-]

jsellens@reddit

Consolidating servers from 4 locations to 1 can make sense. But consider that 20 servers can often fit in one rack in a colo. How much would that cost? Maybe $1,000? Bandwith will likely be more expensive to a home, and less reliable. So this company of 100 people is going to all this trouble and expense to create less reliable infrastructure to save maybe $1,000/month. (Ignoring all the additional capital and operating costs for a home data centre.) Either the company is run by idiots, or it's circling the drain. Or both.

Reply

[-]

Angy_Fox13@reddit

I priced one rack recently for a colocation it was between 3K and 5K with internet access. This was for a 1 rack cage with 24/7 access in a good data centre in the Toronto area. This was with 2 x 30 amp circuits.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

This is the point i made but its as if im being ignored. There are way too many benefits for keeping all this hardware in one data center where some of the servers already are. Even down sizing to a fraction of the servers would do it. All in all the issue is lack of technical leadership

Reply

[-]

jsellens@reddit

I would argue that it's not a lack of \*technical\* leadership - it's that the leadership has no common sense - can't even compare the costs of two alternatives, and won't listen to people who have useful information and experience. That's just dumb. I learned "more than a few years ago" that there is dumb stuff everywhere. You just get to choose the dumb stuff you're willing to put up with.

Reply

[-]

ThinkPaddie@reddit

So he can switch things off and on at 3am. It sounds like a win-win. What does the wife look like...

Reply

[-]

ohyeahwell@reddit

She looks like the CCP

Reply

[-]

ThinkPaddie@reddit

Well she can be the chief electronics operator now.

Reply

[-]

Sinister_Crayon@reddit

Yeah... OK a few things jump immediately to mind... I mean other than the thought that this is ALL a terrible idea; 1. Liability. Who is liable when it goes down? 2. Insurance. Homeowners insurance might actually cancel the CEO's insurance for having a legitimate business asset in the basement. Zoning laws might also prevent this. 3. Business class Internet is most likely not available in his neighbourhood unless he lives close to actual businesses. Even then it might cost a significant amount for buildout. 4. Related to 3; the T&C of a residential Internet connection do not allow business use. If you ARE using it for business then they can shut him off at any time. 5. Is there going to be a facility buildout for the servers? Servers can be very sensitive to ingestion of materials which is why most datacenters are positive pressure environments to minimize the ingress of foreign contaminants. Meaning; are they planning to build a room in this basement to store the servers in that will have A/C, lighting and so on? 6. Emergency facilities; if a fire breaks out in the servers what mitigation system is in place to extinguish it? Related to above, zoning laws might prohibit installation of a proper fire suppression system in a basement. 7. Access to the datacenter facility for staff? 8. Security of the facility? 9. Is the power infrastructure in the neighbourhood up to the task of running a datacenter? Estimate total power draw based on all servers running 100% loads and then add 25% to that number. That's the total wattage draw that the power feed into the house needs to support. If you put too high a load on your electric circuit that it's not designed for and cause damage to lines and equipment then the power company can bill you for those repairs and/or upgrades. 10. Emergency power? UPS/Generator? 11. Are you going to drop 220 into the site for more efficient electric use? How much amperage do you expect to need? God, these just keep coming out of my head. Yeah, this is a terrible idea and I guarantee you it's not being thought through at all.

Reply

[-]

AnomalyNexus@reddit

He just wants to score a sweet halon system for his kitchen smoke alarm.

Reply

[-]

A_Unique_User68801@reddit

Paging /r/ShittySysadmin Paging /r/ShittySysadmin

Reply

[-]

Sad-Garage-2642@reddit

I thought I was already there while reading this

Reply

[-]

pwoar90@reddit

Im from australia, but hope the ceo’s home isnt just classified as a residential zone. Retrofitting a residential basement would certainly be a challenge. Some additional things to think about. - How will backbone cables be run between racks - Power redundancy for facility - Running separate circuits/busbars for racks/rows. - Physical security - Enviromental monitoring hardware and DCIM

Reply

[-]

fresh-dork@reddit

20 servers, guess at 600w/server + 50% for cooling. maxed out 200A service to run servers and AC with no headroom, 1000sf for servers and AC and batteries. i assume 10gb or better redundant service? wow, this is an impressive level of fail

Reply

[-]

PMmeyourITspend@reddit

This sounds like a poorly designed tax avoidance scheme for the owner. Ultimately- set it up with the best power, networking and AC you can get at the address and send at least 8500 emails covering your ass saying its a bad idea.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

Yep thats the plan and exactly what i told my manager but he told me not to say these kind of things because it sounds bad...

Reply

[-]

ZippySLC@reddit

I'd spend some time and figure out the power draw and the cooling costs of the gear going in there. Then then look up the electricity rates for the area where their house is and say "This is a best guess but your electric bill is likely to up by $X/mo." If the whole goal is to save money and you can show them numbers that they haven't even considered yet it may get them to reconsider. Of course if they pay their electric with a credit card they'd be racking up some major points/miles on it.

Reply

[-]

Marathon2021@reddit

> he told me not to say these kind of things because it sounds bad Then this sounds like the optimum occasion for using some of the "Diplomacy is the art of *letting someone else have your way*" or "Diplomacy is the art of telling people to go to hell in such a way that they ask for directions" kinds of quotes. You might not be able to take on the idea head-on with them. So don't try. Instead, start laying out all of the ridiculous specs required - specifically with things that will make it seem unpalatable. For example, figure out how much cooling that much equipment will need (take this as a learning opportunity) and then don't just say "we'll need more HVAC" say something like "the HVAC required for that would be about $XX,XXX installed, and runs at XX decibels 24x7x365" - or, start inquiring about what might be the best way to start working with the CEO's homeowners policy to build an addendum for this additional (high-powered) equipment because wouldn't the CEO want that? Their insurance company certainly would want to know...

Reply

[-]

NightOfTheLivingHam@reddit

..You need to stop planning and you need to start leaving the company yesterday.

Reply

[-]

Mailstorm@reddit

Then you email your manager those things. You'll be the one blames for when it doesn't work

Reply

[-]

Valdaraak@reddit

> he told me not to say these kind of things because it sounds bad... Your only response to that should be "because it *is*".

Reply

[-]

A_Unique_User68801@reddit

REV UP THOSE RESUME GENERATORS.

Reply

[-]

Adderall-Buyers-Club@reddit

Youre gonna end up on the news. You need to document all of this. Cause you are the first one they are gonna throw under the bus.

Reply

[-]

Grrl_geek@reddit

Why? Because it IS.

Reply

[-]

blbd@reddit

It doesn't just *sound* bad.

Reply

[-]

reedacus25@reddit

> send at least 8500 emails covering your ass "Oh, the mailserver was moved into the basement, so we lost all of the emails."

Reply

[-]

natefrogg1@reddit

I got to do this but they had buildings built on the property and everything was crammed into 1 accounting office and a video editing bay, good luck man

Reply

[-]

wireless_brains@reddit

I just completed the exact opposite. We migrated out of my CEO's basement to the cloud. Azure and AWS were too expensive in our research. We settled on a FedRAMP authorized provider. Migrated on-prem AD to Entra. Taking advantage of some Saas but still have a dozen servers for our DC's, web apps, backups. Now my boss can go on vacation without worrying about power outages or tape rotation in his basement. His electric bill was cut in half and the basement is 12 degrees F cooler. The dedicated business line with 5 static IP's cost HALF the amount of our cloud provider costs. Look for a cost effective provider. DM me and I'll give you a name. Their prices are more than fair. Approx. 60% less than the big three.

Reply

[-]

Due-Log8609@reddit

Dont forget physical access. Whose getting in when the CEO's on vacation?

Reply

[-]

natefrogg1@reddit

The CEO would have one of their security staff or cleaning crew or even their chef let me into the property, it was like that regardless of where the CEO happened to physically be, it was the same way at all of their properties

Reply

[-]

Mlyonff@reddit

Besides this being an incredibly stupid idea and that i 100% agree you should update your resume and start lookkng for a new gig, i’m more than happy to help you find out what DIA options are available at this CEO’s house. I’m a broker. Shoot me a DM if you want some options. -Mike

Reply

[-]

5SpeedFun@reddit

OP - this right here you want DIA and not home business internet

Reply

[-]

Viperonious@reddit

Thomas had never seen such bullshit before

Reply

[-]

ghjm@reddit

I had to do this once. In my case the company was going out of business and the CEO made an offer to buy the servers/IP/customers/etc. This was all done at the board level with no consultation with IT, and we were all being fired anyway, so he paid me as a consultant. There was no concept of high availability - he just wanted to run it all on his RoadRunner cable modem. I put together a shit-tier dynamic DNS scheme and got it to where the sites were all reachable. Speed and reliability kinda didn't matter because there were almost no customers anyway. I came across him years later, and damned if he didn't build that thing into a workable business and eventually migrate it all to AWS.

Reply

[-]

SlipStream289@reddit

The old 3am server is down and have to go into the datacenter to fix something and cannot get into the house to fix the issue. LOL worst idea ever!

Reply

[-]

5SpeedFun@reddit

Plot twist - CeO has a thing for Op and this is how she gets him to go to her place after he’s refused multiple times.

Reply

[-]

badlybane@reddit

Tell me the Owner intents to commit fraud without telling me the owner intends to Commit fraud. Owner commits fraud. Feds start sniffing around.. Then suddenly there is a house fire in the basement and the usb connected back up drive was fried as well.

Reply

[-]

rehab212@reddit

IRS is going to be interested in all of this as well. Guaranteed dude is going to use business funds to remodel his house.

Reply

[-]

Malthuul@reddit

Suddenly the entire basement has been refurbished with company funds. To... *Checks notes*.. host a data center. The newly installed bar and fresh stocked liquor was a requirement for post-work team building exercises.

Reply

[-]

NightOfTheLivingHam@reddit

This is exactly what I was thinking. There's fraud going on or about to be. The house is probably owned by an entity that the CEO has ownership in. My old job had a client who had the company data moved to his home office server as he "Downsized" operations. He had some other tech do this on the downlow so that was the first sign of something shady going on. We found out some years later he got raided by the Feds and it was the dumbest thing he did because they were able to use that to pierce the corporate veil and nail him with everything his company did. This CEO is either going to have a house fire soon or is REALLY fucking dumb to think they can hide servers in their basement.

Reply

[-]

19610taw3@reddit

There is really something fishy going on. I'd bounce ASAP

Reply

[-]

CeeMX@reddit

100 people is not something I would consider a small company anymore. But even for 10 people I would never recommend anyone to do this

Reply

[-]

Standard_Text480@reddit

Update website to say “all data proudly stored in the CEOs basement” What happens when the CEO resigns, gets fired, dies, retires?

Reply

[-]

5SpeedFun@reddit

Maybe CEO wants job security?

Reply

[-]

Nik_Tesla@reddit

If you host any services for customers on these servers, you're gonna need some kind of SD WAN. There's no way to never have a circuit go down, especially residential lines. You're going to need at least 2, preferably 3 internet lines. If they all just go into a firewall and you setup basic failover, that isn't the end of it. When one of them goes down (and they will), it will be very noticable during the switchover, both for employees inside the company, and for customers accessing stuff on the servers. Do yourself a favor and get them to shell out for some kind of SD-WAN, whether that be additional features on a firewall or a separate device.

Reply

[-]

ThatBCHGuy@reddit

> Networking equipment strong enough to handle more than 20 people on his network at a time. That's and interesting way to put that, heh. I need some strong switches.

Reply

[-]

wiseguy9317@reddit

Asking for redundant incoming power should slow your idiot mgmt down.

Reply

[-]

RavenWolf1@reddit

Well, I know company where one of the IT stuff had NAS on his bedroom closet and that functioned as off-site backup system.

Reply

[-]

Secretly_Housefly@reddit

Treat it as you would moving into a new datacenter, don't try to define all sorts of stuff that a residence will clearly NOT have. Just schedule a time to install servers and then ask where the network drops are, where the power is, etc. LET IT CRASH AND BURN I repeat: this is not your company, do not invest your stress into this bad idea. They want it to happen, let them do it, let it fail.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

>They want it to happen, let them do it, let it fail. Thank you so much for this... reading all the comments have got me a bit freaked out so i needed this reminder

Reply

[-]

Secretly_Housefly@reddit

I've been through some dumb CEO decisions (although nothing as bad as this) and I've found it important to remind myself that it's not my company. Sure I'll attempt to support it, but I'm not going to stress about it, I'm not going to think about it in my off hours, etc. and if it fails hard enough to tank the company something else surely would have. CYA with documentation of your opposition of this and keep your resume fresh just incase.

Reply

[-]

amajorblues@reddit

Do you get to take( and drink ) beers out of her fridge? Looking for a possible upside.

Reply

[-]

d00ber@reddit

This all sounds insane. How many servers do you have? What kind of stuff are you running on them?

Reply

[-]

dcaponegro@reddit

Just do the best you can and start looking for a new job. This is going to be a mess.

Reply

[-]

UMustBeNooHere@reddit

Time to brush up that resume, my friend.

Reply

[-]

Marathon2021@reddit

> Yes before you ask the main reason were leaving is that **the cloud has gotten too expensive.** Uh, there are other options? Plenty of colocation providers around that will rent you *proper* space, power, and cooling. Just BYO hardware. For example - www.equinix.com is one but there are hundreds of other companies. Many will rent out as small as a single rack footprint. In any case, there are about a billion reasons I can think of why this is a bad idea - and it's either one of two things ... the CEO doesn't *know* there are other options, in which case it's your job to enlighten her. Or, the CEO is a nitwit and/or they're doing this to make improvements to their home off the back of the company's finances or whatever. In which case, honestly, you should start updating your resume and looking for a new job because this won't be the only epically stupid decision they're likely to make. If you have to make a go of it, for power I'd put on solar panels + some batteries - Tesla powerwall, enphase, whomever. That should get you some level of redundancy although it won't last long at roughly ... hmm ... 3.5kw per rack of IT load, roughly 1:1 ratio for cooling and other overhead ... probably 2 racks ... about 15kw of sustained load Speaking of which, probably gotta up that residential HVAC capacity - 20 servers (I'm guessing that's about 2 racks worth) is going to put off a lot of heat, much more than a standard residential AC can put up with. Oh, and residential AC generally is going off and on throughout the day, whereas 24x7 servers throwing off however many BTUs of heat, is going to need 24x7x365 HVAC. I hope they like the sound of living in their home with an industrial HVAC system running every minute of every damn day. Connectivity, I'd put in maybe Starlink as a backup and then whatever is the beefiest fiber-to-the-home is that you can get. But yeah, residential neighborhoods aren't wired for resiliency. There are some routers that can smartly switch over. VPN termination ... shouldn't be anything of concern. It's just IP packets at layer 2/3. You need to focus on layers 1 & 2 mostly. Fire suppression? Pshhh ... you're not going to have that in residential construction. Sure hope the homeowner's policy doesn't tell the owner to fuck off if that many servers starts a fire.

Reply

[-]

lvlint67@reddit

Business class internet ideally with a secondary but that's probably not happening. Redundant power. Either generator or battery. Security. Locking racks. Fire suppression. What the home owners insurance says about this plan. The size of the actual electric box entering the house. Water/humidity/flooding. Any local zoning board concerns

Reply

[-]

ConfidentAmbition601@reddit

Focus your energy on finding a new job. This setup will eventually have a massive failure and you probably don't want to be around when it does.

Reply

[-]

StanQuizzy@reddit

"Why do we need to spend $2k a month on a business fiber internet connection, the $50/month comcast is just fine! My son games on it all the time and he never has an issue!"

Reply

[-]

PlebastianDjurovic@reddit

Business internet doesn't cost that much. It's also only 20 servers. My only curiosity is what's their workload. Also if their home is zoned for business they'll likely not have that problem either.

Reply

[-]

Que_Ball@reddit

Malicious compliance time. Just schedule a nightly UPS test and make sure it triggers the alarm. Get a really obnoxious 1u rackmount like the Asus servers that sound like a jet at takeoff to do scheduled reboots so the fans spin up to 100% on Sundays. You get the idea.

Reply

[-]

Kindly_Recording_322@reddit

No way in hell this is a real post. You meant to post in r/shittysysadmin didnt you?

Reply

[-]

finobi@reddit

Set the fans to run at max speed constantly.

Reply

[-]

silentstorm2008@reddit

Flood area? Homeowners Insurance will surely cover the loses of an SMB!

Reply

[-]

bitslammer@reddit

LOL....his homeowners insurance will love to hear this as will his business insurer, assuming an idiot like this has either.

Reply

[-]

silentstorm2008@reddit

Flood area? Insurance will surely cover the loses of an entire company

Reply

[-]

Terriblyboard@reddit

yes OP bring this up with them and let them know that this will need to be insured as a business and may or may not be allowed to legally happen depending on the location.

Reply

[-]

garaks_tailor@reddit

Cyber insurance too. Jesus!

Reply

[-]

mineral_minion@reddit

I highly doubt a business running this way has cyber insurance.

Reply

[-]

tk42967@reddit

Wait till his electric company notifies the feds of the jump in usage and they think he's running a grow operation.

Reply

[-]

soshwag@reddit

I was thinking the same thing! They are getting their insurance dropped across the board lol. (If they have any)

Reply

[-]

PlebastianDjurovic@reddit

The problems you've listed are simply not problems. It's 20 Servers. As for electricity cost, that's written off on the cost of the business. Also no rent for a space that probably cost 10k to 15k a month. Only problem I see is redundancy and security of data in case of a fire.

Reply

[-]

Jeremy_Zaretski@reddit

* air filtration * the ability to move the equipment * power transmission and distribution * static electricity discharge protection * power surge protection * uninterruptible power supplies * backup power generation * flood mitigation * fire mitigation * on-premises security

Reply

[-]

BabyLinuxAdmin@reddit (OP)

You're the best thank you for answering it bulleted. I didn't consider fire mitigation

Reply

[-]

garaks_tailor@reddit

Insurance will require it and be sending someone out to verify.

Reply

[-]

Valdaraak@reddit

Funny you assume the owner is actually going to tell the insurance company about this plan.

Reply

[-]

garaks_tailor@reddit

Honestly I assumed this whole thing was going to end in defrauding an insurance company with a housefire while covering up some other crimes by destroying the servers

Reply

[-]

Fr0gm4n@reddit

* Vermin mitigation Rodents and insects will happily ruin a lot of gear and some random basement in a home is going to be a magnet vs a real DC where things like food and cardboard are banned beyond the lunchroom/offices.

Reply

[-]

jmbwell@reddit

Setting aside policy questions. 20 servers isn’t that much in terms of power and heat, is it? They don’t necessarily all pull their full nameplate rating at all times. At one job I had, we ran 15 Power Mac G5 computers in the room where I worked, on three 20-amp circuits that were honestly overprovisioned. We had a dedicated 1-ton AC. It was fine. And these days r/homelab routinely posts racks full of equipment running in people’s apartments and garages. I wouldnt do it, but it evidently can be done. So 20 modern servers doesn’t strike me as immediately a non-starter in terms of power and heat. 200-amp service and a 15k BTU A/C could easily be plenty. ”Somehow” getting a VPN? Isn’t this pretty routine? Networking equipment “strong” enough for 20 users? You mean like a 24-port 1G switch for client devices? Also really common, right? Electric bill: not your problem? You can get redundant cooling systems and data connections from multiple providers (AT&T and Comcast both happily provision ‘business class‘ connections in residences). You can also get decent working conditions if money is no object. So no hard stops there. Redundant grid power will be impossible, so you’re looking at a natural gas standby generator, probably 50kW, and it won’t be the cleanest power; the THD on the common generac type systems is bad. The electrician can give better advice here. After hours access and working conditions will be a factor, though I’m sure the answer will be “oh it’s fine, we will get you whatever you need.” As part of a holistic plan, do you need all 20 servers? Can those workloads be combined with more virtualization, etc? Again, this is all setting policy questions aside. Those are for legal. But as IT, if you had unlimited budget to build a DC in your basement, why wouldn’t you?

Reply

[-]

furtive@reddit

The cloud is just somebody else’s basement.

Reply

[-]

PlsChgMe@reddit

Ha ha! Good one!

Reply

[-]

DailonMarkMann@reddit

banger.

Reply

[-]

lelio98@reddit

Get these requests in writing and forward that to a personal account. Cover yourself.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

Yea man this post really blew up. I plan on letting it ride until tomorrow then deleting the post

Reply

[-]

Mammoth_Loan_984@reddit

Keep in mind that many tech vendors support contracts stipulate that they are not liable for issues or hardware replacements if their appliances aren’t properly installed in a safe & clean environment. I once saw a company lose money because they kept sending back failed hardware. Q&A opens it up, and it’s absolutely full of shit - dust, dirt, etc. Unusual considering it had been on-site for only a few months. It ended up being a similar situation to yours - running on someone’s floor somewhere, IIRC.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

So in summary add to the list environment variables since its not a controlled env

Reply

[-]

Mammoth_Loan_984@reddit

This is more about going over any hardware contracts you have and reviewing for any T&C’s about not running equipment in a guys basement. Replacement and support policies, etc.

Reply

[-]

artificialhacker@reddit

> 100 people > 20 servers ?????? How?? That’s could easily be condensed to one or two servers

Reply

[-]

MrPinga0@reddit

sounds shady or your CEO is a monumental idiot or both. I think both. I would GTFO of there asap.

Reply

[-]

Tx_Drewdad@reddit

First thought is: Basements flood. Second thought is that this is such a marvelously ridiculous idea Additional thoughts: Insufficient power Insufficient redundant power Insufficient UPS Insufficient cooling Grounding? Lightning arresters? How do you get someone on site for support? What do you do when the A/C cuts out? We had our DR site at our office building for a long time, and repeatedly ran into issues where facilities maintenance, inspections, etc. would cause loss of power to the entire building, including our equipment. Power loss would cause overheating even when UPS kept power to our equipment (building cooling wasn't on UPS and required manual intervention). Can't imagine what it'll be like in a residential building. What happens when A/C fails, the whole place overheats, and the owner is out of town?

Reply

[-]

matt95110@reddit

Friend of mine had something similar happen, except it was the DR site for the company. Would have been around a dozen servers and all of the associated equipment for restoring from all their backups. The CTO was willing to host it in his basement, but it eventually got shot down because most of the IT staff hated the guy and refused to go to his house.

Reply

[-]

xzer@reddit

I think just planning the electrical should get him to reconsider...lol

Reply

[-]

pantherghast@reddit

CEO gets fired a year from now. ![gif](giphy|jPAdK8Nfzzwt2|downsized)

Reply

[-]

sushifencer@reddit

Please tell me you’re not in Arizona.

Reply

[-]

tk42967@reddit

We had a director of IT at one place who cut a deal with his buddy who owned self storage. We rented 4 large units and had our BCS site there.

Reply

[-]

pianobench007@reddit

Jake is that you? Blink twice if you can answer us safely. Is the CEO Linus's wife?

Reply

[-]

battleBrew@reddit

mmm but the bosses basement will be so toasty warm all the, the cats & dogs will love it.

Reply

[-]

HDClown@reddit

Bro, my company wound down operations and only has a few people employed keeping the lights to deal with latent compliance/audit needs and try to sell the entity. I was asked if we could save money by moving the couple servers we have into an employee's basement. I simply told them you won't have proper power and cooling in a residence to deal with it and they never brought it up again. This is for a company that is basically operating in the red every month, and they didn't push me on it. That should tell you something.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

This is how i thought my scenario would end honestly and i mentioned the same thing but was met with the "Were going to move forward"

Reply

[-]

satchelsofgold@reddit

To be fair, I've been hearing more and more about people moving out of the cloud because of costs, but that highly depends on what you're running. It can be ridiculous for some workloads. So you're also moving from doing zero hardware and networking to doing all of it? I don't see what you wouldn't move into a local professional colocation datacenter. That is also relatively cheap, but still a professional environment with full redundancy and security. Moving it into a residential basement assuming you have a good dedicated network line will be fine until the day it isn't, for instance when the powerline goes down in his neighbourhood or maybe flooding. Make sure you're not around that day or are able to quite on the spot that day.

Reply

[-]

OGicecoled@reddit

I don’t think they’re moving out of the cloud but out of a colo data center. It’s still dumb either way.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

Yep moving out of the data center and wiping the old hardware to reuse it for something new in her basement sums it up

Reply

[-]

OGicecoled@reddit

Yeah I don’t envy you. No advice but thoughts and prayers.

Reply

[-]

host_work@reddit

Have you communicated how loud the servers are when running to them?

Reply

[-]

BabyLinuxAdmin@reddit (OP)

Just added this to the many reasons on my list! Thank you so much for your input. Keep in mind she's loaded so her home is already huge. She may not view this as an issue but i am going to throw it on their since if her guests complain we can say we told her.

Reply

[-]

TheMartok@reddit

Which level of data breach are we looking at? Biden Clinton Trump And will it have proper cooling

Reply

[-]

NightOfTheLivingHam@reddit

The CEO wants more control and there may be some legal fuckery afoot where he wants the servers hidden on his personal property or a property that is controlled by an entity he owns that is separate from your company. Either to leverage control over other executives or because the company is being investigated and they would need to subpoena her or her entity that owns the property to get access to the servers physically. The only time I have ever seen this pulled off was because there was nefarious shit going on and they were hiding infrastructure from authorities. tl;dr: Freshen your resume and RUN.

Reply

[-]

crackerjam@reddit

Instead of cloud companies, have you looked into just renting colo space? It's going to be monumentally cheaper and more reliable than building your own datacenter in someone's basement. Moving away from cloud providers because of cost is valid, but moving into a basement is not.

Reply

[-]

iwoketoanightmare@reddit

Is it public so I can buy a short position?

Reply

[-]

funktopus@reddit

Update your resume and start looking OP. This isn't going to end well at all.

Reply

[-]

Fireman476@reddit

The biggest thing you will need is an updated resume.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

Bro the tech market is so bad right now... i fully want and plan to the second things get better. no one is hiring it seems at least for my area

Reply

[-]

Background-Case4502@reddit

I would go work in fast food or retail over dealing with the terrible time you are about to have.

Reply

[-]

QuantumDiogenes@reddit

Bro, not even McDonald's is hiring. I've tried

Reply

[-]

Adderall-Buyers-Club@reddit

If I could, I would give you an award and a reacharound for this amazing comment.

Reply

[-]

jtbis@reddit

My company is in the process of planning a move of our similarly sized on-prem data center. We are expecting it to cost $175,000-$200,000 just to build out. You need a generator, climate control, halon, waterproofing… Also ISPs aren’t going to bring enterprise-class circuits into a residential neighborhood. If they do agree to, there will likely be massive construction fees.

Reply

[-]

the_syco@reddit

> ISP And a second one. And a diesel generator.

Reply

[-]

topkekcop@reddit

No way this is real

Reply

[-]

bigfoot_76@reddit

Not to mention the possibility of the existing power equipment being setup for 3*Ø* and not 120VAC at a residence. Either way, that home's electrical service isn't adequate.

Reply

[-]

IdiosyncraticBond@reddit

What is your risk assessment on the basement flooding? 😉

Reply

[-]

PowerShellGenius@reddit

Fire and flood risks are probably somewhat low, but may go up astronomically in case of upcoming IRS or BSA audit. I'm just assuming that's why he wants them there.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

I think at this point the best course of action is us just coming at it from a legal and risk standpoint and do no research on if its possible to do from a technical standpoint

Reply

[-]

00001000U@reddit

Fire suppression, physical security, redundant power and generator/ats purchasing, external monitoring. GL with a fat chunk of that.

Reply

[-]

FinancialBottle3045@reddit

OP, did you take the position I just resigned from?

Reply

[-]

InquisitiveTechy@reddit

Luckily basements never flood, perfect place for servers.

Reply

[-]

blbd@reddit

Flooding sinking the servers. Failing formal audits. Fires. BOHICA, and you won't even get any lubrication.

Reply

[-]

AsleepBison4718@reddit

Also access control. CEO can let whomever they want into the house and other people live in it. Unless they're spending a million dollars to build a secure room underground, that's a massive risk. They'd have to probably hire security to guard it too... This whole thing is asinine.

Reply

[-]

TechGuyMSP@reddit

Really, this isn't up to them, it's up to their ISP. Unless you have symmetrical fiber available, I don't see how any of this works. Even then you'd need a second ISP failover. Likewise, are the correct amperage/voltage power outlets even available for your UPS? Data and power may stop this before it begins.

Reply

[-]

BabyLinuxAdmin@reddit (OP)

Symmetrical fiber is a huge factor that i appreciate you bringing up! Thank you tons

Reply

[-]

Frothyleet@reddit

Make sure you aren't just getting quoted for fiber - you need DIA fiber. Lots of residential areas can get shared fiber connections, even billed as "business grade", but it's on the same SLAs as DSL or cable and your service can vary wildly depending on who you are sharing bandwidth with.

Reply

[-]

bensonmojo@reddit

find another job ASAP. this is a disaster waiting to happen. others have provided valuable input, but the honest answer is get tf outta there

Reply

[-]

colin8651@reddit

Is the CEO the owner of the company or is this some power play to stop the board from firing them?

Reply

[-]

BabyLinuxAdmin@reddit (OP)

Yep she is the owner and CEO

Reply

[-]

digitaldisease@reddit

Any chance you have regulations like PCI DSS that you can throw at this to make sure that the cost of compliance makes this even more daunting?

Reply

[-]

deletesystemthirty2@reddit

hopefully they have flood insurance lol

Reply

[-]

fennecdore@reddit

>What are some things i should be considering Look for another job and watch the shitshow unfold with a bucket of popcorn

Reply

[-]

Grrl_geek@reddit

THIS is the way!

Reply

[-]

Adderall-Buyers-Club@reddit

Your company may want to get ahold of their legal department because you may risk compliance and insurance liability. Also, i doubt Telcos will provide MPLS or other business circuits to the home. Plus the backup generators and what not.

Reply

[-]

Ssakaa@reddit

Business access to business equipment outside of hours?

Reply

[-]

gmanley2@reddit

You've got to be kidding

Reply

[-]

sevenfiftynorth@reddit

Are you in an area where residential basement flooding happens on occasion? I would never put my servers and UPSes below grade for this reason alone in any place that I've ever lived.

Reply

[-]

Individual_Jelly1987@reddit

You've made your objections known. Time to run away!

Reply

[-]

LetzGetz@reddit

Dang. Can't wait to see how this plays out @.@

Reply

[-]

Art_Vand_Throw001@reddit

What is he Hilary Clinton. Better get yo bleach bit ready son.

Reply

Reply to Post

237 Comments