I knew exactly how this project was going to go…
Posted by UMustBeNooHere@reddit | sysadmin | View on Reddit | 39 comments
I am a Datacenter Implementation Engineer for a local MSP. I have a project to deploy a Hyper-V cluster for a small college \~3 hours away. We had our planning meeting with customer several weeks ago. We confirmed they were good to go - all hardware delivered on site, racks ready to go, power requirements met, etc. I plan my visit this past Monday, so they had plenty of time to prepare.
I get onsite and call my POC - the network administrator - he’ll be down in a minute. Okay, no biggie, sometimes we have to wait on things. About 15 minutes later he finds me and takes me to the “datacenter”. We enter an outside room with a few (I’m assuming) techs. He asks the lead tech “Have you seen any Dell Servers around”?
My mind does that screeching halt sound.
I’m thinking to myself “you had 3 weeks to locate this hardware that you said you had onsite and get it staged where we need it and now you’re asking a tech if they’ve seen them??”
They make some phone calls and finally “locate them”. He then proceeds to ask me if I have a truck to go pick them up as they’re in another building. I tell him no as I have a car. Now they have to track down transport.
Okay, while we wait, let’s get into the datacenter and start getting ready to rack the hardware. I walk in…and they have a bank of seven 2 post racks. That’s it. This place is ancient. Crap sitting on top of UPSs, cables everywhere. Oh boy.
Now, anyone who’s racked servers and storage arrays knows this is a problem. After I do a quick lookup to verify that there indeed no 2 post mount kits for the hardware, I let him know they won’t mount. He’s okay with just sitting them on a UPS at the bottom of one of the racks.
I recommend against it. But the servers (3) are 1U and the array is the tiniest 2U storage array I have ever seen. It’s like the footprint of a medium pizza box and weighs less than the servers. So we put them on the “shelf” and I begin to run power.
I’m ready to run the Twinax (direct connection) cables and ask where the switches are. Of course they’re 5 racks down - about 9 meters away. So we’re looking at about 15m cable needed - 12 in total. “Alright, “Mr. Customer”, where’s your cables?”
“Uhm…you didn’t bring them with you”?
Fuck no I didn’t bring them. You said in the kickoff meeting that you had all cables!
So I had to lookup someplace local to find these cables…15M!
I find a place 27 miles away. They go grab them, get it all cabled up.
Now, here’s the part where I tell you Mr. Network Administrator said he would have the IP cutsheet with all IPs, VLANs, etc. ready for me….
Narrator: “He did not.”
So I spent the next 3 hours waiting for him to give me IP here, IP there, hold on gotta modify this firewall rule to allow that…
I was finally done about 5PM (arrived at 9AM) with a lovely 3+ hour drive home. What should have been 4 hours turned into 9.
Now I’m fighting with Hyper-V switches as their network administrator tries to troubleshoot VLANs and firewall rules upstream.
BamaTony64@reddit
Sounds like there wasn't much of a site survey or scope of work there. That's a huge failure on the MSP. Disorganized customers are a thing, but a site survey and a scope of work would have mitigated most of that.
PawnF4@reddit
PicsOrItDidntHappen never take a customer at their word.
BatemansChainsaw@reddit
This is why my former MSP had a camera in the rack looking down at all the little blinkinlights for some clients.
1z1z2x2x3c3c4v4v@reddit
Took the words right out of my mouth...
Tyr--07@reddit
That's what I was thinking. We would have went on site and physically looked at everything they have, taken pictures etc before quoting things out. We would have also included some common things people say they'll have and not, so we'll make sure it's clear if additional network discovery or changes are needed, there's additional cost.
Different_Coat_3346@reddit
Dell storage arrays are 100% garbage IMO.
Background_Lemon_981@reddit
So this is a case of not being explicit about requirements. I need: - all servers on site in the room where they are being set up - pictures of your racks needed by x date - pictures of your power set up needed by x date - inventory of cables on hand in room including connection type on each end and length
Etc. A lot of this can be handled up front. I’m a big fan of pictures because what is described one way often turns out to be something else. If you can anticipate problems, then they aren’t problems. It’s only a problem when it’s not planned for.
itishowitisanditbad@reddit
I'd also want all the network info on hand BEFORE I go onsite.
So many red flags on the prep work here.
tdhuck@reddit
Agree, the MSP should have asked for hostnames and IPs before going on site. "Oh you only gave us 10 IPs, we needed 25" the MSP should have prepped better than they did.
IamHydrogenMike@reddit
I had a customer get mad at me once when I was doing a DC install because I sent out a list with every little item I needed before I could perform the install. They said I was being condescending to them because they knew what we needed; I didn't need it to be so explicit. The list isn't just for the customer; it was for me as well to make sure we had everything prepped before I drove down there to put everything in. Having a plan before walking in the door is basic project planning. I have wasted to much of my life waiting around for people like this, and I have better stuff to do.
Ssakaa@reddit
"I can either assume you remembered every single thing on this list, and waste a day, that you are billed for either way, when we're wrong, or I can provide the list for you to explicitly check and sign off on, and we can get this done right."
itishowitisanditbad@reddit
THEY confirmed they were good to go.
You didn't.
I think this is what you're trying to say but it is a very key distinction.
iB83gbRo@reddit
Doveryay, no proveryay!
r1kupanda@reddit
I think everyone here has summarized the scoping issue, but I would also point out a change management issue. When things in an implementation deviate from what you've been told, then you should have a process to bill appropriately ilwince the project scope has changed. It's not always possible to get 100% of the unknowns scoped, so a project proposal should have a list of assumptions, like: Msp assumes client has servers at the install location eith no transport needed Msp assumes client has requisite cabling for the install Msp assumes lan diagram will be provided upon request
And then a clause saying "anything outside of the scope and assumptions are subject to additional billable hours/extra rates"
Ive_seen_things_that@reddit
I dunno... Just sounds like a normal day in the industry to me. Consider yourself lucky this isn't a normal day and sorta an outlier.
dphoenix1@reddit
Yup. Definitely don’t miss my days at an MSP.
OP, use this as a learning experience for things you should have or do beforehand. If you’re gonna do more implementations at customer sites, a site visit can be invaluable. Or at least comprehensive pictures with all involved equipment and infrastructure identified. Obtain lists of IPs/subnets/switchports early, don’t just take the customer’s word that they will have them. Figure out run lengths, don’t assume the switchers will be in the same rack the equipment is. Etc. etc. etc. Every one of these issues could have been identified and impact at least minimized with proper due diligence. Customers are idiots. That’s why they’re paying you.
PDQ_Brockstar@reddit
Yeah, this seems pretty par for the course. You should start requiring visual evidence that all requirements had been met and a virtual tour of the data center for anything over an hour drive. I'm kind of kidding, but honestly, it could probably help lol
dataslinger@reddit
Have them jump on a meeting on their phone before you travel to do a zoom walkthrough: "Show me the space with the racks. And show me where the servers are. And where are the switches?"
aguynamedbrand@reddit
Why did you not require the IP information before being onsite?
In the future do a site surgery first or require pictures at the very least.
This is not all on the school.
Haunting-Prior-NaN@reddit
This. School is (very probably) paying through its nose for a turnkey solution.
udsd007@reddit
“Turnkey” often means “turkey”.
unethicalposter@reddit
I never would have left without a site survey that should be a requirement before the install phase.
CeC-P@reddit
We never get assigned a dedicated Person Of Color where I work. Is that like a regional thing?
dubsyGG@reddit
This isn't on your client. You should have scoped the work to be performed with an ONSITE site survey well in advance.
Never count on your client to have cables on-hand.; either maintain stock from which to pull, or order fresh cables in the colors and lengths you need for each specific job.
Never ask your client if THEY have their network documented properly; ask them to provide you the documentation in advance, optimally during the onsite survey. If they don't have it, work with them to write it yourself.
Never expect the client to be ready for you; expect that you will be relied upon to do most of the work because THAT is what they're paying you to do in the first place.
A job like this should be four trips *MINIMUM*:
I've been in the IT industry for 15 years, MSP field for 12 years. Worked my way up from basic Helpdesk to Senior Engineer and Project Manager; were I in your clients' seat right now, I'd start calling around for other providers, and possibly local to the area.
THIS. IS. NOT. YOUR. CLIENT'S. FAULT.
DankPalumbo@reddit
All I kept asking myself is "where's the PM?" All of this should have been known going into this. How could you have a kick-off meeting an nobody has even seen the site? What an absolute failure on both ends, and of course leaving the implementation engineer holding the bag..... tsk tsk: bad msp.
rr1965@reddit
Sounds like an install I had in NYC late 80s. Confirmed before I flew out that they had power and tiles cut - we provided drawings.
I get on site around noon and they have 1 receptacle done. I said I need to get the system cabled and booted up by eod as I had a ton of other stuff for remainder of the week. The union electricians looked at me and said we're waiting on the receptacles and won't be here until late tomorrow am. I call bs, hoofed about 20 blocks to an electrical supplier, snagged 2 L21-20's and 2 L21-30's, hoofed back and said here ya go.
They were pissed. Fun times.
samspock@reddit
I could see some of this if they did not have any IT on staff but this is just bad.
I had one that was also a storage set up. Two new hosts and a SAN. San was a HEAVY 4u monster that was shipped in a wooden crate. I had gone back and forth with the customer to get it ordered. I get radio silence for three weeks. Then one day their "IT guy" (DBA actually) calls and says it's sitting at UPS store about a mile away from their main office. Been there for....three weeks. It was supposed to be delivered to the data center 20 miles away. Apparently they forgot to give Dell the actual delivery address and it defaulted to their billing address, a box at the UPS store.
The manager of the UPS store was very grateful to ship it out to the data center and get it out of the middle of the front of the store. $100k unit just sitting there.
YellowOnline@reddit
Sounds like a normal day to me
ConfidentlyLearning@reddit
I was once directed to do a 'floor mount' of 5 2u Dell servers in a stack. The poor machine on the bottom of the stack was visibly compressed, and never behaved quite right.
applevinegar@reddit
"We confirmed they were good to go"
no, you clearly didn't.
This is on the MSP.
alabamaterp@reddit
Come on man, never show up to a site "blind". I always ask for pictures, it takes less than minute and saves so much more time and hassle. Also, you failed to state and confirm your expectations - room and racks free and clear, IP address ready, cable lengths stated and ready, servers unboxed. You should have had a go/no-go meeting the day before, especially with a 3 hour car ride. People have cameras on their body 24/7 and you didn't confirm by sight? Don't be afraid to show some leadership and delegate and ask questions up the ying-yang, be aggressive if you have to. Let your balls hang low and be "that guy". Your customers will appreciate it and there will be less headaches.
purawesome@reddit
Honestly bud, that didn’t go too bad. Usually ends up with me scrambling to find a hotel because it’s way too late to drive home and I haven’t slept in 36 hours…
aprimeproblem@reddit
I have the same experience with preparations for security assessments. Have the scoping call a couple of weeks in advance, explain the process, provide a document with all the steps we need in advance to do our job…. Week in advance we reach out if everything is set….. sometimes just silence, sometimes a yes….
We go on-site and they still need to do the preparation…..
I feel your pain!
Ryansit@reddit
I would have gone home with a directive for them to setup their equipment correctly as discussed and charge them for that days work and schedule for when they are really ready to work.
Aim_Fire_Ready@reddit
Sounds like a textbook case for an upcharge to me.
ProperEye8285@reddit
I get paid by the hour not by the piece :)
RedGobboRebel@reddit
Sounds par for the coarse if you didn't have project management. How to bill it out is going to be up to your MSP's management.
The IP/VLAN/Hostname sheet deliverable should have been required before the final site visit.
If the equipment is expected to be racked ahead of time, rack locations should be another pre-install deliverable. Which with enough detail can help lead into cables.
Cable needs should have been part of the project plan. Connectivity media type should have been part of the plan. SFP? Ethernet? DAC? Fiber? If fiber the type of fiber needs to be clarifies as well.
Even with these in place... shit goes sideways. SFP modules that should have worked, don't. Someone mis measures the cable lengths needed. Then you roll with it and get it done and bill it out accordingly.
MalletNGrease@reddit
Lots of billable hours in that quagmire.
Abject-Temperature31@reddit
Not in this industry but have seen similar in mine - we often ask for photographs shortly before we are booked - it can expose a lot of this before you get dragged in.