Completed a full VMware elimination in 24— happy to share what we learned

[-]

ubrtnk@reddit

Are you me? Literally had this meeting Thursday. Have about 12k and they wanna try to do it in 5 months...

[-]

robiika@reddit (OP)

It's doable, but you have to have a solid plan. We moved 5K VMs in about 2 months. Lot of sleepless nights, but glad its behind us.

[-]

LocalNefariousness95@reddit

Can you explain what your work schedule looked like during the migration?

[-]

I am not OP, but I did migrations for 11 years at IBM I can tell you it was grueling at times. I think 52 hrs with one 4 hr break was the worst. I cracked the fast-wide pipe problem using HPN-SSH. In 2008.

60 to 80 hrs was the norm on the week up to the migration, migration weekend, and the "critical care" and remediation the following week.

Workaholics, most of us.

Burnout was expected.. Drug usage, legal and not, was not uncommon.

Death was rare, but not unexpected, usually by those who used drugs to avoid dealing with life/work balance.

[-]

dignity_optional@reddit

In my experience it’s easy to move a large chunk of the “easy” workloads quickly. Getting the remaining more complicated workloads moved took significant coordination with different teams which was not quick. We moved from VMware to AWS and AVD. With VMware we didn’t really have to open support tickets much. AWS support isn’t great, but Microsoft support is fucking terrible.

[-]

thegoatcarlwheezer@reddit

How many times did you think of quitting your job or ending it all?

[-]

robiika@reddit (OP)

It was a challenging time, but rewarding. I hate Broadcom more than ever.

[-]

electrowiz64@reddit

Shareholders HAVE to be pissed with everyone exiting their contracts, this is such a disappointment. Even RHEL had some pretty crazy renewals at my last job, I’d GLADLY help companies migrate to Ubuntu

[-]

Jhamin1@reddit

They gave a presentation to Shareholders in 2022 where they laid out everything they are doing now.

They understood and expected to lose all but 600 or so of their biggest clients. By doing so, they could cut support staff, cut R&D, and still make a ton of money. Layoffs (19,000+ thus far) combined with profits: Wallstreet catnip.

What's the long term plan? Who cares? VMWare will implode over several years but by then Broadcomm will have eaten some other company to pump & dump and all the current leadership will have cashed in

[-]

treefall1n@reddit

What’s rewarding about this work? Did you get life changing money?

[-]

jameson71@reddit

People with large accomplishments many times feel rewarded or a sense of pride for their accomplishments. If some day you accomplish anything you will understand.

[-]

heapsp@reddit

Do you know how nice it would be to work on something like this, without shoulder taps, without random side projects, without some director shifting your priorities. Being able to sink your teeth into a large project like this and be well paid through salary would be enough.

[-]

deliciousnightmares@reddit

Successfully giving a middle finger to Broadcom with over 15,000 VMs is its own reward

[-]

treefall1n@reddit

I have 600 to do and I think about it every day!

[-]

tarvijron@reddit

Yeah when you fantasized about the end of the world was it like a After The Bombs techpocalypse or was it more of a The Computers Cannot Help Us zombies situation

[-]

Superb_Raccoon@reddit

During my tenure at IBM doing data migrations, moving clients into IBM DCs, we developed a system for determining if a migration was possible given constraints. Out of 145 migrations I planned and/or executed, 1 failed. The method was designed for 2 STD DEV error, and I hit that. The one that failed was the one we warned was pushing the envelope, but they signed the deal anyway... oh well

Been 8 years, but the numbers are sort of burnt in my head. 15k in 24 months is entirely doable. Moving VMs at a pace of 400 per wave, per team was the standard, with one team doing a wave every 3 weeks.

Impressive that you had ten people, which would equate to just under 2 teams, which were 5.5 FTE. PM, Wave leader, and 4 specialists. Wave leader and PM were half time, they also managed another team.

Well done!

[-]

fintheman@reddit

Did you propose and architect this and do you get a kickback from the cost savings that you've earned for your company?

[-]

merc123@reddit

This is your job…

[-]

fintheman@reddit

If he was the leader and architect of this, that's absolutely a bonus moment.

[-]

robiika@reddit (OP)

Yes I was the overall leader to make this happen and now I see value taking this out to the marketplace. There is definitely a need and has life for at least the next 3-5 years.

[-]

fintheman@reddit

You are thinking right man! There is a lot of money out there once you start thinking technically. The closer you are to how a company makes and saves money, the more money you'll make.

It's also a perfect thing to start as a consultant for and make your money based upon the savings you can implement. Nothing easier to sell to a company.

Had a friend who became a license guru for SQL and others who ran a large project for a Fortune 200 and saw that he and his small team saved over 10 million a year in licensing and wasn't okay with the attaboy and now is killing it!

[-]

AnythingGuilty5411@reddit

I work for a large VAR, and honestly we would hire someone with just the learning experience for our post sales side immediately. You’re more valuable as an SME for this. 170k with bonus take it or leave it.

[-]

Marathon2021@reddit

You could probably make a reasonable side-hustle income putting together some YouTube videos walking folks through a framework for it.

Commercializing something and finding, contacting, and and convincing clients to pay you money … is not a trivial challenge.

YT has a low barrier to entry, and immediate global reach (and it may lead to some orgs reaching out to you asking to consult).

[-]

iLikecheesegrilled@reddit

What do you think ? Of course not

[-]

Kelsier25@reddit

Man I'm jealous. Probably doxxing myself with this, but F500 here and in the middle of a VMware to Hyper-v migration. Not sure how long planning was because it's outsourced, but it seems like it was minimal. Most of the migration took place over 2 months. Broadcom hit us with a 1500% increase in pricing. Migration seemed smooth at first, but we've been struggling the past few weeks. We're seeing resource usage spike on random servers until they lock up and eventually reboot.

[-]

FrostyMasterpiece400@reddit

What is your storage backend?

[-]

ttgl39@reddit

Can you share a high level plan?

[-]

TechGeekTraveler@reddit

What tooling did you use for linux? Windows vmware -> hyperv just works 99% of the time. linux vms? Ugh

[-]

robiika@reddit (OP)

Let's just say I hate Hyper-V. Openshift-Virt was much smoother once we got going. Linux VMs was pretty easy using the migration toolkit provided by RedHat.

[-]

hitman133295@reddit

Wtf really? We’re moving to openshift and it’s absolute shit show

[-]

Superb_Raccoon@reddit

Since I work at IBM, an i have patents in migration... that is not the typical experience.

Difficults usually stem from decisions made long ago that make migration difficult.

[-]

panopticon31@reddit

Hyper-V bare metal is fine for small deployments.

Failover cluster is a lovecraftian fucking nightmare.

[-]

kriebz@reddit

Not calling you out, but I'm surprised. In my experience changing almost anything (or sometimes nothing) causes Windows to fail to boot. Plus shoveling workloads is tedious and usually fraught with licensing problems. Linux otoh never seems to balk at booting up in a different "body", plus shoveling software is usually just tar and scp. Of course, p2v and v2v tools have gotten a lot better.

[-]

robiika@reddit (OP)

We moved Windows workloads to Hyper-V and Linux to Openshift as it made the most sense at the time. However, there were times where we wanted to just do all Openshift. We have had more problems with Hyper-V honestly.

[-]

Zerowig@reddit

15,000 servers and no vendors telling you they only support VMware?

[-]

robiika@reddit (OP)

We got that multiple times but powered through. Vendors are realizing companies are switching and must do the same. This mainly was an issue with appliances.

[-]

Hotdogfromparadise@reddit

Can you define “powered through” in this case? Seems like the companies saying that will use this against you when it comes to post migration support.

[-]

treefall1n@reddit

Hmmmm

[-]

GroundChuck117@reddit

With HyperV, what sort of iac tooling do you have in place?

[-]

yukantspel@reddit

How did you decide what went to which hypervisor?
What features from VMware are you missing now that you wish you still had?
How are you handling backups?
Do you have some idea for how much is being saved with the new architecture[s]?

[-]

robiika@reddit (OP)

Linux = Openshift
Windows = Hyper-V

We didn't want another Broadcom moment. The biggest feature is VROPS. We somewhat get what we need in Dynatrace, but still not as powerful as VROPS.

Cohesity for backups. Openshift wasn't supported at the time, but we pushed the vendors to fix it.
Millions in overall savings. We would continue to get screwed from Broadcom and we knew and so will you.

[-]

moonblaze95@reddit

What was the toughest thing to architect around?

Did you have any “eureka” moments / discoveries where, had you learned them earlier, would’ve saved you a boatload of effort ?

Awesome post. Good on ya

[-]

TyLeo3@reddit

24 months from assessment to execution for 15 000 vms would be impressive. I assume all type of workload. How big was the team?

[-]

robiika@reddit (OP)

That was the beauty. Less is better. We had a core team of 10 people and got the shit done. Very talented and committed people. Linux, Windows and Appliances.

[-]