Anti-rant: Virtualization still feels like magic
Posted by MediumFIRE@reddit | sysadmin | View on Reddit | 125 comments
This is a graybeard / kids don't know how easy they have it now post. I finally received a new Hyper-V Server cluster after shipping delays. Moving from Server 2019 to Server 2025 and Intel Xeon processors to AMD EPYC on the host nodes. Started moving Windows VMs over and everything just works. Then I move over a Rocky Linux VM expecting things to break...nope. Everything just works - Windows activation still active, static IP carries over including Linux, all services start with no issues. It's in the same bucket as in-place upgrades. We've come a long way
Helpjuice@reddit
There is no way this could run faster, new IT guy gets CapEx to buy multiple new all flash SANs, now there is more storage and IO than people know what to do with and the whole need to wait for it to finish generating the large report to disk is now mmm it's done already??? Or mmm backups seem to finish in minutes I need to look into that, or mmm what do you mean live migrating is already done that was a 2TB VM???
TKInstinct@reddit
I love virtualization, it's one of the coolest concepts in IT as far as I am concerned.
Confusias1@reddit
Remember managing application/service loads across your bare metal systems? Now, everything just gets its own VM...
anpr_hunter@reddit
Still true in the OT space. Ask your Siemens rep if you can run Desigo in a VM, and they'll ask "Why would you ever want to do that?!" and then just start vomiting blood and feces
Confusias1@reddit
Yea, there are still plenty of those. I remember 12-13 years'ish ago, I had a medical EHR vendor that kept insisting that their application would only run on bare-metal. It was a 100% Windows based with a MSSQL back-end. That VM ran for over a decade, first on a VMWare host, then moved to Hyper-V, and is likely still sitting warm on a hypervisor to this day. I can't recall any catastrophic failures while that app was in use.
tuxedo_jack@reddit
eMDs!
Confusias1@reddit
LOL, yup!
tuxedo_jack@reddit
You must have been working on them right as I was getting away from one of their major purveyors. Hell, they used to have offices right up Parmer from the company I was working for at the time, and one of my coworkers from there went on to be one of their senior in-house admins.
These days, it's all Athena and AdvancedMD, and Athena sucks out loud with Device Manager.
Confusias1@reddit
We started with them in 2013-14. Kept them for 3-4 years until moving to eCW. The org I was with is in Dallas, but I did make to Austin for a conference or two. Had a great time grilling their DBAs about poorly written sql reports.
tuxedo_jack@reddit
eCW made me pray for a cerebral hemorrhage.
anpr_hunter@reddit
It always comes back to the vendor wanting to control their field variables.
I had a call with Mitsubishi a few years ago about their Meleye elevator management system. The sales engineer straight-up said it has to be airgapped and unpatched, yet relies on IP connectivity to the cabs. I called him out on every engineering call asking whose bright idea it was to sell a Windows-based elevator system in a lab building that can't be patched.
Within a month of deployment my users were asking for access to the Meleye web UI from the engineering office. Nope, sorry guys, I warned you.
renegadecanuck@reddit
Thank you for making my fear of elevators even stronger.
anpr_hunter@reddit
I spent many years in property management IT and I say without a trace of hyperbole that there are major, national real estate operators whose buildingI will not step foot inside.
I've inherited their properties through M&A's from the Brookfields and CBREs and seen it all - FACP lines tied to non-functional SIP trunks, public-facing BMS headends, you name it. CRE IT is the wild west and it's absolutely going to get someone killed.
Confusias1@reddit
I work above the 10th floor, new fear unlocked, thanks bro... lol
thortgot@reddit
WTF. That's literally insane.
Ganthet72@reddit
I had a Point-of-Sale system that was the same configuration and vendor demands. Insisted on physical hardware. It got to the point where I told t hem I was considering another system that did support virtualization where they agreed to "explore the idea".
sembee2@reddit
Many years ago I was in the room with my client and a vendor. There was a high 8 figure contract literally on the table. My client had a VM only policy, vendor was insisting on bare metal. Vendor blinked first. We insisted on their public policy changing as well, not an exception for us that someone would forget or try to wriggle out of. Their entire stance on VMs changed in less than 48 hours. It was in the RFP as well, they just thought they could brow beat us in to submission.
ipreferanothername@reddit
im in health IT - yeah. that sounds right.
we still have to nag some vendors to get onto VMs, but im only down to maybe 15 physical servers now? and 1100 vms, so its mostly good.
Confusias1@reddit
Health IT is a riot...
fresh-dork@reddit
aka "we didn't test that scenario and don't want the liability if it breaks"
BrokenByEpicor@reddit
Fun fact, you can tell if the application really needs bare metal or if they're bluffing by how far their head turns around. If it's at least 180 degrees it's real. Anything less is just hyperflexibility being used to sell you on their story.
patmorgan235@reddit
Oh the app needs more computer, let just change the number of procs in the GUI and reboot it.
Confusias1@reddit
Oh and dynamic RAM... glorious.
Signal_Till_933@reddit
I had the opposite experience with dynamic RAM. It either was the cause of the leak itself or it didn’t spin up fast enough and crashed everything
Confusias1@reddit
That makes me sad bro. I'm curious what guest OSs you experienced that with? Maybe some of the other greybeards could provide some insight..
Signal_Till_933@reddit
I mean Windows for sure. Does it work better on *nix? What hypervisor?
tankerkiller125real@reddit
Worked great on Linux for me, you just have to pay attention to the initial amount provisioned, if you don't follow the docs (not a power of 2 basically) the Linux guest may not allocate more RAM correctly.
I've also had no issues on Windows, however we don't have any particularly demanding workloads on Windows.
Signal_Till_933@reddit
I’d wager your experience as more about the hypervisor you used than the OS from what you’re saying.
We were using Hyper-V lol. I’ve since left that job a long while ago though.
tankerkiller125real@reddit
Hyper-v is/was our hypervisor.
wallguy22@reddit
I had a memory leak issue on a Debian 10 guest that thankfully no longer exists.
tremblane@reddit
For me it was using LVM to add disk space to a live system without even needing a reboot.
And then, again using LVM, shuffling a running host off of a virtual disk that wasn't thin provisioned to one that was, again without a reboot.
Sunsparc@reddit
Someone doesn't have hot plugging enabled. I can change CPU/RAM allocation on the fly without interruption.
My_Big_Black_Hawk@reddit
I thought there was some kind of recommendation to not use hot-plugging unless absolutely required…
montarion@reddit
:( still using vmware?
My_Big_Black_Hawk@reddit
Very Large org. Actively moving away from VMWare over the course of a multi-phased project.
SupplePigeon@reddit
VMWare isn't bad software, they are just a bad company. Some shops either have the money or expertise and just prefer to use it over the alternatives.
reni-chan@reddit
You can't if you pass through a PCI device
Ganthet72@reddit
Oh yes! I was just telling a youngster tales of long ago when we had racks of single-use servers or trying to figure out (and then document) what apps could share servers. Then we had that magical transition where we could say "Ok, let's just spin up a VM for that"
spinrut@reddit
I used to work in a lab that ran simulators on multiple servers that would also talk to themselves
we had racks with 1u-2u servers and a couple of routers humming along doing their thing.
nowadays that whole stack (servers, routers included) could get replicated with a modestly powerful server hosting all of that on VMs
we'd borderline need earplugs in when the servers booted up and always worry about the cable management involved with all that gear. things are definitely easier now
Ganthet72@reddit
Yes! The load on our server rooms AC dropped dramatically and there was a noticeable in power consumption.
spinrut@reddit
i bet! just the logistics aspects (whether power needs, ac needs, cabling needs) drops so much with virutalization
steverikli@reddit
Heh heh, sometimes. Other times, management decides you need more virtualization servers so they can run more VM's. :-)
IME racks and machine rooms are kinda like disk space -- whatever you're doing eventually expands to fill up everything that's available.
Iwanttoberich_8671@reddit
Weve gone from careful resource juggling to basically VM lego blocks everywhere, kinda wild when you think about it
sobrique@reddit
Oh yeah. I 'got in' to sysadmin before Virtualisation really existed. OK, so there were mainframes and lpars and stuff, but 'most' computer systems were standalone servers, running a grab bag of applications, because you couldn't really justify one-per-role.
So you had maybe a handful of Windows Servers that did some file serving, some domain controller stuff, maybe hosted IIS, and perhaps a couple of SQL databases, and just maybe had a couple set aside to do just Lotus Notes....
And the Unix estate was much the same.
cjcox4@reddit
Cloudy people reminisce about "the VM days" of long ago.
caller-number-four@reddit
I remember the good old days when we'd VM DOS to run multi-line BBSes!
INSPECTOR99@reddit
Ever run D'Bridge BBS in a VM? :-)
caller-number-four@reddit
No. If I recall correctly, we ran PCBoard under DESQView. I think that was right.
VM is probably the wrong term for what was going on back in those days.
per08@reddit
It was a VM. DESQView (or WIndows, even) made the application think it was running by itself on its own DOS computer. We just considered it really to just be multitasking, not a true VM.
cjcox4@reddit
Wow. My last DOS oddity was probably 17 years go or more, where a software company I worked for still maintained a DOS version of their product for one customer. So, we had this one 286 physical host that was cared for.
chypsa@reddit
Naaah. I mean, somewhat. But honestly, working with some big companies and...the VMs are still very much there, just on someone else's computer.
cjcox4@reddit
True, just "old school". While "lift and shift" was "the way" to expensively get into "the cloud", strategies are shifting to temporal event based functions. Well, at least for "the modern" users. Mostly to save a ton of money of course.
I guess it depends on how old the big company's infrastructure is and how "stuck" it is with regards to legacy design. Would think that most have something in the works to move forward though (?).
I do see, that in some cases, that some companies (that lift and shifted to cloud) are "moving back" to their own "racks" because they can manage it better and at far less cost. Whatever works.
redditduhlikeyeah@reddit
I kind of hate virtualization tbh. I don’t know why. Living through the early VMware days maybe. Or the later ones.
jake04-20@reddit
The first time you patch and reboot production hosts in the middle of the business day, it hits like crack. I remember getting nervous like I was doing something I shouldn't be.
Since it's on topic, RIP VMware... The golden days of virtualization.
tremblane@reddit
I remember my first mid-workday production patching. Load balanced VMs. Zero service interruption. It was when I came up with the phrase "ninja patching".
worthing0101@reddit
Upgrades could be TERRIFYING back in the day. Something went wrong then and you were potentially starting from reinstalling the OS on one or more servers, reinstalling and configuring the app, upgrading them to the point they were at pre-upgrade, etc. and THEN trying the upgrade again and hoping for the fucking best. This was back when the OS came on a stack of floppy disks as well so nothing went quickly. And no internet so support was from the vendor and only the vendor. (Or whatever knowledge you had saved somewhere, usually in a few notebooks full of notes, printouts, etc.) I think people take for granted just how valuable the internet is for finding solutions to problems.
jake04-20@reddit
Big time. Hypervisors are awesome for that. You just need to figure out the drivers and hardware stuff on the individual hosts, then every guest OS after just works. HV's are great at abstracting the compatibility. Doesn't matter if Windows XP won't run on the bare metal hardware, it will run if you put a layer of virtualization in between (not sure why you'd want to, but you can!). No fiddling with drivers on individual VMs (for the most part). Also the fact that you can reboot VMs in a matter of seconds vs. a bare metal host that has to go through initializing all the hardware for 5+ mins. Live migrations, snapshots, full VM backups. They just make too much damn sense not to use.
vectravl400@reddit
We've adopted a practice of asking for forgiveness instead of permission for virtualization, especially from OT vendors. Most of them will say no because they haven't tested it, not because it won't work. The only things we haven't virtualized are systems with funky controller cards. It's been great for getting computers out of harsh manufacturing environments and replacing them with thin clients.
LargeMerican@reddit
Doesn't it?!
itguy9013@reddit
PoE is still the thing that all these years later feels like magic to me.
Need to power your phone/AP/Random Device? Plug it into your switch and watch it power on.
Excellent_Pilot_2969@reddit
What's this? A Microsoft advertisement? Or the 'testimonial' of an intern :)
wtf_com@reddit
Love working with hardware; certain physicality that makes it enjoyment for me (once all the kinks are worked out).
jake04-20@reddit
The beauty is you still touch the hardware when you rack and stack a new VM cluster. In some ways, more so since you have to make sure all the redundancy and HA is up to snuff.
wtf_com@reddit
For sure - never wanna go away from virtualization but racking new hosts is amazing
Confusias1@reddit
Nothing more fun than spinning up a new VM cluster. This, 100%, is why we do what we do...
FlyingBishop@reddit
okay, I know everything used to be harder, but adding a Linux VM used to deactivate your Windows on the host?
GoogleDrummer@reddit
No, OP is talking about two separate things.
dude_named_will@reddit
And let's not forget snapshots (seriously please don't). Makes upgrades and major modifications way less stressful knowing I have a "reset" button.
UMustBeNooHere@reddit
I agree. I just love how easy it is to move VMs from one host/storage array to another and upgrade hardware with zero downtime. And snapshots/checkpoints prior to changes? [chefs kiss]
Anonymity_Is_Good@reddit
You really want to break your brain, spend some time learning how to correlate a running VM to a process on the host.
marmarama@reddit
It's 20 years since I first saw seamless live migration when I got my hands on an ESX cluster. It was magic then, and it's magic now.
But it's maybe slightly lesser magic now.
It does make me wonder why this kind of magic isn't more common in other areas of computing.
Why can't I just checkpoint all of userspace - or maybe just my user session - and seamlessly migrate it to another computer? There are plenty of things that make it look a bit like that's happening, but they're all fakes and workarounds. Plan 9 had something like that back in the day, but none of that ever went mainstream.
tecedu@reddit
I mean domain joined windows with onedrive gets pretty far tbh. Thats how Im setup across my various machine, rarely notice any diffferences
meostro@reddit
vMotion is the most impressive thing I've ever seen a computer do.
I'm amazed by Claude Code and whatever ML and AI and GPT stuff has come out in the past few years, but moving a running system between physical servers is still the most impressive. I remember the over-the-top marketing video from a lonnnnnnng while ago where they had 2 physical servers (A and B) running vCenter (or whatever it was called at that point) and had something like video streaming running in virtualization. At that point it was probably RealVideo RTSP. They ran the migration from A to B, then blew up A one second later. The video didn't even pause, it kept going without a hitch.
IRL vs marketing, I remember seeing one ping packet drop during a switchover, but other than that there's no indication that anything fucky happened. Absolutely magic.
thrwaway75132@reddit
With HCX I was able to vMotion a system from Ashburn to Las Vegas. I thought that was pretty cool.
thrwaway75132@reddit
The company I worked for at the time had been using GSX, and was in the Elastic Sky beta program.
The first time we did a vMotion in our lab we were jumping up and down.
vMotion and then DRS made everything easy. Need more capacity add a host. About ESX 4.1 we displaced all our physical compute except for giant databases.
In 2012 with 5.1 we virtualized the big databases, adding a class of host with 4 processors and 1 or 1.5 TB of ram for high performance workloads. We had some oracle boxes that were 1:1 with a host just to take advantage of SRM.
Then we added NSX for the DFW and started segmenting between zones all inside the same /22.
There was a lot of magic that came out of VMware.
1z1z2x2x3c3c4v4v@reddit
I loved GSX and installed it everywhere I went back then...
stevemk14ebr2@reddit
Because it's really hard to do lol
DropTheBeatAndTheBas@reddit
i mean you can migrate entire exchange/sharepoint with migration wiz at a click of a button now
Poops_backwards@reddit
When did you place that order and how long were the shipping delays? While I had a ship date at one point in time that has come and gone without a new anticipated date available.
MediumFIRE@reddit (OP)
December 25
pc_load_letter_in_SD@reddit
I do enjoy working with HyperV and Failover cluster but I just wish MS got something better than WAC.
Give me WAC but in an MMC!
(I know System Center VMM)
farva_06@reddit
WAC is WACK.
I had an instance where WAC was showing all cluster volumes offline, but failover cluster manager showed everything healthy as did all powershell cmdlets. Finally found out there was a pending reboot on one of the nodes. Not sure why that was causing volumes to show offline, but after reboot it was all good. Also, the thing is just a laggy mess.
pc_load_letter_in_SD@reddit
Yup, the lag is a nightmare. I can get cluster details in cluster manager in about 1\4 the time it takes my coworkers to get it from WAC.
MediumFIRE@reddit (OP)
Hand up, I still use Failover Cluster Manager in the MMC
pc_load_letter_in_SD@reddit
Yup! As do I !
biffbobfred@reddit
I wanna test Tahoe. Lemme spin up a VM and try it
sieb@reddit
I still have vendors tell me they don't support their app being virtual and insist on physical hardware, and go on a rant about needing this RAID and that striping etc.... Yeah sure, OK buddy.. proceeds to spin up VM in Nutanix cluster
They didn't even notice when they did the software install.....
sobrique@reddit
In 20 years I've a couple of cases where I've 'needed' a bare metal install.
But only a couple. IIRC back in the day SAP used to fall apart due to absurd numbers of context switches making your virtualisation layer really unhappy, so it was just easier to dedicate tin to it.
InfiniteTank6409@reddit
HPC still works better with physical if you use infiniband but you can definetly do VMS or at least containers, and it's bad only of you have a job running multiple nodes (like 512 core job)
natefrogg1@reddit
Yeah, we went though that recently and they wanted a full 42u rack full of systems, the whole stack is running happily on a 5 year old 2u server under 750watts
xixi2@reddit
What the heck software wanted an entire dedicated rack of servers?
natefrogg1@reddit
An ERP system that wants separate servers for pretty much every sub process, the upgrade we wanted to do doubled the amount of task servers required and according to them separate servers ensures load reliability and proper segmentation. If all goes well we will be able to turn it off soon
InfiniteTank6409@reddit
No VMS? No docker? They wanted Physical servers???? 42 u how many cores are nowadays with AMD? Like 4 nodes every 2U so 84 nodes 168 socks of like 128 cores = more than ten thousand cores for an ERP? Whats happening here
le_suck@reddit
that deserves a name and shame.
Tetha@reddit
Some time ago, I adjusted our on-prem sizing to basically say: "Based on user count, assign x/y/z vCores, x/y/z GB of memory and x/y/z GB of disk. Monitor and scale CPU and disk as necessary and contact support if either seems silly or out-of-memory crashes occur"
It was a funny social experiment while in that place. A lot of admins were really happy with this. Small footprint, easy scaling, give it more stuff when it needs and reboot, business as usual. They usually got a few hints on monitoring the solution too, and then you'd never hear from them again for years, only for updates.
Some admins were like "But how do you monitor the CPU usage?". Those were the interesting contracts.
Sudain@reddit
Speaking as a voip provider, there are cases where bare metal is required. But for most cases, your right.
Confusias1@reddit
VOIP border gateways. Voodoo magic, best to do bare metal...
jake04-20@reddit
Anyone else ever have those hour long meetings where 2 mins in, the vendor asks you to share your screen and take control so they can "drive" the new product implementation? We had one vendor that operated like that. I had a VM so I could multi task on my "host" computer while they fucked around in the VM lol. Worked great.
sieb@reddit
I keep a stripped down Windows VM just for vendor/support calls like that, mainly to keep them away from my stuff they don't need to see.
Sudain@reddit
Had that happen last year for a full month, 40 hr weeks. It was painful since they had no idea what they were doing.
jake04-20@reddit
A vendor not having any idea what they're doing!? Say it ain't so! That was this vendor too lol. Incredibly painful watching them fumble around, insult to injury that I had work that needed to get done and my computer was held hostage for the entirety of the meeting. Those meetings always ran over the scheduled time too. The VM thing was essential to keep my sanity in check lol.
Sudain@reddit
:D You get it! :D
I wish I would've been smart like you and did the VM. I had to wait for a month before my boss told me to not come to the meeting's he'd have his PC clobbered.
root-node@reddit
I had a job interview a very long time ago where I was talking about a project I was working on.
I said that I spun up a server and did a demo to some test users.
The interviewer asked how I managed to get capex to buy a server just for a quick demo.
No no, I spun up a small VM and deleted it afterwards.
How times have changed
shadeland@reddit
It was 2005/2006 when AMD and Intel released processors with the virtualization extensions that allowed VMs to run at near bare metal speeds.
I don't think anything was adopted as quickly as virtualization. I saw data centers go from zero production virtualization in 2006 to at least 50% in 2010, and by now it's probably close to 90%
ProgressBartender@reddit
Remember having rack upon rack full of servers and their local storage arrays?
shikkonin@reddit
Why on earth would you expect more breakage from Linux than from Windows?
stashtv@reddit
There was a time (insert grey beard here) when attempting to migrate VMs was a bit of a crapshoot (even on same OS). Migrating to a new server OS with a VM was practically unheard of. Add-in changing the CPUs? That was also a 100% non starter not too long ago.
shikkonin@reddit
Live Migration, yes. But moving a Linux install between machines was a non-issue even 20 years ago. Pull disks, insert disks, boot.
rollingviolation@reddit
I dunno, I've seen plenty of
LIOLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
or whatever it would display when LILO would barf when you'd try to clone from IDE to SATA or something like that.
No, I didn't know what I was doing at the time. Now I do, so I accidentally erased the BTRFS root on my laptop.
fresh-dork@reddit
i ran into one problem with plex. as it turns out, my hypervisor defaulted to a generic cpu, so plex would blow up with an opcode error trying to use SSE type instructions
MediumFIRE@reddit (OP)
Early days of Hyper-V (2012 / 2016) running a Linux VM was a little dodgy before Microsoft fully embraced it. I worked, but I remember needing to setup some VM settings in a specific way. Anyway, that's my point. Even a moron like me can migrate without googling obscure error messages.
DheeradjS@reddit
Experience. They break about as often.
Devs gonna Dev. Doesn't matter if they write their software for Windows for Linux.
Erok2112@reddit
Friggen live migrations are magic too. Need to do a host upgrade/update? Migrate running VMs to a new host, do the needful and then migrate back. Or not. And no one notices the changes at all.
mpdscb@reddit
I remember when esx first became a thing and we started virtualizing all our old hardware systems and they just worked. It was magic. When I think of all the stuff we used to have to do to get stuff to work when we moved or upgraded hardware in the past, it makes my head spin now.
shaded_in_dover@reddit
The real magic is a Veeam VMware to HyperV live recovery. Where static IPs move because the NIC MAC address also carried over, all the activations work … everything. Made my exodus from VMWare much easier than it rightfully should have been.
JediCow@reddit
Just finished this migration as well and couldn't believe how easy it was.
ThePixelHunter@reddit
Virtualization has ruined me to the point where I hate dealing with bare metal. There's nothing worse than troubleshooting hardware problems. Heck I don't even want to rackmount it... just let me at the terminal!
CaptainZippi@reddit
Concur - every time I vmotioned a VM to another host/cluster a part of me muttered “witchcraft” under my breath.
Sudain@reddit
Yup. Eventually the concerns you know to pay attention (because it matters) to will seem so antiquated as modem atdt codes.
DiseaseDeathDecay@reddit
Another anti-rant: god I love when people's app is down because we told them it should be virtualized and they got their VP to tell our VP they want bare metal.
There is nothing at work I enjoy more than telling someone, "If this was virtual, you wouldn't be down right now." when we're working with them to get things online after hours or over the weekend.
Close second is, "I can't increase your disk space because you wanted local disks when we told you SAN was better, and you wanted physical when we told you virtual was better."
OpenGrainAxehandle@reddit
I remember struggling to move a VMWare vm to a new host back in 2007 or 2008. What a chore. Trying to select the correct NIC and get it talking... You're correct, we've come a long way
OmenVi@reddit
Fast forward 2 years, and I remember the first time I vMotioned a term server to another host. 40ish users, and nobody noticed. 2 packets dropped. It was glorious.
rosseloh@reddit
In my homelab I'm using LXC containers and even as someone reasonably familiar with full-fat hypervisors/VMs from my career, it absolutely feels like magic to just "oh spin up another container, open the terminal, welp just feels like I'm sitting on the host system doing things" (ignoring permissions issues, that took a little learning).
bloodguard@reddit
You should listen to the real grey beards go on about IBM z/VM. Running on big iron and serving up VMs since the 70s.
kizzlebizz@reddit
I love explaining my job to folks.
"I run computers that don't know they're not computers"