Production server black screen nightmare - In recent VMTools upgrade of 13.0.10.0
Posted by Enough_Return_5261@reddit | sysadmin | View on Reddit | 15 comments
I tried upgrading one of our production servers from 12.4.5 to 13.0.10.0 yesterday, and it was a total nightmare. After the reboot, I just got a black screen, and once I finally got back in, VMware Tools was completely gone, i can't even move my mouse, fk microsoft.
Luckily, I took a snapshot before starting. I eventually managed to get it installed successfully by using the vCenter HTTP Console. Just a heads-up for anyone else planning this upgrade—be careful and definitely make sure you have a backup or snapshot ready!
AdInevitable8483@reddit
Move to Linux
techvet83@reddit
What OS version is on the server?
Stonewalled9999@reddit
I told our NOC I’d upgrade the tools as you need some prerequisites that you can see in the gui. Blindly clicking the update in VUM can remove them and the reinidction doesn’t always work
St0nywall@reddit
Why are you blaming Microsoft for a driver pack from another company?
If you're going to blame someone indiscriminately then how about blaming yourself for whatever you did to that server VM that cause the issue to happen.
I bet you won't like taking the blame for something like that, and neither should any other company or person for something they didn't cause.
Especially when you couldn't be bothered to do rudimentary root cause analysis.
When your boss asks you "will this issue happen again?" on the next maintenance cycle, you can confidently say "I don't know".
Enough_Return_5261@reddit (OP)
At downtime period, it just can't find the root cause, that's why it is the follow-up step now currently working on it, i suspected the Remote Account working on VM tools upgrade would cause the link down, but it is difficult to explain the cause of the display driver corruption for specific VM under the same ESX, things just happen.
And why i am blaming at Microsoft? It is emotional acting, just a joke.
So if my Boss or client ask this question, i can just explain the obvious fact, that's why this case is very tricky to me.
superb3113@reddit
This is interesting. I've installed 13.0.10.0 using both the in-Windows .exe, and via the Updates tab in vCenter. This was on Windows 2012 R2, 2016, 2019, and 2022. Haven't encountered that issue. Did the test environment first, then production a few weeks later once no issues show up. What was your first method of installing and the operating system?
My rule of thumb: if you have to go from v12 to v13, definitely do snapshots/backup. My understanding is that v12 is now considered "legacy".
Enough_Return_5261@reddit (OP)
this is the fun part, dev servers are successful, it was really difficult to explain why it is only happen on my production, i can only say the VM under the same ESX all have this problem, but it is still unexplainable why this happen.
crystalbruise@reddit
That snapshot probably saved your day. I’ve learned to treat VMware Tools upgrades like firmware updates now, even when they seem routine. Testing on a non critical VM first and having console access ready has saved me more than once from exactly this kind of mess.
tarvijron@reddit
Did it do the "sets every color variable in appearance to #FFFFFF" black screen bug or was it literally not displaying on a default vesa driver?
Enough_Return_5261@reddit (OP)
i really don't have time to study the root causes.
BoltActionRifleman@reddit
This sounds like us with VMWare bullshit. Don’t have time to find the root causes, and likely wouldn’t find it anyway, so we just get it running again and stay a few versions behind on VMWare Tools.
alpha417@reddit
Did you have the same issues when you upgraded your dev servers in the same manner?
Enough_Return_5261@reddit (OP)
this is the fun part, dev servers are successful, it was really difficult to explain why it is only happen on my production, i can only say the VM under the same ESX all have this problem.
LeeFrann@reddit
Sounds like a failed install since it resulted in the cm having no drivers. Did you do patch via lifecycle manager/esx or a 3rd party tool (sccm/big fix)?
Enough_Return_5261@reddit (OP)
no, i mount the ISO on the Windows to upgrade, but i do have using the CyberArk remote account to upgrade.