Windows Server 2019/2022 upgrading to 2025 - any way to roll back?

[-]

HerfDog58@reddit

Not using any patch management at my workplace (LONG story...) but any of the 2019/2022 server I manage showed the 2025 upgrade as available when I checked yesterday. However it did NOT autoinstall on them - if I wanted to initiate the upgrade process, I had to click on the "Download and Install" link. When I did that, I got the warning that I needed to have backups to ensure that I wouldn't lose access to data, as I would NOT be able to do a rollback upon completion of the upgrade. The warning also stated I needed to purchase license keys to activate the upgrade once installed.

So I cancelled it.

Checked them just a moment ago, NOT showing the 2025 install as an available upgrade...maybe MS changed their mind?

[-]

Toby_7243@reddit (OP)

That's interesting... It almost sounds like in-place feature upgrades are a planned feature of Windows Server now.

But if I need to buy a key, why even offer it via Windows Update when I could instead buy the retail media and manually upgrade when I have a key in-hand? I guess their logic is I'm gonna prefer to spend a few quid on a new license as opposed to taking a production server down for hours or days at a time because I didn't read the message about needing a new key... Shady of Microsoft if this theory is correct.

[-]

HerfDog58@reddit

I can spin up a new VM from a template and join it to my domain in 5 minutes. I update my templates a couple times a year to refresh updates and features.

I wouldn't ever do an in place version upgrade. But I'm in Walt Kowalski mode these days...

[-]

nVME_manUY@reddit

How do you keep an updated OS for a legacy application?

[-]

ITStril@reddit

Same for me - upgrade notification did disappeat

[-]

newtekie1@reddit

Wait, so MS is just giving away free upgrades to Server 2025? And this is a bad thing how?

[-]

WaldoOU812@reddit

Regardless of licensing costs, you never want to upgrade a server blindly without ensuring that whatever application(s) run on that server won't just stop working. Also, you really should be testing every new OS in your environment thoroughly before just rolling it out.

I'd also point out that it's not a great practice to upgrade servers at all. Where possible, you should be building an entirely new server and replacing the old one.

[-]

fireandbass@reddit

This may have been true 10-20 years ago, but it is outdated advice today.

[-]

Odd-Pickle1314@reddit

Feel free to tell the software vendors who refuse to support underlying components like Windows or SQL released 2-5 years ago this is the case.

[-]

joefleisch@reddit

No in place upgrades of OS is still current advice on some application servers like Microsoft Exchange.

Everywhere else it is up to the Sysadmin team.

When we upgraded VMs from Windows Server 2012 R2 to Windows Server 2022 we changed MBR to GPT, turned on Secureboot, and added vTPMs. Building new and transferring workloads can help refresh best practices for security.

[-]

WaldoOU812@reddit

I've been bitten in the ass way too many times to upgrade in place I replace servers today as opposed to upgrading as I feel there are too many unknowns with most applications. Upgrading an OS leaves you with no way to roll back in the event some weird legacy issue bricks your server or hoses your application.

I'd be curious what your justification is for upgrading and why you feel the extra bit of caution is unwarranted?

[-]

fireandbass@reddit

I verify a backup and also take a snapshot before an in place upgrade. If there's a problem, just restore the snapshot. I used to agree with you, but in place upgrades have been really solid since Server 2019. I wouldn't IPU a DC or Exchange server but pretty much anything else is fair game.

My environment had hundreds of servers below 2016 and IPU have been successful on about 95% of them so far, its still a work in progress. But setting up a new server and migrating is a huge PITA when there are third party vendors involved and then you have to juggle DNS and hostnames and Ip addresses and a lot of other stuff. With an IPU no DNS changes are needed, no third party vendors needed. Imagine setting up 100 new servers and 100 new IPs and 100 new DNS entries and an in place upgrade starts to sound like a good idea. It would literally take a year longer than in place upgrades. It's supported officially, so why not.

[-]

WaldoOU812@reddit

Those are actually some really good points. How long do you keep the snapshots in place?

[-]

fireandbass@reddit

I verify functionality, and a script automatically deletes snapshots after a day or 2.

[-]

WaldoOU812@reddit

I'm going to bring that up with my team, and see what they think. I would think we should be able to try this out with a server or two.

What about SQL? How do you handle SQL upgrades?

[-]

fireandbass@reddit

There's a dedicated SQL admin, so I don't do those.

[-]

BlackV@reddit

VMs are the best

[-]

Unable-Entrance3110@reddit

This is the way.

[-]

BlackV@reddit

"depends" with an in-place upgrade (as a rough example) you don't get the updated defaults for things like security value x or tls levels or similar, you keep whats configured in you existing system, loosing a small benefit that a new install would give you

I prefer new build, but I'm perfectly happy with in-place, especially if one is 1 hours work and one is 4 hours work

[-]

shadyman777@reddit

Not free... from my understanding once get hit with the update you need to activate it with a license.

[-]

jake04-20@reddit

Don't forget user CALs.

[-]

augur_seer@reddit

dont worry, i do. havent bought CALs since 2016 and wont be going forward. My Company opperates in Can and EU, where CALs have been determined to be unlawful.

[-]

zyeus-guy@reddit

This is really interesting and News to me. Do you have a source for this? I’m amazed this hasn’t been bigger news in the EU.

[-]

jake04-20@reddit

That's nice. User CAL expenses add up.

[-]

SkullRunner@reddit

So production server license extortion, to be more accurate.

[-]

DoctorOctagonapus@reddit

The Ferengi would be proud.

[-]

1Original1@reddit

The upgrade is free - you just need to license the OS and CALs after - and risk data loss if you don't

This is the greatest christmas bonus guarantee scheme i've ever seen

[-]

_Frank-Lucas_@reddit

This part I don’t understand. I thought the 2019/2022 KMS keys were the same and that they would activate 2025. I have software assurance whatever it is but no new keys in VLSC.

[-]

dustojnikhummer@reddit

Key =/= license

[-]

anxiousinfotech@reddit

"Just grab a key off MSDN"

[-]

Unable-Entrance3110@reddit

It's like that overly gregarious person at the checkout who says something like "I'll give you the for free, but the service will cost you !"

Hilarious.

[-]

RiceeeChrispies@reddit

Enjoy getting vendor support for any of your LOB apps if you’re running Server 2025, that is a decent get out of jail free card for them.

[-]

Toby_7243@reddit (OP)

They are... If you don't mind the server being unlicensed.

[-]

alexschomb@reddit

Free upgrade process, but you'll need a new non-free license to keep using your automatically upgraded server

[-]

Engineered_Tech@reddit

I have some bad news to add to this. The update to Windows Server 2025 is being offered to Windows Server 2016 and 2019 as well.

Careful where you click people.

[-]

hulahoop97@reddit

2016 for sure???

[-]

Toby_7243@reddit (OP)

If it's affecting 2016 then I'm gonna roll everyone back to 2008 R2! 🤣

[-]

bigkahuna1986@reddit

My 2k3 installation is safe then.

[-]

awit7317@reddit

Your 2k3 installation is still so much better than 2016

[-]

icebreaker374@reddit

Fuck it go to Server 2K.

[-]

awit7317@reddit

The wily veteran of the 2k space. OG.

[-]

FalconDriver85@reddit

NT 3.51 or bust

[-]

awit7317@reddit

Rock solid SQL Server platform

[-]

anxiousinfotech@reddit

Hey, I just decommissioned that!
(I mean, it was being run by a company we acquired, and got axed ASAP, but still)

[-]

CeeMX@reddit

That was actually a wonderful edition, still miss it

[-]

Stuffer007@reddit

And 2012r2

[-]

patjuh112@reddit

Trying to still grasp this, I run a variety of clouds/services from 2012 upto 2022's but none of them are offering this update. Also not seeing it in Azure Update Manager being pushed?

[-]

Toby_7243@reddit (OP)

It looks like it may be affecting people using 3rd party patch management solutions only.

Microsoft have miscategorised the update and 3rd party patch management tools are pushing essentially a feature update/enablement package automatically to these servers.

I guess because Microsoft approve the updates for WSUS/Azure Update Manager (and most likely don't rely on an API and some predefined logic) they haven't pushed this to servers which shouldn't have it?

[-]

patjuh112@reddit

Thanks for the clarify

[-]

Healthy-Poetry6415@reddit

You guys are not helping the license sales division of Microsoft with your technical discussions.

Please do the needful and just pay for new licenses - Love, Microsofts Shareholders

[-]

kyoukidotexe@reddit

No

[-]

Dizzy_Bridge_794@reddit

Hope it doesn’t auto update any domain controllers without extending the schema.

[-]

grimson73@reddit

This is why Exchange CU updates never gets offered through windows update. As I understand schema updates require permissions which the local system account lacks on member servers. But I guess the local system account on a dc does have or might have permissions to update the dc schema.

[-]

jake04-20@reddit

Does the registry approach for locking OS versions for windows clients work for windows server? I believe it's HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate that I'm imagining.

[-]

krodders@reddit

This worked for us but I'm not sure if I guessed the correct registry settings. Whatever, the prompt to install has gone. 3.5k servers. Check my most recent comment

[-]

anxiousinfotech@reddit

Yup, that blocked the prompt on our servers as well.

[-]

RiceeeChrispies@reddit

Don’t give me heart palpitations like that.

Imagine if it auto-updates your RDSH but not the broker.

[-]

Dizzy_Bridge_794@reddit

That wouldn’t be good either.

[-]

philrandal@reddit

What patch management tool are you using. It looks like that is to blame rather than anything else.

[-]

ISeeDeadPackets@reddit

Who gives a crap whose fault it is? Are you here to feel superior or provide some assistance to OP?

OP there is no rollback option for in-place server upgrades that happen via this patch. You'll have to either revert to a backup of some sort or stand up a new VM and migrate the workload.

[-]

philrandal@reddit

Got out of the wrong side of the bed today, did you?

If this is only affecting some patch management systems and not all, the more information shared about what affected people are using, the better.

[-]

ISeeDeadPackets@reddit

It's not the patch management systems fault for deploying a patch Microsoft issued according to the policy you configured in the tool. Patch management systems push out patches, it's what they do, it's the operators job to configure them in a way that meets their risk tolerance whether that's pushing straight to prod on day one, observing a waiting period, or using a test group.

Your "answer" wasn't an answer, it was just a snarky dickish reply that helped the OP in exactly zero ways. I got out of bed on the same side as I always do, the side that doesn't feel some false sense of superiority when someone else falls victim to an issue they were unaware of.

[-]

philrandal@reddit

It does matter. WSUS didn't deliver it to our 2022 servers.

Take your hostility elsewhere.

[-]

ISeeDeadPackets@reddit

Well congratulations to you. I didn't get bit by this either, didn't stop me from just providing the answer to OP instead of accusing them of being bad at their job.

[-]

YnysYBarri@reddit

Slight detour but, whoever's fault it was, this is completely inexcusable from Heimdal:

I had a gutter level view of Heimdal before today, but this is off the scale. You don't disable automatic updates, you manage them. This is barking back to the bad old macho days of "my server has an uptime of 2.3 squilion years!"

[-]

ISeeDeadPackets@reddit

Yeah, I feel bad for the folks that had an unpleasant wake up this morning, but I'm also shaking my head at the number of people who patch without even a waiting period let alone testing.

[-]

YnysYBarri@reddit

And this makes Heimdal look even worse. An hour ago they had a blog post that contained a few digs at Microsoft... And now it's a 404.

[-]

YnysYBarri@reddit

[-]

YnysYBarri@reddit

[-]

TheCopernicus@reddit

I wonder if it is dependent on having Windows 11 24H2 computers as that is what the KB is marked for. I don’t see it as needed in my WSUS either but we don’t have any Win11 24H2 computers yet.

[-]

Antiapplekid239@reddit

Agreed Hostility is not helping the cause at all

[-]

Toby_7243@reddit (OP)

I thought this would be the case, but thought it'd be worth asking the collective minds of Reddit in case I had missed something stupidly obvious.

I was spinning up a test server to try and get it to upgrade to see if uninstalling the patch would work (as it is showing in Windows Update as uninstallable) but my test server wasn't playing ball.

We have backups of both servers so I'm not worried. It's just we're an MSP so trying to get the servers restored takes some coordination with the customer.

[-]

ISeeDeadPackets@reddit

Yeah it's never fun when it's your own hardware. Hopefully they're machines you can just revert without worrying about migrating any data out of the current into the restore, but I never get that lucky!

[-]

ZealousidealTurn2211@reddit

The patch management tool in question is relevant to helping others identify the vector that caused the issue and therefore prevent it. It's troubleshooting, not blame direction.

[-]

xangbar@reddit

Yesterday I saw it was Heimdal that people had the most issue with. The thread I saw several people all identified they had as their patch management solution where servers upgraded to 2025. Not sure if any others were in the mix or not.

[-]

philrandal@reddit

The Register has this piece on the issue: https://www.theregister.com/2024/11/06/windows_server_2025_surprise/

[-]

SadMadNewb@reddit

That seems wrong? It says Microsoft has the wrong guid etc, but it doesn't.

[-]

anonaccountphoto@reddit

MS misclassified the update as a security update.

[-]

fireandbass@reddit

Wrong.

[-]

anonaccountphoto@reddit

No, correct.

https://old.reddit.com/r/sysadmin/comments/1gk2qdu/windows_2022_servers_unexpectedly_upgrading_to/lvl4of4/?context=3

[-]

Secret_Account07@reddit

Kinda.

From my understanding the MS update API marked the update incorrectly. So not really vendors fault.

[-]

fireandbass@reddit

People are saying this, but I think its bad info. There are 2 KBs with the same number, which isnt anything new. One is classified in WSUS as a Security Update, the other is an Upgrade. It sounds like the update systems that were upgraded approved both the Security Update KB and the Upgrade KB.

[-]

roll_for_initiative_@reddit

Which RMM do you use? We use nsight by nable and I've been watching but don't see this KB listed as available yet.

[-]

Toby_7243@reddit (OP)

We use Ninja - not sure if they've built their own patch management system or leverage another provider's. I think their app patch management system is 3rd party but integrated.

[-]

ithium@reddit

Ninja had a message this morning about this on our dashboard. We have all OS patches in approval so I promptly rejected it. It was listed as being available to deploy. So go block it!

[-]

ChrisDnz82@reddit

It didn't impact us, however we temp blocked the patch to be sure but will likely release it again. I don't believe this is a MSFT issue as this is what they have been doing for some time. IMHO these tools Installed a Feature Update, which are now capable of updating full OS versions as we now see with Win 10 to 11. The confusion is they are looking at the KB number and that KB number is used for both the CU (security update) and th FU (Upgrade). From what i gather Ninja werent affected either and nor were some other patch tools. Considering between us we manage in to the 10s of millions of devices we would have seen this if it were indeed only a MSFT issue with the metadata / patch info

[-]

TheVillage1D10T@reddit

We’re still using WSUS at the moment and don’t have anything configured for auto-approval. Should I be on the lookout for this behavior?

We’re a gov. shop so I just do what I’m told, and use what they want me to in regard to patch management. Please don’t come at me for it lol.

[-]

marcorr@reddit

Well, looks like roll back is not an option. I would restore from the backups.

[-]

Apprehensive_Bat_980@reddit

Is this happening on DataCentre licensed servers?

[-]

Toby_7243@reddit (OP)

We don't have any Datacentre servers in the wild but I can't see why it wouldn't as they're the same updates regardless of SKU.

[-]

Apprehensive_Bat_980@reddit

Costly request for 2025 budgets requesting new DataCentre licences! What did you end up doing?

[-]

trail-g62Bim@reddit

I don't see KB5044284 in SCCM. Anyone else see it?

[-]

zonuendan16@reddit

I did this on our servers reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate /f /v DisableOSUpgrade /t REG_DWORD /d 1 reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate /f /v ProductVersion /t REG_SZ /d 10 reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\OSUpgrade /f /v AllowOSUpgrade /t REG_DWORD /d 0 reg add HKLM\SOFTWARE\Policies\Microsoft\WindowsStore /f /v DisableOSUpgrade /t REG_DWORD /d 1 reg add HKLM\SYSTEM\Setup\UpgradeNotification /f /v UpgradeAvailable /t REG_DWORD /d 0 reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate /f /v TargetReleaseVersion /t REG_DWORD /d 1 reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate /f /v TargetReleaseVersionInfo /t REG_SZ /d "21H2"

[-]

Sroundez@reddit

https://tachytelic.net/2019/01/group-policy-defer-windows-updates/

Just apply these GPOs and don't worry about this issue..

[-]

VexedTruly@reddit

So at this point is there no registry key we can push to block this? No-one with half a brain cell is going to accept an OS upgrade over Windows Update!?? I can’t believe this is something I need to even think about.

[-]

krodders@reddit

Check my most recent comments

[-]

Vel-Crow@reddit

Rapid rollback has been the best for me - we are a datto shop.

[-]

amiralen@reddit

Do.you guys handle patch management with datto as well?

[-]

Vel-Crow@reddit

No, NinjaOne RMM. Ninja put up a banner, with instructions to block the update. We only had 2 devices hit, we got kinda lucky woth the timing.

[-]

Toby_7243@reddit (OP)

This is our situation, we use Ninja for patch management and I've rejected the patch as per the banner and again we've only had 2 servers hit from what I can see.

[-]

Vel-Crow@reddit

You should check the other server with a custom field and script. The update does not seem to show the proper OS in ninja. So the devices on 2025 still show 2019/2022 in ninja.

If you make a Role Custom Field called 2025Installed and assign it to the server role, you can deploy the below script to your servers, and then use the "Devices" utility to view all devices that have this patch. You can also go to the main dashboard, click patching, pending/approved, and search the KB. But tbh, I though this was fast and quicker to reference, rather then sending CSVs around the team.

$updates = Get-HotFix | Select-Object HotFixID

$TRASH = "KB5044284"

Ninja-Property-Set 2025Installed "False"

foreach ($update in $updates){

$update = $update.HotFixID

if ($update -eq $trash){

Ninja-Property-Set 2025Installed "True"

break

}

[-]

Toby_7243@reddit (OP)

Absolute legend. Thank you for this. I'll look to get this set up tomorrow when I'm in the office. I've set up a few custom fields before so shouldn't be too difficult for me!

[-]

Sneeuwvlok@reddit

Does anybody know if datto rmm patch management also has this issue?

We have excluded the KB but don’t know if datto rmm patch management also has the same problems.

[-]