Fell asleep to Windows Server 2022, woke up on 2025.
Posted by JoeyFromMoonway@reddit | sysadmin | View on Reddit | 200 comments
I just got a nice Zabbix Warning - "Operating system description has changed" - and thought, okay, might be a Ubuntu update, had that before. No big deal.
But no, 2022 updated to 2025. On 14 VMs. Unwanted.
I mean, i am going to roll back via backup, but... why even? How? Where did i go wrong?
I am second guessing all my life choices now.
chuckbales@reddit
Search around for it, you're not the only one
https://www.reddit.com/r/sysadmin/comments/1gk2qdu/windows_2022_servers_unexpectedly_upgrading_to/
https://www.reddit.com/r/sysadmin/comments/1gkz4ot/windows_server_20192022_upgrading_to_2025_any_way/
JoeyFromMoonway@reddit (OP)
Thanks - just saw it after posting.
This is catastrophically bad. I am out of compliance without doing anything wrong. Just.. i do not even know what to say about this.
Capable_Tea_001@reddit
I'll say here what I said on another thread.... Never auto approve patches to your production environment.
Auto approve them to a test environment sure, but don't trust any vendors.
Microsoft and Crowdstrike have a lot to answer for.
thenebular@reddit
Exactly. Just because it's labeled a security update doesn't mean it won't seriously mess things up. Update to test environment and give it a week before even thinking about prod.
BobRepairSvc1945@reddit
You realize that 75-90% of businesses have no test environment?
Deadpool2715@reddit
Everyone has a test environment, some people are just lucky enough to have a separate production environment
nmincone@reddit
đ
Capable_Tea_001@reddit
That's a made up statistic, however probable.
That's also a choice.. As a sysadmin you have influence as to the strategic and operational decisions around this.
Grizzalbee@reddit
Everyone has a test environment; not everyone is fortunate enough to have a separate production one.
TxJprs@reddit
That was funny. Thank you.
Zortrax_br@reddit
Test environment can at least be some non critical servers if you don't have many resources.
what-the-puck@reddit
If a company doesn't and can't have any type of test environment whatsoever, then they'd be wise to be N-1. Or more specifically, to set their patch auto-approval and auto-install to a delay of a couple weeks or whatever.
BobRepairSvc1945@reddit
Most definitely.
MorpH2k@reddit
Might be so, but this is one of the many reasons why they should stand up a test environment. Even just one or two machines that are similar to some of the core prod servers would be better than nothing.
My last SysAdmin job had the whole nine yards, just about every server with something even remotely close to important had at least an identical test server for every prod server. Some had three or even four. This was a very special case though and certainly not the norm, but still. Testing on the Windows side would get the updates a few days after patch Tuesday, prod usually hot updated the week after if nothing broke.
The point is that having even just one test machine for every OS version you're running that you update before you do anything to prod would let you catch stuff like this before you lose and have to recover your whole your production environment.
PURRING_SILENCER@reddit
My management wants patches installed nearly the moment they are available to all production box. They also don't want to fund extra ftes to allow for the work required to test updates and push them to prod that fast.
Luckily I don't manage windows boxes
DiseaseDeathDecay@reddit
Then when it breaks things you have a wonderful excuse for why things are broken.
I mean, you have told them that waiting 3 days has basically no risk and will prevent outages, right?
PURRING_SILENCER@reddit
I mean, ultimately not my circus. But yes we originally had a week soak period for updates. New management came in and cited 'industry standards '.
hath0r@reddit
industry standards nearly brought down the world ....
ElfNeedsFoodBad@reddit
It's a risk acceptance question - risk of hack vs. risk of buggy patches. That is a management function to choose what risks to take, hopefully they were somewhat well informed?
RockSlice@reddit
My understanding is that it was labelled as KB5044284, which was an October update for Windows 11.
So if you approved that update for your environment last month, it would have pulled the new (mislabeled) update and installed it.
SlowGT@reddit
Thankfully our N-central instance only pushed this patch to Windows W/L machines due to how we set up our filters and device classes. No â25 upgrades for us this month.
jake04-20@reddit
Trying to recreate this now, this will be helpful. Thanks.
DrunkenGolfer@reddit
âTest environmentâ, lol.
TheFeelsNinja@reddit
We test in production, right guys?
Geek_Wandering@reddit
If course. How else are you going to be sure it will work in production if you don't test in production?
DrunkenGolfer@reddit
Every Friday afternoon.
dathar@reddit
Mosyle, that you?
Capable_Tea_001@reddit
That's a strategic and operational choice.
Choosing the right path can save you so much pain long term.
BeyondAeon@reddit
Everyone has a Test Environment, some are lucky enough to have a separate Prod Environment !
timbotheny26@reddit
Wait, what has Microsoft done recently? (Apologies for being OOTL, I'm not a sysadmin yet, but I plan to be at some point in the future and like spending time in this sub.)
Capable_Tea_001@reddit
Have you read the thread?
mycall@reddit
Minor vs. Major patches should have different approval workflows.
ghjm@reddit
What if you don't trust the vendor to correctly label whether a patch is major or minor?
mycall@reddit
I guess the question is if has breaking changes or not, but for sure, vendors like Microsoft do screw up.
ghjm@reddit
Right, which means you can't have different approval workflows, because if you don't trust the vendor, then you don't know if an update is major or minor until you do your own testing.
sonic10158@reddit
Microsoft: âwhatcha gonna do bro, switch to Linux?â
lev400@reddit
Fuck Microsoft
Sure_Acadia_8808@reddit
Spot on. Microsoft's shoddy work has normalized a very abnormal situation: entire industries one missed patch away from a catastrophic failure; zero-day exploits showing up with comical frequency; forcing multiple third-party products into the broken ecosystem to try to band-aid their systems' unconscionable defects; being unable and unwilling to internally address the defects, the need for RMM solutions, or basically any issue whatsoever.
The culture is broken, and it's big tech's fault. But it's also customers' fault for buying trash and rationalizing it.
changee_of_ways@reddit
I mean, if all the choices are bad, you gotta hold your nose and pick one.
zm1868179@reddit
This wasn't specifically Microsoft everywhere. I have seen that this happened. It was everybody using third-party patch management.
If you have or have stayed with Microsoft update tools, they did not Auto upgrade to 2025. It seems that a lot of third-party patch management software miscategorize the update and applied it.
Gawdsed@reddit
Running WSUS and verified, no problems here... That patch was set as a feature update though
zm1868179@reddit
Yeah, it seems like Microsoft's own tools classified it correctly, but whatever the third party patch management tools use to classify, it didn't seem to classify it correctly and pulled in a feature update as a security update.
Efficient_Ad_4162@reddit
I don't want to kick you while you're down, but crowdstrike really was the shot across the bow for this one.
andrewsmd87@reddit
Nothing, you say nothing but document what happened in some obscure place. You make sure it's generic like patching had unexpected upgrade, rolled back.
and then IF anyone every notices, you just say you followed process
MBILC@reddit
As noted, are you not giving a window before applying newly released patches, or applying to a test env first to make sure they do not break things (we know MS has a history of patches breaking their Server OS or AD functionality)
AwesomeGuyNamedMatt@reddit
What product clarification is this patch? My shop runs 2022 21h2 and we are not getting this kb. It's not in our declined list either (wsus). I just want to make sure it doesn't become approved.
cluberti@reddit
It would appear that something happens where some automated patching systems see (saw?) KBKB5044284 as applicable, as it appears that it is both the update that's behind the "upgrade to server 2025" link when you would search/seek for updates, and also the KB article number for the regular October CU for Win11 24H2. Thus, at least for the examples I've seen, some patching systems had downloaded it and automatically applied it to everything applicable - in one of the posts yesterday, someone (ostensibly) from Heimdal Security had confirmed that this is what happened to their customers.
It would seem there's some problems with this update for now, and it would be best to block it explicitly for anything not Win11 24H2 or already running Server 2025 anywhere you are able, if you use an automated patching solution that isn't Windows Update or Autopatch, at the very least.
BrentNewland@reddit
I read 5044281 was the one that shows the link, and 5044284 just installs Server 2025.
19610taw3@reddit
A coworker and I were just talking this the other day. Neither of us thought that Microsoft would actually force people to '25 ...
We were wrong
randomusername_42@reddit
Oh the poor innocent children.....
cluberti@reddit
Maybe that's it - my memory isn't as good as it once was, and it wasn't good then either ;).
Protholl@reddit
This was so wrong for MS to just spawn this upon the masses. It's also happened to companies running Windows 10 even with some of them trying to stop it. Welcome to the walled garden you don't own your OS, Microsoft owns you.
Jazzlike-Love-9882@reddit
WSUS has prevailed đ
Phyxiis@reddit
And they want to kill it off eventually huh đ¤đ
Fragrant_Reporter_86@reddit
having managed a WSUS server before they can't kill it off quickly enough. Moved to WUFB and haven't looked back.
Wanderer-2609@reddit
We circumvented this after I saw the first post on reddit phew
jake04-20@reddit
How do you trigger this? I have windows server 2022 and I'm trying to recreate this behavior and I am not getting any offerings from windows updates alone.
Phyxiis@reddit
If you use wsus then the options for windows 11 or server 24h may not be enabled to pull the update. Also at this time Microsoft may hang pulled from updates
jake04-20@reddit
What do you mean by this:
Do you just mean to update the classification and re-sync? I'm doing that part as we speak.
Phyxiis@reddit
I mean in the first place of not letting those through. Our environment we only manage server updates via wsus and let the clients pull from MS directly. So since we donât sync those classifications we didnât get the update pulled down to wsus
jetski_28@reddit
No idea, but I also ran up a test of 2022 and no offering of 2025, and this was directly from MS. No WSUS.
neko_whippet@reddit
You have a rmm. ?
JoeyFromMoonway@reddit (OP)
Action1 seems to have deployed it based on rules (Install security updates automatically). Just saw that it was an security update (the irony!)
dustojnikhummer@reddit
Wait, which A1 update was that so I can block it??
87hedge@reddit
I'd like to know too.
From what I had read in other theads the culprit is KB5044284, but that doesn't show up for any of my Server 2022's as a missing update. So I thought we're safe, but I must be missing something here.
armonde@reddit
I'm only showing it for one endpoint - a W11 24H2 system we are testing with.
So far the folks at Action1 have been pinged in several channels, including their discord, on reddit, and I've seen at least one spiceworks thread in relation to this and it's radio silence thus far.
GeneMoody-Action1@reddit
We were gathering details and initially not seeing or hearing about it in any of our systems.
https://community.spiceworks.com/t/windows-2025-being-pushed-via-windows-updates/1138286/23?u=mike-action1
armonde@reddit
Thank you sir
GeneMoody-Action1@reddit
Quite welcome, it was a zinger from MS for sure. I just commented not too long ago about how long it had been since a windows update torched a system under my control, and fortunately in this case it never crossed my Action1 systems. Accidental OS change though in a business environment is one for the history books though for sure.
dustojnikhummer@reddit
I'm on Win19, nothing there too
iB83gbRo@reddit
What's it like in 2065?
throwaway9gk0k4k569@reddit
You (and everyone else) needs to ditch Action1. People were talking about this issue early yesterday. Other RMMs blocked it because it was a big public issue. The fact that they did not is some big-time negligence. They either don't monitor social media and ongoing events, or they just don't give a fuck.
GeneMoody-Action1@reddit
Yes they were discussing it, yes we do monitor social media, yes we were watching, and the root cause of this is admin created policies mixed with a bad patch from Microsoft. So not negligence, just studying the problem. We were helping people while listening. Once understood we started speaking out.
Since this is a configuration problem compiled with a defective patch, and blocking a patch at the client level is available as an admin function. It did not and still does not appear to need intervention on our part. We are considering that none the less depending on if this issue continues to be an issue.
"Silence isn't empty; it's full of answers."
https://community.spiceworks.com/t/windows-2025-being-pushed-via-windows-updates/1138286/23?u=mike-action1
So all that said we are considering how to potentially handle situations like this in the future and if we would do it differently.
derfmcdoogal@reddit
They are active in this sub. The update is not available in my portal.
GeneMoody-Action1@reddit
Correct, we are, and we did not jump on it and in the vast majority of our users it was not present. So when we determined what *Had* happened and it was not directly an Action1, problem, we had tools users could use to address in their own environment. We waited to speak intelligently on the matter. https://community.spiceworks.com/t/windows-2025-being-pushed-via-windows-updates/1138286/23?u=mike-action1
JavaKrypt@reddit
We use Heimdal and thankfully when I first saw this posted to Reddit (they also used Heimdal) a few hours later they had blocked it from upgrading their side. They sent an email with their analysis and fix, this is an excerpt from their email. It was Microsoft's fault ultimately.
"On 5th Nov 12.16 UTC, Heimdal was notified by a customer about unexpected upgrades related to Windows Server 2025 in their environment. Due to the limited initial footprint, identifying the root cause took some time. By 18:05 UTC, we traced the issue to the Windows Update API, where Microsoft had mistakenly labeled the Windows Server 2025 upgrade as KB5044284.
Our Analysis and Fix: Our team discovered this discrepancy in our patching repository, as the GUID for the Windows Server 2025 upgrade does not match the usual entries for KB5044284 associated with Windows 11. This appears to be an error on Microsoft's side, affecting both the speed of release and the classification of the update. After cross-checking with Microsoftâs KB repository, we confirmed that the KB number indeed references Windows 11, not Windows Server 2025."
Beefcrustycurtains@reddit
I checked our RMM n-central, and the KB article that is said to have been pushed to update servers to 2025 only shows as a Windows 11 update. Doesn't have any server OS's on it. In your RMM is it classified as a 2022 or server 2025 update?
drnick5@reddit
Thats because they pulled the update.. and decided not to tell anyone that they did so...... I have an open support case about it, the agent I'm dealing with still knows nothing about it, so thats fun.
hunterkll@reddit
Yea, this is an automatic deployment rule/patching schedule based on the upstream labelling of it issue.
On unmanaged or other types of endpoints, it shows as an optional feature update like it would on client windows. Microsoft's fault for misclassifying it - but it shows properly on unmanaged systems, but also the tools fault for just shoving it in with zero review nearly right away, and configuration management's (you guys) fault for not having test rings and just blasting immediately everything without review, either.
This didn't make it through our SCCM gauntlet, fortunately, for 6k servers ;)
But yea, multiple things had to line up to have this happen, and it's a lesson learned in patch management configuration and techniques for a lot of people - i've had something similar (though not entire revision change) happen with some of our red hat fleet.... breaking upgrades/updates that slipped through because of automation rules due to specific flags and no review.
accidental-poet@reddit
This is clearly the sysadmins fault. We all know Microsoft occasionally does a stupid like this. To blindly apply patches via your RMM without any vetting is asking for what happened to you.
None of the systems we manage via RMM were affected by this. Why? We approve all patches prior to rolling them out. And they are only approved after proper vetting.
PappaFrost@reddit
Some people are defending Microsoft on this and blaming the sysadmins. There is a term for that :
Stockholm syndrome is a proposed condition or theory that tries to explain why hostages sometimes develop a psychological bond with their captors.
zero_hope_@reddit
Iâm definitely blaming sysadmins but I donât think itâs Stockholm syndrome. At least I donât feel like a hostage in the GPL world over here. đ
Efficient_Ad_4162@reddit
You mean like the bond where you just let a third party company yolo software patches into your environment with no oversight?
bstock@reddit
I don't see many people defending MS, it's pretty clear they messed up and misclassified the update.
But admins are still responsible and partially to blame here. Good patch management does not mean just applying updates without any sort of checking or overview first. Test servers are important, or at the very least, do a staggered rollout with lower importance servers first and DC's last, or something to that effect.
zeroibis@reddit
We you know for your security you are going to need to pay up for protection. You would not want to find yourself without security around these parts ya know.
lindino08@reddit
Did you test any of those servers out? What if they were all running beautifully and now you're all upgraded đ
autogyrophilia@reddit
it is convenient to either postpone upgrades or keep in touch with them, at least in the windows world.
Because when you are Microsoft you can do this kinds of things and what are you going to do, deploy ubuntu workstations?
KaptainSaki@reddit
Was expecting our company to ditch windows for good, already had the basic ms issues and every legacy windows software we use are now developed to browser so all client side stuff can be done from any os.
Then they announced dynamics crm, azure etc. So we're basically now 140% Microsoft.
hath0r@reddit
is azure not entra ?, they keep changing the name of shit on the backend its confusing
asedlfkh20h38fhl2k3f@reddit
>what are you going to do, deploy ubuntu workstations?
Stop you're getting me all excited
BloodyIron@reddit
Would you like help with that? (the Ubuntu part)
asedlfkh20h38fhl2k3f@reddit
All linux workstations + google web-only + EDR + RMM. Throw me a recipe off the top of your noggin
BloodyIron@reddit
No I mean do you want that as a provided service?
anotherucfstudent@reddit
Iâm aroused just thinking about it
hihcadore@reddit
I canât believe so many people are pushing untested upgrades to production like this! Iâm in a SMB and updates scare the hell out of me.
autogyrophilia@reddit
And that's why I guess it's something you need to get burned with as an org.
Hopefully without doing a "crowdstroke"
hihcadore@reddit
I like how I got downvoted lol. The big enterprise admins are prob mad I shamed them setting auto updates lol.
Zentriex@reddit
Fun fact! Happening a lot and it's apparently windows fault (surprise surprise) hope you were able to roll it back without too much of a headache.
double-you-dot@reddit
Does this happen on DCs? How about hypervisors?
BlackxGoblinx@reddit
Nice to see others using Zabbix
Wolfram_And_Hart@reddit
Donât approve feature updates
mnvoronin@reddit
It was (briefly) misclassified as a security update.
RCTID1975@reddit
Don't auto approve all updates
mnvoronin@reddit
Smaller companies often don't have enough manpower to manually review every single update. And don't say "tough luck".
RCTID1975@reddit
It's not like there's a list of 100+ updates every week that need reviewed and approved.
Additionally, why are you approving anything the second it gets released? Even if you somehow can't spare 10 minutes a week, waiting 1 day to auto approve would've prevented this as well.
If you don't have time to click approve on a handful of items a week, how do you have time to fix fuckups like this?
"but I don't have time!" is a cop out excuse here and you're opening yourself and your company up to unnecessary risks.
mnvoronin@reddit
Why do you keep shifting the goalpost to "all updates"? It was misclassified as a security update.
As to why not everyone can wait to auto-approve... Australian government cybersecurity guidance, control ISM-1877:
RCTID1975@reddit
Fair. I should have said any updates.
48 hours is plenty of time to review and approve updates. Hell, even 12 hours would've prevented this.
beritknight@reddit
48 hours means you can't manually approve. A hypothetical update released after close of business Friday needs to be applied by the same time on Sunday.
Automatic approval with a 24 hour delay would work here, but manual approval would not for anyone without 7 day a week IT presence.
RCTID1975@reddit
I disagree, but in your scenario, my next question is why are you auto approving this immediately at release?
You have 48 hours. Why wouldn't you delay that auto approval to 24?
Immediately auto approving any update is nonsensical and will absolutely run you into problems no matter who the vendor is. We see this time and time again, so why do people continue to do it?
mnvoronin@reddit
Even being released Friday morning won't work unless you have capacity to review the new updates daily.
Wolfram_And_Hart@reddit
Legitimately less than 20 a months if you ignore the drivers
Jarasmut@reddit
Fair enough but then you need to have some test environment even if it's a single windows server vm. When you see that vm suddenly upgrading you don't need any testing or lots of man hours to catch the problem. Directly applying vendor updates to production systems is wild.
At least roughly classify the severity of security patches and delay patching a couple days if possible. We don't install updates as they come in, instead we designate a time once or twice a month and upgrade during the day so if anything goes wrong there's people available to fix it.
Small companies letting Windows server automatically apply updates and randomly reboot overnight is a sysadmin issue, not a Microsoft issue. And I am not a fan of Windows or MS, I avoid them like the plague whenever possible. But you can have a faulty update from any vendor, like crowdstrike. That was the exact same thing, sysadmins letting a vendor install patches without any oversight/testing.
There is a big difference between manually reviewing every single update and just letting your production servers grab upgrades straight from the internet whenever and reboot over night.
mnvoronin@reddit
As I indicated in the adjacent comment, some patches have to be applied within 48 hours of release in order to stay in compliance with specific government regulations. For these, auto-approval based on category and severity is the only way, and this upgrade was misclassified as a security update.
cvc75@reddit
It also apparently had the same KB number as a previous security update(?)
So maybe you don't auto approve but had tested and approved this update last month, and whatever patch management software you use unfortunately considered this as approval for the wrongly labeled update as well.
NotAMotivRep@reddit
It's not always that easy.
omfgbrb@reddit
I really don't think an OS upgrade should be considered a patch or update. Those are really different things to me. Besides, this upgrade is not free. It shouldn't be made available as an update at all.
In my opinion, MS putting this out like this should make the upgrade to 2025 free. The post office says any package I didn't order is free. Why should this be any different?
Between the cost of the server license and the new CALS that may be necessary, this could get really expensive.
Yes, I am aware that it can be blocked; but it shouldn't be necessary. A workstation mistakenly updated to a new OS is inconvenient. A server updating to a new OS could be a major impact.
HotMuffin12@reddit
My server infrastructure at work is that bad, itâs a fucking mess. I literally want our servers to be on 2025. Heck if MS can push them to magically upgrade from 2008 to 2025, weâre golds
oloruin@reddit
For what it's worth, these never showed up in WSUS for me. So maybe an unfortunate error in KB numbers found some funky logic in a 3rd party patch management solution? I'm only doing 10-22H2, Server 2016, Server 2019, Server 21H2. So there's that.
Thats-Not-Rice@reddit
I've seen KB5044284Â thrown around as the culprit, so I've preemptively declined it in WSUS. None of our 2022 servers unintentionally upgraded, though we did spin up a new test VM to see what the upgrade was like (direct from Microsoft of course, as WSUS was declined).
No idea what's going on but I'll play it safe for now lol. As much as it's bad to avoid updates, it seems like the lesser evil at the moment.
UninvestedCuriosity@reddit
Yeah I have a hard stop in place up to 22h2 or maybe something later in gpo until we figure out our windows 11 deployments but my wsus has automatic approve security updates. So I think the gpo rule is what saved me this time. That was something to figure out for later.
I checked wsus yesterday and didn't see any approvals needed but didn't take the opportunity to dig around into this further.
It sounds like ms has pulled the cause for this though? It would be nice if they would offer more transparency but it sounds like they are doing damage control and trying to just deal with the affected. I've got meetings all day tomorrow so I'll take a closer look then at all the information. Hopefully nothing weird happens tonight. They don't pay me to work in the evening.
joefleisch@reddit
Are these domain joined member servers?
Domain joined Windows 7 did not upgrade to Windows 10 without action. Same with Windows 10 to 11.
Fire_Mission@reddit
Any detail on what exact update did this?
tooongs@reddit
KB5044284
AtarukA@reddit
Isn't that just october security?
RockSlice@reddit
Yes. But this update was also labelled as KB5044284. So you do your monthly testing, approve the patch for rollout, and a month later, that KB pulls a completely different update. Which is now pre-approved.
AtarukA@reddit
Oh dear.
Waste_Monk@reddit
Good old Microsoft quality controlâ˘
Fire_Mission@reddit
Thanks!
Substantial-Reach986@reddit
This could have been avoided if we manually micro-managed all patching, yes. It was our fault. I was sound asleep when things went sideways though, and I don't get paid until I'm in the office. So here we are. The damage was thankfully minor in our case.
wideace99@reddit
M$ force the update without its user's consent...
... and the solution is RMM and how to block it... lol...
Welcome to the wonderful world of vendor lock-in ! :)
M$ is like having a zoo full of captive customers... and they feed them... what they want, when they want, how much they want and the most important billing them based on whatever criteria they want.
My respect to M$ that has managed to train to obedience so many big or small companies worldwide !
xfilesvault@reddit
M$ force the update without its user's consent...
Nope, they didn't.
... and the solution is RMM and how to block it... lol...
No, it was the RMM that wrongly installed the upgrade. If they weren't using that RMM, it wouldn't have upgraded to WS 2025.
RCTID1975@reddit
But they didn't....
Superb_Raccoon@reddit
Wait till you see what else happened after you went to sleep last night...
FabricationLife@reddit
It's fine just restore from backups
sup3rmark@reddit
fell asleep in 2024 america, woke up in 1933 germany
kuldan5853@reddit
It's in German, but I assume you can get the point without knowing the language:
https://www.medi-learn.de/cartoons/1933-2/
nascentt@reddit
Thanks for sharing this. I don't speak German but it made me laugh.
usrdef@reddit
Tell me about it, the price of chicken went up $0.12 a pound. The day is OVER.
SmalltimeIT@reddit
Dang and I was planning on shopping for groceries tonight. Should've done it yesterday đ
JoeyFromMoonway@reddit (OP)
I really needed that laugh, thanks :')
Unable-Entrance3110@reddit
This made me LOL (I needed that, thanks!)
TheManInOz@reddit
Can I just get some clarity on this?
From what I am reading and understanding, these updates and ones like it are applicable to the product suite such as 'Windows Server Standard' which began with 1709 and went up to 20H2 and then changed to annual release.
And are not applicable to the standard server product suite such as 'Windows Server 2019 Standard','Windows Server 2022 Standard','Windows Server 2025 Standard' which I have installed now and are currently not showing any such update?
HunnyPuns@reddit
You need to monitor the update services, and if they ever turn on, you need an event handler that logs in and fucks it the fuck up.
MrJacks0n@reddit
If everything is working, why roll back (I'm assuming everything is working anyway), sure it sucks but I'd just let the good times roll.
Lopoetve@reddit
Thatâs not how security or security certifications work.
Aggravating_Plant990@reddit
So you're saying the older systems are more secure ?
Lopoetve@reddit
Theyâre currently supported and thus up to date, have gone through hardening and testing as needed, and are a known state. A major version jump is not known or tested. Most security teams would freak out - especially since it came out a couple of weeks ago!
Aggravating_Plant990@reddit
If anything, 2025 should be more secure than it's predecessor :
https://learn.microsoft.com/en-us/windows-server/get-started/whats-new-windows-server-2025
Lopoetve@reddit
Like I said - thatâs not how security certifications work. Doesnât matter if it is or isnât - it hasnât been tested or certified yet. If you are beholden to certain ISO or CJIS guidelines, or actually follow the federal STIG, you have to wait for it to get documented and tested.
Itâs a combination of liability, reliability, documentation, and proof.
Aggravating_Plant990@reddit
So yes, these guys are saying the older system is more secure. That's all I wanted to know :)
Lopoetve@reddit
The older systemâs security level has been proven and documented. And I ainât getting in trouble if I did my job and it fails.
New stuff? Canât say that yet! đ.
Aggravating_Plant990@reddit
Yep, never disputed why they say that. But the fact is that yes, they are saying that :)
Lopoetve@reddit
Thereâs a notable difference between more secure and known levels of security - one I can get insurance for. The other I canât.
YSFKJDGS@reddit
Dude if your configuration pushed an update like this 'automatically', you've got a lot of more important stuff to deal with. These threads are hilarious because frankly these guys have no one to blame but themselves at this point.
Lopoetve@reddit
While I agree 100%, Microsoft isnât blameless since this is labeled a security update, not feature or major.
JoeyFromMoonway@reddit (OP)
No. This was clearly a mislabeled update on MS Part. Not my fault in the slightest. I am required by Company Guidelines to push Security Updates asap. This was in upstream as security update. Get your faulty logic there? great.
nascentt@reddit
Autopush security updates to a QA group first.
RCTID1975@reddit
You either auto approve updates, or you blindly approve all updates.
This is absolutely your fault
sysadmin189@reddit
Get a test group setup you can roll first to test things out. It sucks, but you would catch these types of things.
YSFKJDGS@reddit
A company that says IMMEDIATELY push all patches shows a serious lack of maturity. Why don't you have a standard SLA that dictates the patching based on risk? Don't you look at the vulns fixed this month to see if there are any serious ones affecting your org, which would adjust your cycle?
This is just another microsoft fuckup, so what happened when you auto approved the update that caused domain controllers to memory leak and crash every X number of hours?
Even compliance like PCI would not require someone to literally auto approve and install a patch the same day.
SirLoremIpsum@reddit
100%
"I must comply with a strict of regulations and compliance tools. Also I have auto updates on for my whole fleet without any management or review"
Two opposing ends of the spectrum.Â
FalconDriver85@reddit
Pretty sure OP servers donât have a valid license for 2025. Itâs not like MS offers free upgrades for WS.
TechIncarnate4@reddit
Everyone keeps jumping to that assumption. It depends on your licensing. We carry Software Assurance and have rights to upgrade.
FalconDriver85@reddit
And what about, for instance, RDS CAL? Are they upgraded automatically as well? Iâm asking because at our company like 20% of the servers are terminal servers used by thousands of users, so if that isnât the case, server automatically upgrading would be a huge problem.
Putting that aside, most of the vendors providing software to us still donât support windows server 2025, so for instance for us is absolutely not an option if not maybe for DCs and some file servers.
RCTID1975@reddit
Same scenario. If you have SA then yes. I'd venture to guess anyone heavily reliant on RDS that has SA on their server OS would also have SA on their RDS CALs
MrJacks0n@reddit
That's what volume licensing is for. Any company of a reasonable size should be using it.
renegadecanuck@reddit
And if you're an MSP, you may support 100s of servers that aren't in companies that would justify Software Assurance.
OstentatiousOpossum@reddit
If you're an MSP, you could offer SPLA licensing, which includes SA.
TaliesinWI@reddit
You misspelled "Software Assurance", which isn't otherwise appropriate for everyone.
MBILC@reddit
MrJacks0n@reddit
Updates were auto-applied by a 3rd party update manager, I doubt most of these apply here.
monkeywelder@reddit
ahh the number of newbies who dont disable windows update is amazing
JoeyFromMoonway@reddit (OP)
No, we have quite expensive RMM Tools for that. Tell me you are not an IT Professional, without telling me you aren't... :')
menace323@reddit
Auto approve updates are enabled in RMM.
Tell me you are not an IT professional, without telling me you are not an IT professional.
armonde@reddit
I'm an IT professional and have been for 25 years.
I have an RMM utility that I configured so that changes in my environment, particularly to critical infrastructure such as my server operating systems are handled in a way that has been approved by ITM and the business.
Standard patching is handled via a scheduled maintenance window, and has a standard change associated. Emergency Zero Day patches are handled manually and do have an associated emergency change to document the execution within the environment.
This process was put in place specifically due to scenarios such as this in the past. My team and I have been burned by the unexpected many times in the past and those painful lessons lead to you protecting your environment to minimize the impact depending on acceptable risk - which is unique to every environment.
If everything always worked the way it was supposed to, there would be a LOT less of us working in this industry, so maybe chill with the rhetoric, thank the FSM that you aren't feeling the pain of those that are struggling with this issue, and try and learn from their mistakes without trying to score fake internet points mocking the same people you will inevitably be reaching out for help from when your own "Oh Crap" moment hits your environment.
menace323@reddit
Well, you asked where you went wrong.
You enabled auto update approval. Cool that it worked great for years, but that was the part that you could control that got you burned.
Your question was answered. Itâs your right not to like the answer, but donât pretend you didnât ask.
RCTID1975@reddit
The irony here being you're the one with the bad policy of auto approving updates
monkeywelder@reddit
tell me nothing here updates with out going through change control, you should get your money back
MrJacks0n@reddit
This is specifically caused by using a 3rd party update management product, it won't happen via the built-in automatic updates.
AttackTeam@reddit
Just to clarify. Are these OS with Windows Server Current Branch or the year LTSC?
nighthawke75@reddit
The big question is licensing
joey0live@reddit
I thought this was blocked by Microsoft?
DoctorOctagonapus@reddit
Yep. TL;DR Microslop put the wrong payload on KB5044284 in their API. RMM thinks it's installing a security update, it's actually installing an in-place upgrade.
sham_hatwitch@reddit
That is an insane oversight for a server update.
ChrisDnz82@reddit
KB5044284 is both a security update and a feature upgrade and devices will be offered both unless its already on the FU, this is normal and happens every month.
josemcornynetoperek@reddit
Did you disable unattended upgrades on Ubuntu and Debian? This should be one of first steps with it for admin ;â )
reddit_username2021@reddit
I am on sick leave. Can't wait to come back and see how many servers have been upgraded to 2025
czenst@reddit
Reading it I won't sleep as I am EU timezone - still having flashbacks from Crowdstrrike...
Sroundez@reddit
Do you guys really not deploy the
Feature Update Deferral PeriodGPOs?ElectroSpore@reddit
So what definition are you going by? Since MS seems to have stopped providing quality control for patch Tuesday releases and in-between we have release rings.
We have a VERY small number of systems that get patched first and if there are NO ISSUES a week after patch Tuesday we auto deploy to other servers unless there is an urgent zero day.
JoeyFromMoonway@reddit (OP)
I used this as a prime example why we should use ring-based updating a few minutes ago. I think i made my point finally. :D
Distinct_Spite8089@reddit
Are servers different than desktopsâŚkinda surprised it would so seamlessly start upgrading them to 2025âŚ.like itâs a defender patch or something. Seems like theyâd sorta idk ask you or require a user acknowledgmentâŚ.
hunterkll@reddit
It's misclassified, somehow, on microsoft's upstream side.
So people's third party tools are shoving it in as a security update, without any manual review if they have automatic deployment rules set up and aren't checking on them.
Distinct_Spite8089@reddit
Oh god I wonder if any of our servers got âpatchedâ
NoReallyLetsBeFriend@reddit
It's a mistake. Another post showed it was linked as an update/patch so it's rolling out like one
NightOfTheLivingHam@reddit
I have my servers set to manual updates because of fuckery.
Though I don't think that will stop microsoft from pulling this fuckery.
ranhalt@reddit
This is why I'm on reddit at work every day. Actually found out first from my NinjaOne trial where they had a banner and instructions to block it, but saw it on reddit too and notified the team and disabled our patching schedule for now.
AHrubik@reddit
The amount of people auto installing updates on production equipped has me flabbergasted.
Furki1907@reddit
Seeing all these posts and me just being happy that our Patch Management Solution didnt fall into this issue (Shavlik). Saved me some pain i guess, didnt want to roll back over 1k of servers.. :D
xangbar@reddit
Welcome to the club! We actually reached out to our RMM and they blocked the update on their side before anything bad happened. As others pointed out, others have had this issue too.