Vendor refuses to do updates during maintenance windows
Posted by Embarrassed-Lack6797@reddit | sysadmin | View on Reddit | 74 comments
I currently work for a company that has software/hardware that is used in a production/manufacturing environment. We've had the product for about 2 years now. It's been plagued with bugs and other problems the entire time.
The vendor every so often releases patches, but never notifies us of the updates because "they only do updates when there are issues." I let them know that we would like to do regular maintenance so that these bugs get resolved. They indicated that's not how they do things.
How standard is this practice? I'm used to updates being done at times that production is least affected (i.e. a maintenance window).
kaiser_detroit@reddit
If they are "just doing it" I'd be setting up automation to take internet connectivity away from those assets during business hours. And having my lawyer look at the SLAs.
DistanceMurky9800@reddit
Agreed, review that SLA asap. This is not common and super idiotic on their part.
Big-Industry4237@reddit
It’s more common than you’d think.
If the vendor is pushing updates and it’s SaaS, you’re a bit screwed, but if a local desktop app, then I’d look at traffic and block any automatic updates from the vendor 😂, I did this before with a vendor and would allow traffic from their update url only during off hours to push the update so they don’t mess with our users work day - but would stink if it’s a forced update.
Big-Industry4237@reddit
What’s the contract say? Make this a risk and known to management at your corp. when vendor renewals come up, this should be a topic.
Icolan@reddit
You have a product in production and the vendor can patch it without any intervention or control by you? That seems like a product that is not designed for an enterprise or business environment.
ReputationNo8889@reddit
Our Admins at one plant WANT it like this. They cant be bothered to actually have maintenance windows ...
Icolan@reddit
Wow, that sounds awful.
ReputationNo8889@reddit
It is. But nothing I can do about it outside of complaining
Alienate2533@reddit
Manufacturing is a whole other animal.
ethnicman1971@reddit
I would think manufacturing would be even less inclined to do anything that would stop/pause production during business hours.
Recalcitrant-wino@reddit
A lot of manufacturing plants work 24/7. When is your maintenance window?
Embarrassed-Lack6797@reddit (OP)
We have a small windows to work through which we tend to utilize a lot. That's why we like to have a test environment that we test before deploying during that window.
Lower_Fan@reddit
It is usually 3rd shift still.
ang3l12@reddit
That’s actually the case in my experience. This vendor saying they don’t do regular maintenance / updates is probably just that: don’t fix what isn’t broken.
Now has OP been up front with the vendor that they have issues? If any of our automated welders / cnc’s / robots had problems you better believe I would be hounding my vendor until it was fixed or the machine was removed from our shop with money coming back to us
ethnicman1971@reddit
Use this as a learning experience. I don’t mean to pile on your issues but for the future it would be good to discuss and make sure that you agree with the maintenance (patching/upgrades) schedule for any software before you purchase and implement it.
Once the vendor has your money/signed the contract they have no incentive to make changes to the agreement.
Embarrassed-Lack6797@reddit (OP)
We've opened multiple tickets with the vendor to indicate the problems, basically with no response.
jimicus@reddit
This is a case where you delegate it upwards. Advise management of where you are and ask for guidance.
vandon@reddit
Yes, manufacturing is a whole other animal. You have quality and quantity metrics you have to meet for your customers. You have to adhere to ISO standards to pass those certificate audits from BSI or whoever is doing yours. And part of those is not allowing random patches and recipe changes to your line without QA testing and acceptance.
Firewall those tools. Access should only be allowed with notification and during maintenance windows.
CARLEtheCamry@reddit
Damn straight. I'm so tired for the mindset "OT is different, so we can't do anything".
No, you should be doing more. This is how pipelines and water supplies get cut off, some plumber installs a PLC controller open to the internet with a default password because they just don't do computers good
Syde80@reddit
What's your opinion on cloud products in an enterprise?
Icolan@reddit
I don't see how cloud products are relevant to the conversation at hand. OP is talking about software/hardware used in a manufacturing environment that the vendor has the ability to patch/update without notice or intervention by the onsite IT staff. That is a recipe for disaster with many manufacturing machines.
Cloud products are an entirely different situation, so your question is quite irrelevant.
Alternative-Mud-4479@reddit
This is such a generic question that there is no one answer. Cloud can be amazing or a huge liability depending on the enterprise.
Brad_from_Wisconsin@reddit
can you check on their web site to see if and when updates to the software are released?
You could probably create a shell script to do this for you and notify you when an update is posted and down load it to your local code repository.
Embarrassed-Lack6797@reddit (OP)
Unfortunately, to my knowledge, they don't make that information available publicly.
Brad_from_Wisconsin@reddit
If they have an ftp sire that holds the latest release of the software you can do a check to see if the verson or modification date has changed since the last time you checked.
MeatPiston@reddit
This is how this type of equipment works and you have to adapt to it.
Yes it’s a pain and yes you will have to airgap or implement annoying and strict controls.
Industrial equipment isn’t squishy amorphous blobs of poorly maintained code updated every 30 seconds by an intern in Bangladesh.
It’s a squishy amorphous blob of poorly maintained code connected to servos and motors and mechanics that must perform in a certain way and once it’s tuned right you don’t touch it without a very good reason.
Embarrassed-Lack6797@reddit (OP)
When they conduct updates on the equipment, the parameters are carried over so they don't have to do fine tuning to get it to work.
ntw2@reddit
Sorry, vendor, that’s not how change management works
ExceptionEX@reddit
What is your SLA say, don't buy software without having this sort of shit sorted out.
NullaVolo2299@reddit
Red flag. Any vendor refusing scheduled maintenance windows is setting you up for failure. They'll wait until something breaks during peak hours, then blame your environment.
Document everything. Time to shop for alternatives.
uncertain_expert@reddit
Manufacturing has a long history of ‘If it ain’t broke don’t fix it’ mentality. Patching installed software or upgrading to new versions is, quite frankly, uncommon. This may not be how you wish to operate, but it is likely the approach that the vendor has come to expect of their clients.
How often patches are applied and when is something that should be specified in your maintenance agreement, as should any expectation of how suitable maintenance windows are communicated. Maintenance windows have to be suitable to both you and the vendor - if your maintenance windows don’t align with the normal working hours of the vendors support team then you need to negotiate this further.
Embarrassed-Lack6797@reddit (OP)
That's been the common theme for me at this point. Based on this, should there be a change in this mentality in the industry? Or is it forever bound to this methodology?
jimicus@reddit
Can you clear something up for us?
The way you've worded your question, it's not clear if what you mean is:
Embarrassed-Lack6797@reddit (OP)
The manufacturer has patches that we have to explicitly tell to apply to fix bugs that we have. They don't notify us of these updates. For some of our tickets with them, we've waited a few months before they came to address another issue and they indicated "oh that's been fixed in so and so update."
We do have another vendor that does number 2, though at this point I've come to terms with it unfortunately.
jimicus@reddit
So - just so we're all clear on this (please correct any misunderstandings):
How active are you in chasing issues that you raise? Some vendors do tend to need more pro-active (read: firm) management like that; that's simply the nature of the beast.
Embarrassed-Lack6797@reddit (OP)
That's basically it. They don't allow us to repair not even the software side as it requires a technicians token to do so.
I get notified of the issues and see if I can resolve it without the tech token. If not, I have to wait for the vendor to get back to me on the issue.
jimicus@reddit
Sounds like they just need regular chasing then.
Strictly speaking, it's not a technical job, but you might not have the luxury of a service desk you can pass it on to.
uncertain_expert@reddit
I think it is changing, slowly. Pharmaceuticals used to be among the worst for this due to their validation requirements for quality control; but attitudes have changed and quality procedures are now written to allow for patching and upgrades without full validation testing.
IT teams are slowly being permitted to (or forcing themselves) into managing OT networks they previously (were told to) turned a blind-eye too.
I’d like to say that vendors are doing better at making products easier to perform in-place upgrades and patches on, without requiring monolithic upgrades where every component is linked by version dependencies, but it can still be a challenge to account for all the dependencies across distributed systems.
I haven’t seen a shift towards maintaining dev or pre-prod environments for testing updates. It’s encouraged, but licensing costs make it expensive to implement.
Virtualisation and thin-client architectures have helped for server-based applications. The ability to snapshot or otherwise quickly roll-back changes has shortened the time required for maintenance windows.
Soggy-Camera1270@reddit
Personally, I think what has helped is that companies are finally understanding GAMP better and realise that patching software and performing routine infrastructure maintenance should not impact the process and validation of said process.
I still remember once reading some validation documents where the "test" was - "plug the network cable into the socket and check that the light comes on" lol.
cctsfr@reddit
Oh boy is some of that shit stupid. Wasted years of my life retesting shit for no reason.
The main issue was people stopping when there is a fault, processing a fix, then restarting all the validation until the next fault.
mr_data_lore@reddit
Sounds like you need to find a different solution ASAP. This absolutely would not fly with my employer. We've killed contracts for less.
trail-g62Bim@reddit
Sometimes there is no choice. We have a vendor that won't work during our maintenance windows. It's a niche product in a niche industry with no competitor. They know you can't leave them.
pdp10@reddit
It's a classic marketing tactic for a vendor to claim to be sui generis, but don't be fooled. Especially not by some poor-quality but lightly customized ERP, just because it's marketed into a specific industry vertical.
Saying that software is always replaceable, isn't the same as claiming that migration is easy, though. For example, we once found a 50-person SMB using a poor-quality vertical ERP, but migrating away wasn't going to be easy. The ISV's support was being leaned on to keep business processes moving, and the prices being charged were so surprisingly low that they wouldn't even come close to covering the cost of a half-time app engineer on-site.
The organization could easily have left, but they didn't have any intention of investing in that process. It was cheaper to live with the flaws, and besides, all that business inertia...
trail-g62Bim@reddit
That's true for most software, but definitely all. There are definitely niche spots where there really are no alternatives.
dontmakemewait@reddit
Is this SaaS? Because that’s common.
If the hardware is on prem, are you able to apply the patches on your schedule? Why would you be accepting automated updates? The release should be separate to the deployment of the update.
Any vendor with more than one client is not going to release the updates on your schedule, but you should have a mechanism to push the patch within a certain window that you control.
Embarrassed-Lack6797@reddit (OP)
The issue isn't necessarly release of the updates, but we have no control over installing the updates as making any modifications requires them to enter a technician token only the vendor has. Only they are allowed to install updates.
lost_signal@reddit
Sounds like you need to negotiate that scheduling etc, into the next renewal but as a vendor who's run SaaS solutions, and used to work for a MSP let me explain why:
I and my employees have children, and we like to get a normal nights sleep.
Let's say we send out a red shirt to do the update at 3AM on Friday night like you wanted. Great, now there's no one awake for him to call to help. Doing patches when vendor support has gone home and engineering is asleep is a bad idea. yes SOME of us in vendor land have 24/7 global engineering teams who can handle critical escalations at 3AM US time (We do this, it's outrageously expensive) but most do not.
Is the vendor on the hook for faults from the system, replacing damaged components, or for outages? over 90% of support calls we get are the result of... People not being on current patch levels, or people deploying or configuring our products in ways we do not recommend. If this vendor let you go do muppet things, it would increase their support costs possibly by 10x... Even let's pretend you and your operations group are enlightened gods, and infinitely lucky and never would hit these bugs they fix. Great, they still have 99 other customers who would do dumb things and inflate support costs.
Now i agree not letting you do patching earlier is kinda odd, but it may just be a product where everyone's use case is so snowflake it's basically 1 branch of code per customer. This happens and it's messy as hell.
Ssakaa@reddit
So they're remoting into your systems and making changes that knock out production and, as such, impact your company's bottom line financially? What's cost on average? This also implies they have relatively unmonitored administrative rights on systems on your network (I'm assuming, based on the situation, that the computers themselves are technically the vendor's, licensed for your use in the process of running their machine that your company's completely dependent on).
There's also a very big concern of a conflict of interest in that, given they could play favorites with clients (or worse, directly be in the production space themselves, so you're outright their competition) and hit one but not the other at decidedly inopportune times.
There's just a metric ton of reasons this is a terrible id
sirachillies@reddit
What does your contract say and when does it end? That will be the only way to get them to do anything. If the contract says they only do updates if there are issues then unfortunately you're stuck until it ends. If it ends in the next few months to a year... Then you will have to wait it out or breach the contract early and pay fines and draw up a new contract where they do updates periodically. If they can't do the updates periodically (won't do).. idk? Find different software..?
SquirrelOfDestiny@reddit
I had a meeting today with one of our vendors to discuss their planned updates this year, and align when they will be deployed to our environment with our service windows. I really couldn't imagine letting a vendor manage a service in our environment if they wouldn't adhere to our service windows...
dracotrapnet@reddit
It seems standard for anything to do with embedded devices. "You have no functional problem that isn't fixed by customary monthly, weekly, daily or hourly reboot, no security updates for you."
Hate list:
SandeeBelarus@reddit
I know we have to secure our ICS and OT infrastructure in the US. I used to work in that environment. But given the different contractors and vendors who work in this space. It’s gonna be a while…
crnkymvmt@reddit
Manufacturing is the last industry to get it together when it comes to doing IT and security right. In my experience, money and losing your business will turn heads before asking for better processes.
cbq131@reddit
What does the contract say?
Some vendors are poorly managed and just update when they feel like it. Others have a set schedule besides security updates. Others let you set update rings. Some you update yourself, and finally, some don't have updates.
It is important to ask these questions when you vet out vendors. If you have leverage with account management or their executive or pay enough, you might be able to get change/exceptions.
ZAFJB@reddit
Effectively vendor is an MSP for the manufacturing machines.
Outages and maintenance windows are for discussion between production mangers and vendor. If things are not working; again a discussion between production management and vendor.
Why are you getting involved at all?
Do you get involved when the CEO's BMW firmware gets updated by the BMW service agent?
The only involvement we have with manufacturing machines is ensuring that stuff is done securely. A lot of the time that boils down to isolating stuff on their own LAN, or no LAN at all.
Soggy-Camera1270@reddit
Who's the vendor? I'm genuinely curious. I work in an ICS environment, and a lot of smaller vendors drive me nuts with their "beliefs" of how things should be done.
seengineer@reddit
I work in industrial automation and no patch gets installed for our core product without a work permit!
Barrerayy@reddit
That's dumb, what's your SLA say?
peekeend@reddit
Time to do a security audit and find out how bad they are.
Secret_Account07@reddit
This is dumb but when people are stubborn you hit em with paperwork/writing. Make sure there is paperwork with signatures on who is responsible for applying security patches. By definition, security patches typically aren’t convenient. But in inconvenience is not a valid excuse to ignore em.
If this vendor is really taking the stance- yeah we will decide what gets ignored and the time frame. Okay great- here is a legal paperwork absolving our organization of all responsibility if (honestly should be ‘when’) some kind of security incident happens. We just need it signed by your CEO.
Magically several days later- hey the vendor has a new rigid schedule for patching. They also have a CAB. That’s weird, huh? Oh we are also having a meeting this afternoon to address current infrastructure that is vulnerable.
The one good thing lawyers are good for…
BadgeOfDishonour@reddit
If it is a SaaS product, you're stuck with the offerings they give you. If it is on-prem, then you need to remind them that you are the customer and they are the vendor. I'd be immediately looking elsewhere. Your company's ability to do business should not be at the whim of some random external vendor.
Embarrassed-Lack6797@reddit (OP)
I had that discussion with management. They basically indicated that we are "at their whim." It's an on-prem solution.
natefrogg1@reddit
Since it’s on prem, would it be feasible to just deny internet access to these systems during normal business hours? There has to be some way to not allow them to remotely patch whenever they would like.
vertisnow@reddit
Yeah, just cut them off. Tell them you moved to a just in time access model.
unavoidablefate@reddit
You need a new product
BadgeOfDishonour@reddit
That would be unacceptable to me, and I would document the risk clearly, so that the Higher Ups understand that if they let this go, it is their heads on the block when things fail.
Natural_Sherbert_391@reddit
So you are saying they are pushing updates whenever they want and taking down your systems in the process to do the updates? This is something your management needs to sort out with the vendor. Ideally this should have been dealt with in the contract.
GladObject2962@reddit
Yeah, @op, there will be some sort of SLA that dictates the acceptable downtime, etc.
This shouldn't be happening
Ok-Double-7982@reddit
Hotfixes are different than patches. This is a cloud SaaS vendor?
Also, if you don't like the software, invest the time to look at other solutions. Everyone's needs are different.This approach might work for other companies, but it doesn't sound like the product or vendor approach works for your company's needs.
mercurygreen@reddit
If you're locked into the product (you can't afford to change, or no alternate product is available) it's distressingly common, especially with manufacturing equipment.
My best advice is to actually read the contract and see what they promised. Also to talk to your CFO and tell them that it might be time to renegotiate the contract the next time this comes up because it's costing the company lost revenue during unscheduled maintenance.
BoltActionRifleman@reddit
Your company pays them (I’m assuming) dearly for support, updates and maintenance. If they want to retain you as a customer they can learn to do things on, or at least somewhat close to your schedule. And as far as allowing them always-on access to the system, don’t. At the very least appoint someone they have to contact in order to be able to remote in. That person being the MFA gatekeeper, or at the very least the person who reads them the remote sharing code etc.
prodsec@reddit
That would be something built into the contract.
Special_Luck7537@reddit
If they're just knocking your prod offline to patch whenever they feel like it, somebody would be getting a bill for the downtime ...
dunnage1@reddit
Even ServiceNow notifies admins of pending updates. 🤣