Doing big IT changes on Monday or Friday?
Posted by CeC-P@reddit | sysadmin | View on Reddit | 388 comments
Help me solve this debate because we did not see eye to eye on this at the last 2 places I worked. Assuming both are equally allowed by your labor hours usage and your company generally doesn't operate on weekends, answer the question below.
We want to do big IT changes, changeover, new gear, firewall reconfigs, mail server changes etc on:
1. Monday so we have the night and rest of the work week to fix it if it goes wrong
2. Friday so we have the weekend when nobody is working to fix it goes wrong
Trying not to bias this with how I wrote it, but I have strong feelings on this and anecdotes from 15+ years in IT to back up my position about what the safest, best answer is.
beanmachine-23@reddit
Only on a Friday if you are hourly and will get OT. As a salaried sysadmin, it’s Read-only Friday.
_cheezehead_@reddit
Depends on how big your company is and what they do. If being down for a while is costing the company money, then you do it after hours or on weekend.
FireCyber88@reddit
Monday. This way you have full resources available for 5 days if you need them.
Neat-Researcher-7067@reddit
Read only Fridays!
lanekosrm@reddit
If we’re making a large change on Friday, it’s because we’re expecting to work Saturday and possibly Sunday. The same applies for any changes in the days leading up to a large business event - you don’t do anything big in the ~72 hours leading up to the business event unless the team is prepared to spend those hours in crunch if something goes wrong.
kungfu1@reddit
How is this even a question.
CeC-P@reddit (OP)
I know, right? "I care more about my personal weekend time than the company's downtime risk or stability" - everyone.
Didn't expect that but most people work at shit companies that they don't respect, that pay like crap, and don't offer emergency OT, etc. I'm a little more on the fence about this issue but not one single person saying Monday mentioned "well we have until like 7 AM the next day to fix it. That's like 12 hours!" That's the only justification in my book. Otherwise, it's just showing that they don't care about their company or their job, don't want to be there, and defend their weekend accordingly.
Blues-Mariner@reddit
OR they have a family, with children, whose significant life events happen ONCE and only once and who are grown and gone in the blink of an eye. I like my job, mostly like the company as companies go, have a great manager. But my weekend is MINE, goddammit, unless someone’s got a very good reason. State your point of view on the question, don’t judge others’ pov. You’re not living their lives.
AdmRL_@reddit
People who say don't implement changes on Friday are people on teams who don't know how to properly scope, plan, implement & test changes, or are under management who's been burnt one too many times by people who don't know how to properly scope, plan, implement & test changes.
Blues-Mariner@reddit
Sweeping generalization much?
Blues-Mariner@reddit
We have a very small staff - one person doing each job, mostly - for a not-that-small company. We don’t do any changes to production on Fridays. For dev/test, it depends what projects those users have in progress. If something would have to be fixed immediately if broken, then not on Friday. If they have nothing happening on a deadline, maybe. We’re fortunate to have a director who believes evenings and weekends are for your own life, not the company.
_litz@reddit
Only ever do it on a Friday if you're committed to losing your entire weekend fixing it when it fails.
egamma@reddit
Only ever do it on a Monday if you're okay with getting fired for impacting production when it fails.
worjd@reddit
Can’t do it friday cuz you’ll be stuck all weekend, can’t do it Monday cuz you’ll crash prod, guess we can’t do it 🤷 Problem solved.
pdp10@reddit
Don't laugh. More than one organization has decided that they like this option the best. Even, at times, quite large organizations that don't lack skilled engineers.
As flawed as compliance regimes are, those regimes mostly exist to force organizations not to have the policy above.
Blues-Mariner@reddit
These orgs are sometimes known as “owned by private equity.” 🤬
Blues-Mariner@reddit
Not sure whether you think you’re kidding but we have one system, a job scheduler for SAP, that we can effectively never shut down due to organizational paranoia. For some reason they’re all less paranoid about the security patches it doesn’t get.
CeC-P@reddit (OP)
That explains all the 23h2 computers in this environment
Blues-Mariner@reddit
Have you already run through the change on dev and test, and worked out the kinks to greatly reduce risk when changing prod? (Assuming you HAVE dev and test…)
40513786934@reddit
so... Wednesday?
CeC-P@reddit (OP)
Wed is when we fix windows updates related problems though lol
TheLastRaysFan@reddit
Patch Tuesday, homie
BrilliantJob2759@reddit
We run Change Tuesday and Patch Thursday.
mnemoniker@reddit
From another perspective, if a change could break so catastrophically that it needs to be fixed immediately and takes 3 days to fix it, i think I'd rather not do it working the work week.
Competitive_Smoke948@reddit
would you rather do it & cancel plans you've made with friends & family for a company that will kick you out at a moments notice when the CEO wants to bump the stock price.... take it from someone whose done seizures, panic attacks & other stress related things due to work, I have refused to work on weekends for even risk to life systems without prior planning & double time
PowerShellGenius@reddit
Yeah, toxic jobs deserve nothing. I don't see my employer doing that, given a school district has no stock price to bump, and having no reason or notice would go against the union contract...
knightress_oxhide@reddit
I too like donating my priceless time that I will never get back to corporations /s
CeC-P@reddit (OP)
I like costing the company millions in losses due to an outage on a Tuesday because everyone was too lazy to do their occasional job a few times per year upgrading or changing a critical system during off-time at the safest time with the most cushion, making IT look bad and threatening my job's future.
BaconEatingChamp@reddit
Overtime here so yeah I do prefer Friday after hours just in case. Shit goes wrong I have less stress while fixing and more pay. If I was OT exempt and didnt 'only' work 40 hours id probably feel different
GwentMorty@reddit
Which is why you don’t make changes on Monday.
ofd227@reddit
Usually has to do with having access to 3rd party support. All my change windows are scheduled around support availability of whatever equipment/product I'm working with.
Suntory_Black@reddit
This, 100%. I'm an enterprise level TSE. We are available but in general if things go really sideways the best Dev Engineers usually don't work weekends.
Flat-Photograph8483@reddit
Good point
Mutiny32@reddit
You're not calling me on the weekend for something you broke installing at 4 on a Friday afternoon. Miss me with that shit.
Old-Flight8617@reddit
It's the access to support that becomes trivial during that time.
Internet-of-cruft@reddit
Friday is also known as FAFO Friday or Readonly Friday, depending on your orgs CAB (or lack thereof)
Speeddymon@reddit
The team I'm on, we run the platform and even the big important stuff is done during the week because it's easier to get support from vendors during business hours in the event of an issue.
xot@reddit
I think it depends on the number of people required for the worst case outcome. Ive worked outages where literally hundreds of people have lost their weekend because someone lacked the foresight or patience, pushed through a bad change or deployment, and made a complete mess. It would have been better to do it on a Tuesday morning, so that all those people could do it during regular work hours.
Some things are inevitably disruptive. A small company needing to perform a data migration, is better served by giving a handful of people a Monday off, and scheduling them to all be available for the weekend, so that things can happen outside business hours. As others have mentioned, this might include lining up vendors, outside contractors, customers, stakeholders to also be available.
Changes always carry risk. We do our best to plan ahead and mitigate that, and generally, “we’ll do it Friday arvo just in case” is a shortcut that carries more risk than adequate planning.
The goal is minimal customer disruption, and minimal team disruption. Screwing up everyone’s weekend is more disruptive than planned downtime during a weekday. In most orgs we block Friday and weekend changes, because it’s way too hard, and unfair, and expensive, to pull in subject matter experts on a Saturday night.
Arillsan@reddit
Just curious, when 100 lost their weekend fixing it, how many would be impacted doing that fix monday? Like, if only the people fixing it lost weekends but workers relying on them to fix were not impacted I'm having a hard time motivating why monday, where everyone else besides the fixers, would be blocked?
I'm clearly missing context and details of your setup, hence why Im curious :)
xot@reddit
Great clarification.
Those are all people who are paged in, starting with oncall engineers, who quickly page their peers and leaders for backup.
People having to cancel plans on the spot, due to cascading failures across orgs, because a human did something dumb, when they could have waited.
Multiple companies.
Arillsan@reddit
Right, so 100 people are affected on the weekend, those extra 90-95 people being called in are the sum of everyone working in the company, so the issue could have just as well been handled on monday when everyone of those would still be the only ones taking a hit?
xot@reddit
Right. And these are tech providers, so if it’s disruptive enough, customers monitoring systems start paging them in, so it can end up disrupting a lot of people, even if they don’t need to stay engaged.
Cheezzz@reddit
Agreed. But…
What if the change will cause critical apps to go down? Or a sustained outage will cause you org to loose revenue?
I have been involved with “IT Changes” that went horribly wrong; the outage was 6 hours and would have cause major revenue loss for clients.
These changes were scheduled for 21:00 on a Sunday; so only the SysAdmin’s noticed something was wrong.
Are you suggesting to rather do it at 21:00 during the work week?
TheRealLambardi@reddit
lol yup. Rookie mistake
michaelhbt@reddit
Or get overtime.
rire0001@reddit
IM<HO, you should be scheduled for the whole weekend anyway, and if things go to plan, it's a win.
Adept-Midnight9185@reddit
As long as they're scheduled and committed to either paying me for that time or giving me weekend number of hours off during the next work week regardless of whether those hours are actually worked, AND this is all arranged in advanced.... then fine.
That way I won't make weekend plans that may or may not have to be cancelled, but can make plans during the next week instead that won't have to be cancelled.
rire0001@reddit
Oh, absolutely, yes. Scheduled in advance, and either OT or equivalent time off during the week. I used to give my teams two days off, regardless of how the upgrades went.
Grand-Height9907@reddit
You might need the help of other techs
So be vary of that it is not just your time
niomosy@reddit
Hah! As if we're allowed to touch production before Saturday morning at the earliest. Worst case, it's the midnight to 5am maintenance window Sunday morning.
Dwonathon@reddit
A huge reason I negotiated to stay hourly. It broke? That sucks, I'll be back on Monday.
cannon19@reddit
"read only friday". say what you want about working in IT but probably THE best perk of our field is the fact that friday's are for kicking back and reacting to alerts. for some ungodly reason fridays is my wife's busiest day with meetings meanwhile im putzing around the house getting chores done while keeping 1 eye on my instant messages and emails. i see it as our gift for getting our asses kicked monday to thursday
VulturE@reddit
If you are IT for a smaller business, And that business does not operate on the weekends, or the computers being down for the weekends would not stop physical operations from continuing, then yes, it might make sense to make a change starting on a Friday so that you have the whole weekend to keep working on it as part of your disaster recovery plan for the major change.
In any larger business, there are typically people still working or completing projects on the weekends and it makes more sense to have downtime and additional technical staff availability during the weekdays. Also for many of these staff they are salaried, so working on a weekend does not look good unless they're getting comp time or overtime.
Personally, I prefer no change Friday. It allows me to dedicate the day towards planning, discussion and other deep dives, with it being a day to create CMs and further along projects on non-production systems.
AzBeerChef@reddit
Depends on the ENV and business needs. You support the critical business services that allow a business to run effectively and efficiently.
There is some merit to both. I prefer a maintenance window which impacts the least amount of users as possible and, yes, you can make it work with your life too. Research, notify, implement, validate, support. Always have a rollback plan.
peaceoutrich@reddit
Tuesday morning is my absolute favourite major change time. Friday is a no-go for obvious reasons, but Monday as well as you can't prepare for the change the day before. Never after hours.
tech-guy-says-reboot@reddit
There are a lot of variables to consider that you haven't mentioned. Third party vendor support is one very important one. If it goes bad late on Friday, are you going to be able to get help before Monday morning? But there are valid arguments for both Friday and Monday. But if you plan for Friday, you better have your weekend open.
mixduptransistor@reddit
Last place I worked we didn't make any changes on Friday, and unless it was going to cause a very long outage we didn't do anything on the weekend. We tried to do things early in the week, even if after hours, because if you break something you want people to find it as soon as possible
Changing it on Friday or the weekend means something might sit broken for 2-3 days before anyone notices
AdmRL_@reddit
Are you just making changes and then.. not verifying whatever you've done works? Nothing should be sitting broken for 2-3 days because you should be scoping your work and then testing after implementing.
sccmhatesme@reddit
I’ve seen instances where it worked for the person after implementing but then 30 users had issues that wouldn’t have shown up until normal working hours for example.
cha0z_@reddit
frequently it comes down to poor testing plan than anything - it was obv lacking and didn't cover basics if it works for 1, but not for 30. This is not edge case at that point.
ConsciousEquipment@reddit
no, there are clearly processes and use cases that are only carried out during business hours, and it might not be possible to simulate all of these things in advance. Especially in e commerce, sales etc you might see that the config works, the views are ok, then 30 people log onto looker studio and submit their stuff and suddenly the dashboard does break because someone eventually had something that you couldn't account for ahead...also, if I am not on call for the weekend I could in theory have a flight booked for friday evening and literally just leave. In the meanwhile they will lose 2 days of orders or whatever that don't get forwarded right because customers can of course still go on the website at any time. With my environment would be a shit idea to make major changes that late in the week or right before any kind of holiday etc especially then it needs to be rock solid.
auto98@reddit
Sounds like you need some automated journey testing that can do all of those tests. That's not to say some won't slip through, but working for the 1 person testing and then not for the other 30 means the testing is 100% not complete enough.
tfsprad@reddit
Is the testing ever complete enough?
jasmeralia@reddit
Or customer SSO integrations where they refuse to give us test accounts and they don't staff weekends. Queue Monday morning tickets... It's not really a lack of planning as much as it is customer refusal to let us properly validate changes.
Thankfully, not something my team has to support, but I've seen it happen to one of our other teams a number of times.
cha0z_@reddit
To not get into wall of text post - what u/auto98 replied to sum it up. You are responsible for the implementation and to test it properly. You need to find what needs to be simulated in advance and do it, if needed involve someone that usually is not working in those hours to test it. You will be surprised how many will be willing to help, especially if you have good connection with them (double if your business is paying overtime as it should).
slyck314@reddit
There are impacts that only application specialist or users are going to notice.
mixduptransistor@reddit
it is entirely possible to have the most perfect test plan, and something slip through. obviously the scope depends on the system and the change being done, and the maturity of your monitoring
tfsprad@reddit
Put in a change on Friday afternoon and nobody notices that it doesn't work until Monday, and the outage is on Monday anyway.
BamCub@reddit
No unexpected extended downtime with change control and roll back plans > Monday
Long extended outage or complete unknown outcomes > Friday, prepares for through the weekend
auto98@reddit
Users should never be the people finding issues.
Obviously it can sometimes happen, but it should not be part of the bloody plan!
mixduptransistor@reddit
why shouldn't it be part of the plan? it obviously shouldn't be your primary testing method, but you should absolutely be planning on what to do in case a problem slips past your testing and that problem impacts your users
If you aren't planning for what if something goes wrong you're way off the reservation
auto98@reddit
Yes you have a rollback plan, but it should never get to the point of user-impacting (or even worse, customer-impacting) - maybe we are reading the post differently, but I got the impression it was the type of thing i hear on CAB everyday along the lines of "my testing is making sure the new certificate for our main customer site is showing as active and if any issues happen the call centre/customers/incident management will make us aware. We want to perform the change in-hours so that we will be made aware more quickly if something fails". When you then ask if anyone is literally going to just open the site to see if it loads it's like you've shagged their mum or something.
Honestly, one of the major things about tech people is that while they understand the technical side of the job, they often don't appreciate the primary reason for the work, which is to provide a working service to the customer (customer in an ITIL sense).
Temporary-Living@reddit
*My view is from a normal corporate IT team juggling numerous apps, services and operations. It changes a little if you are a software dev *
Scream testing should never be the primary test. You should be testing the main user paths/workflows. However it is impossible - unless you have far more resource than I’ve ever heard of happening - to exhaustively test: Every feature Every input type - text, symbols, emojis. Every file input - word docs, zip files, etc Every query/report/page of the app As every user or user type With a user in each group and role of the app And every permutation or combination of how the above interact
Once you test the most frequent or impactful flows, you have to then it over to the users. And they may discover a glitch immediately. Or in two weeks. Or in two years. Or never.
If you are testing exhaustively as above please tell me your industry, team size, etc as I’m genuinely fascinated.
accidentlife@reddit
I work for MSP.
In many cases, I found users are more familiar with specific apps (LOB apps in particular) than I ever will be. And they should, considering they use it more often.
Whenever we do a project, we like to schedule time with the user to verify that the results of the project are what they were expecting. This gives us time to make any changes that the user wants now that they’ve seen the project results, and allows us to catch any issues while we are still engaged with the project.
cha0z_@reddit
There are ways and you should always verify the work you have done. For bigger businesses there will be always someone who works over the weekend as well if you need users help. Doing it Friday gives you time to fix it/revert without major impact during the weekend.
mixduptransistor@reddit
Didn't say that you shouldn't test your work or changes, but things happen and things can slip through your verification process. The nature of the business we were in, the weekends were heavy batch processing times so it was almost as critical (or in some cases more critical) than doing something during the week
cha0z_@reddit
Got you, business specifics are real and can lead to shift of schedule for major changes. Still, for most businesses the working week days are generally more critical and doing changes Monday/Tuesday evening can lead to, I will be blunt - searching for a new job, but not less importantly affecting way too many people you most likely to some extend care about (leaving the: I will feel bad/like a failure part :D) - in a bad way. The "If something happens, I will fix it" is 10 hours window at the time you are done during the week, but the weekend is 60+ hours - you tell me what time window you prefer if something goes horribly wrong.
tornadototes@reddit
uhh do it in DEV and TEST. The in PROD on Thursday.
mtnbunny@reddit
No change Friday!
Morpheus_90_54_12@reddit
There is simple rule.
If you broke something on friday, i will try to fix it in monday. 😉
PEBKAC-Live@reddit
We do big changes on a Friday so they can run in to the weekend for any issues
NanobotEnlarger@reddit
At least part of the consideration should be whether you can get decent support over the weekend if things don’t go well. If not, weeknight is preferable.
BamCub@reddit
No unexpected extended downtime with change control and roll back plans > Monday
Long extended outage or complete unknown outcomes > Friday, prepared for through the weekend
TheCookieMonsterYum@reddit
If you don't operate on weekends then who cares if it's down? Friday rule doesn't apply in my opinion.
Fistofpaper@reddit
I'm not sure why this is a "debate" at your work, but major changes are only Tuesday through Thursday for more reasons than I care to list off, but have undoubtedly already been documented in this thread.
Defiant-Chip6513@reddit
Tuesday or Wednesday. NEVER Friday. Never push to production on Friday. You're basically committing yourself to working that weekend.
Lukage@reddit
We're salaried. Our office hours are M-F 9-5.
We're expected to work any weekday the same as any other, and off hours any time as needed.
Work is work.
micromsp@reddit
With few exceptions, we only work M-F 8-5, so we tend to avoid changes on Fridays. If they are unavoidable, we typically have someone on standby Monday morning to deal with any fall out. We also typically avoid scheduling big changes on Mondays. Because unexpected issues that come up over the weekend have to be handled Monday morning.
We do make exceptions for customers that have to have something scheduled outside normal business hours but it's double the rate.
We also have a very small number of clients that are 24/7 for emergencies.
RocketSurge@reddit
Mondays are fine, but it’s Read-Only Friday over here unless we are all hands on deck, or there is something planned with cross-team partnerships for modifications and validation.
Sleepytitan@reddit
We usually do them after hours on a Tuesday or Wednesday.
We still have time to do a rollback before work the next day.
We also have a good chance of getting someone decent from vendor support if there is an issue that isn’t a full scale outage and can be resolved without a rollback.
It hasn’t killed the business and no one has lost a weekend over it.
Round-Classic-7746@reddit
yeah same here, having that buffer for rollback aand still being midweek is kind of the sweet spot
CherryChokePart@reddit
Never do anything important on Fridays unless you're all hands on deck through the weekend.
kigam_reddit@reddit
Definitely make changes on Friday, this has saved my company a lot of time and resources over the years. We also got rid of QA and made our developers add tests for every new feature, then we replaced the developers with AI. So now we make changes and push them out to production on Friday and by Monday when I turn my phone back on I have a record of what issues we didn't find and which customers it affects along with direct contact information for senior management at our various customers.
hosalabad@reddit
What's the compensation for losing your weekend?
GwentMorty@reddit
No. Absolutely not. No changes on Mondays or Fridays.
wraithfive@reddit
We have “no change fridays”. Which isn’t a hard and fast rule. Changes can go through for business continuity reasons with VP approval (IT VP, not the marketing guy) but there are rarely any valid reasons to do a Friday change. We don’t ban Monday changes but they are rare in practice. We do major maintenance events on weekends but that is always planned months in advance and every one works it.
helphunting@reddit
It comes down to risk and or costs.
If its down for a week, what is the cost to business?
If its down for a weekend what is the cost to business?
Depends on what "it" is and how much fixing it will be on a weekend vs week day.
aringa@reddit
We do maintenance items 7 days per week. If it has the potential for major disruption, we normally do it Saturday or Sunday.
above8k@reddit
monday mornings and testing jobs can be delegated to users and customers
SapphireSire@reddit
Tuesdays are best....you can always blame a Microsoft update (at least to get some heat off temporarily) even if its not true.
gskv@reddit
Usually schedule tuesdays or wednesdays. Having a work week, vendor support available on phone, and 2 additional work days for staff to catch up on their work on improved systems and learn (if there’s new changes) is highly beneficial.
I avoid Mondays because people forget how to turn on their monitors. And faFo Friday is a lesson learned one too many times.
Xaphios@reddit
I'd say you have two wrong answers there. Pick Tuesday, Wednesday, or Thursday for any big changes. Probably Thursday as most people are winding down on Friday so a bit of lost time doesn't have the same impact.
arcimbo1do@reddit
If the company doesn't pay overtime: Monday: reminder we'll change prod Tuesday: migration Wednesday: Mitigation Thursday: finishing touches Friday: reminder we changed prod (then, beer)
If the company does pay overtime to all the teams involved in the migration: whenever is causing the least amount of trouble to the business.
TreborG2@reddit
It really does have to depend on what the change is.
Let's say you're changing mail providers, making changes to DNS, etc, it has to be at an off-peak time, and overnight doesn't cut it because it can take anywhere from a few minutes to 72 hours for changes to replicate throughout the internet.
Stuff like that is best done on a Friday evening so that if there are problems detected, you have that window of time that is not business critical for you.
When it comes to in office networking, you can generally have your old product sitting there and configure the new product in place and then either gradually start switching over, or have a hot cut over, but leaving everything else intact so if something fails you can switch back. That's a local immediate change and can happen anytime.
Anything that requires time to propagate has to be done in a window that would minimize downtime as seen externally.
Beautiful_Duty_9854@reddit
I'm a read-only Friday kind of man. I prefer to do my big changes on week nights.
tranceandsoul@reddit
Me too. Read only fridays is the way to go here.
MickCollins@reddit
Monday is cleanup for stupid shit that happened over the weekend, like at least half the users who reset their passwords forgetting them two days later.
Friday is for Read Only and No Meetings. Whenever someone's like "I have Friday free" I'm like "that's nice, let's meet Monday." Especially vendors and project managers.
ElveTaz@reddit
Thursday night then fix Friday if needed. If it takes longer then you now have the weekend to get it fixed by Monday if necessary
Sokanas@reddit
we usually do Sunday mornings due to our near 24/7 operations. Its the only window of the week when folks generally aren't working for a few hours so we have generally available period for downtime.
EmotionalVegetable48@reddit
Friday. Shitty I know. We signed up for this.
ButtThunder@reddit
Wednesday for low risk. Friday/weekend for downtime or high risk. Depends on the env though and who has to fix it when it breaks.
tuxsmouf@reddit
We never do this on friday because if we need some external help (from an editor for example) we have no one or a limited support until next Monday. And unofficially, we all want a quiet weekend.
That said, when we were only 2 IT guys, we Always did it saturday morning so we had all the week-end in case of troubles before everyone comes back and I never needed to call help. It felt better than working in night in the middle of the week and be available the next morning at 8 am.
Disastrous_Yam_1410@reddit
We target Thursdays because if the shit hits the fan we only lose one production day. And usually things go fine so weekend is protected either way.
gearcontrol@reddit
The only time it was Friday (or weekends) was if there was expected downtime or degradation, or a customer requested, and key people/stakeholders were available to support.
Apprehensive-Oil-890@reddit
I usually try to do office work in office time. After so many years in the industry you get to know how expendable you are so respect your time, it's YOURS.
Simple-Kaleidoscope4@reddit
Depends on the impact to the business and if they pay overtime. Its money.
Like it or not most big infra changes happen over a weekend.
If you are not paid overtime take the time back asap on a tuesday religiously as time in leu. Or they will take advantage.
bend_forward5639@reddit
I generally follow these basic rules. 1. No major changes on Mondays or Fridays 2. Don't take any new calls the first 30 and last 30 minutes of your day 3. Don't take any new calls 30 minutes before or after lunch breaks 4. Take your mandatory 15 minute breaks, away from your desk preferably 2x a day 5. Wednesdays are reserved for new buildouts or implementations. But the rest of the week is game : )
LittleEarBigEar@reddit
Depends if you need to do it without business hour support. Length of time to do the job, if it takes more than 16 hours you kind of have to do it Friday.
You didn't leave much info to make a decision.
Every change no matter how large can always be undone if done with care and a plan.
So make the call based on that. Both answers monday or Friday are correct.
Assumeweknow@reddit
For that stuff, do it on the weekend. For stuff that requires outside vendor support(monday or thursday).
CommunicationNo2660@reddit
Monday
dc88228@reddit
Read-Only Friday. You lose too many work-hours by not doing changes on Mondays. I’m all about cranking out work Monday-Thursday. I triple-dog dare you to even send me a meeting invite for anything after lunch on Friday. I won’t be attending your meeting
Dank_sniggity@reddit
Depends. Most of our clientele is mon-fri, we did have to start at 2am that sat so the bar could do cash out tho. Makes sense to do jobs like that one on the weekend.
Tscotty223@reddit
Making changes on a Friday is great if you plan to stay at work all weekend without going home.
fusiturns@reddit
Words to live by my friend, no change Friday Monday. My old boss used to say.. you do changes on Friday are you going to ruin your weekend to fix it? And Monday is to be set aside for anything that broke over the weekend including nursing the handover you have ... lol
Decantus@reddit
We're porting our entire phone system over tomorrow. Our annual contract with our current provider would renew on Tuesday and the increase is absurd so I have no choice.
Wish me luck.
transham@reddit
Where I work, we try to avoid scheduling things for Mondays, especially Monday mornings. That's the day we get the most calls for login issues, and nearly half of all hardware failures we see are reported on Monday. We don't want to add troubleshooting or handholding users on a little change at the same time....
kerosene31@reddit
I'm not saying it is right, but one reason for the no Monday/Friday changes is that those are vacation days for people, so statistically fewer people are in the office. Again I'm not saying this should matter, assuming you know who you need is there, but that's the logic. Personally, I never understood Monday being a no-change day.
The thing about the weekend is:
-Is everyone on your IT staff available? I've been on upgrades where there's a dedicated team, but maybe some weird issue comes up and you need a person who's not scheduled on call. Then you have to track them down and hope they aren't out camping.
-Is the vendor as available? Something goes wrong and you need vendor support, the weekend crew is never the best team. Vendor support is bad enough without getting whoever is stuck on the weekend.
-No customters to test.
It obviously depends on the change and the team. Maybe you know everyone you need is available. Maybe the vendor has good support.
It should be a discussion and not just a hard rule. Sometimes, weekend just works. There's a lot of factors.
artimaticus8@reddit
It depends a lot on the company and how big of an outage.
At a previous company, Monday thru Thursday nights were most mtce windows, with large lift-and-shift type projects reserved for a scheduled weekend so we would have enough hours if need be. Really large projects were held for 3-day weekends just in case. This was an 8-5, M-F manufacturing company.
Currently, I work for a credit union, and Friday is traditionally our busiest day (payday, people coming in and cashing or withdrawing paychecks). Some of our branches are also open Saturday mornings, so we pretty heavily enforce Readonly Friday, and our mtce windows are M/Tu/W nights, because we don’t want to even break something on a Thursday night and affect Fridays.
garthoz@reddit
Depends on support contracts. Thursday is what I do.
HoosierLarry@reddit
The goal should be to implement a change as transparently as possible to the user. This means no down time or the minimum required. It also means as bug free and error free as possible.
It depends on your risk tolerance, rollback capabilities, lab rehearsal abilities, and vendor support options. Have a known and tested rollback method (and know how long it takes). Rehearse, test, and validate in the lab. Know your vendor support hours, contact information, and account number or whatever they will ask of you to initiate support. Know what their support capabilities are and the SLA.
slashinhobo1@reddit
Never do it on a friday. Shit i dont even do small changes on a friday unless required. Monday evening is fair game.
TheShmoe13@reddit
Thursday
TheRealLambardi@reddit
Do Friday afternoons at your own peril. Unless forced I advise and hide against it. My number #1 reasons that usually works. If things go sideways we are calling you for both assistance, support, comms so please make sure you inform employees the might be needed and to not do anything in their personal life that would preclude them from working. Like having a drink as it will impede recovery
Va1crist@reddit
Hell no on Friday , I call it read only fridays for a reason
ChiefBroady@reddit
We have no-change fridays. One team ignored it and had to spend their weekends fixing shit because no one was available for proper testing. We’re not doing that.
thejohnykat@reddit
Always on Monday. End of the week is ROF. Read Only Friday.
Shimitzu1@reddit
I worked in a trading company and since the market is closed every weekend vast majority of changes, upgrades and flips were done over the weekend.
Now I work in production company, since factories are working 24/7 (starting the machines is 3x more expensive than running it over the weekend) I need to avoid anything on Fridays. I prefer to implement stuff mid day Monday (since I implement stuff for US and Brasil) I want local support to be at least after their breakfast and coffee before flipping their switches.
kaka8miranda@reddit
Never in my life will I make a Friday change again.
I have 0 desire to lose my weekend would rather bring down critical projects Monday then go in Saturday/Sunday to fix shit
marquiso@reddit
I used to have very strong opinions on this and I guess I still do, only different…
You should architect and run your network for RESILIENCE, so that any screw ups are quickly and easily recoverable/wound back.
Then the specific timing doesn’t matter.
So whether you have an outage due to a change gone wrong, or due to a ransomware incident, you can still recover quickly.
I’ll admit that this is much easier in theory than reality.
SlightAnnoyance@reddit
Friday changes are only for changes with meaningful expected user downtime. Then I can declare "it's a weekend, so you're welcome". Unless we're talking about anticipating 8+hrs of downtime though, that Friday change becomes a Sunday change so I can at least have my Saturday.
Everything else is Monday and if it hits the fan, roll back and apologize with a "thankfully we were able to recover services quickly, so you're welcome."
Capital-Fall5471@reddit
Never make a change on a friday.
jasmeralia@reddit
I'm a firm believer in Read Only Fridays. But we do monthly maintenance for our legacy environment at 11pm (local to the region, so UTC & PST) on the 4th Friday of most months because that's off peak hours doing things we know are potentially likely to disrupt customers, and that's the schedule we have arranged with them in the past. That's not for any application releases, though; that's for patching (with reboots), firewall/load balancer/VPN server/file sharing appliance upgrades, failover testing, that kind of thing. I don't entirely agree with that choice (if it needs to be on a weekend, I'd prefer Sundays so that we can react quicker if something goes sideways and we don't catch it right away), but it's in the existing contracts, so not my bailiwick. One of the guys that reports to me handles the EMEA side of what my team supports, and I handle the AMER side. (And I won't complain too much because I just work a split shift... I sign off around 10am and try to nap in between, rather than needing to sign in on a day that I don't normally work at all.) Looking forward to the eventual sunset of that environment where I don't have to deal with monthly maintenance work like that any more.
Derpolium@reddit
Only on days where critical operations and meetings are happening and always with at least 17 hours of extended/unscheduled downtime
Wise-Butterfly-6546@reddit
Neither. Wednesday.
Monday you're still putting out fires from whatever broke over the weekend. Friday you're gambling your entire weekend on a rollback that "should be quick." Both options assume the change will either work perfectly or fail cleanly, and anyone with 15+ years knows that the worst failures are the ones that look like they worked until 3 hours later when something downstream quietly breaks.
Wednesday gives you two full business days of buffer in either direction. If something goes sideways Tuesday night / Wednesday morning, you have Thursday and Friday with full staff to troubleshoot before the weekend. If you need to roll back, you're not doing it at 11pm on a Friday with half the team already checked out.
The real answer though is that it depends on your change management maturity. If you have solid rollback procedures, tested in staging, with clear success criteria and a defined rollback window -- the day matters less. If your change plan is "we'll figure it out" then no day is safe.
Also -- the framing of "Monday so we have the rest of the week" assumes things break immediately. The nastiest infrastructure failures are the slow-burn ones. Changed a firewall rule on Monday? Everything looks fine until Thursday when some batch job that only runs weekly tries to hit an endpoint that's now blocked. That's when you realize you have no idea what changed because it was 4 days ago and nobody connected the dots.
Document everything. Test rollbacks before you need them. And do it on a Wednesday.
dwarftosser77@reddit
Always on Fridays.
No-Swan4213@reddit
Never do changes on a Friday or Sunday…
i8noodles@reddit
my company allows changes on weekends because, as a true 24/7 business, there isnt actually a good time to do changes. even at 2 or 3am things still happens every day even on fridays.
i work for a large enough company that we have basically a full team working every single hour of the day for every day.
we still try to limit it to the weekday but sometimes it cant be done so it goes onto the fridays or even weekends.
hisheeraz@reddit
25 year plus as an MSP. We always do it on Friday incase things don’t roll out as planned. Might be different for in house IT
IntelligentComment@reddit
Depends on what the task is and if it's a regular occurrence , if your org operates Monday to Friday, etc.. Sometimes weekend work is part of the job but work life balance is important.
rossumcapek@reddit
https://isitreadonlyfriday.com/ exists for a reason.
DharmaPolice@reddit
It really depends on the change and whether 3rd party support might be needed (even indirect support like getting access to a building or something). How much downtime? Is overtime being paid?
Monday feels like a bad choice regardless though. There might be problems from the weekend and in a lot of places Monday can be busy for other reasons (e.g. call centres are often busy on a Monday). Tuesday would be a better choice, an extra day is rarely going to matter and you're in a better place to confirm everything is OK on the Monday. Rather than planning to start on Monday and then realising someone is sick when you get in.
But if we're talking about a change that's 100% down to me, I prefer to work on it on a Friday night with the buffer of Saturday and even Sunday if there's issues. I know I'll be compensated if I do need to do that. And it's less stressful.
But like I say, it depends. In reality it's rare that a big change is going to be down to just me.
AmountOk1689@reddit
We call it no change fridays.
carbon_brz@reddit
Friday. If it fails you have the largest time to recover, and your users are probably not on (or not as many)
Smaller releases, depending on the risk, weds/thurs
TheOzarkWizard@reddit
I think it depends on the users and workload.
If the system or important applications go down, will the company still be able to operate? You want the weekend for safety factor.
Would it be better to have the users present, to troubleshoot problems with software changes/breaks etc, or do you want to have everyone show up on monday and find out a bunch of things dont work?
Is the workload sensitive or important? Maybe do it on a Friday so you have the weekend to ensure everything is back up.
I dont think theres one easy answer for everyone, other than "it depends"
stumpymcgrumpy@reddit
With proper change controls and testing in dev and staging environments you should be able to feel confident about the success of the rollout to prod. If You can Architect your prod environments for minimal down times and quick roll backs then there really aren't any good reasons not to push to prod during business hours.
wengla02@reddit
Public facing changes were always made at 12:01 AM Monday; if it's going to be a longer (8 hour) outage, 12:01 AM Sunday. Major telecom sales and service website. Lowest sales and service traffic Sunday through 8AM Monday, so lowest impact to the retail customer.
Internal changes were typically Wednesday. I assumed that had something to do with patch Tuesday cadence; I donno - I was external facing only.
dblock1887@reddit
no... change..friday
No_Interest_5818@reddit
Wednesday/Thursday. Enough time for change to take effect, users to report bugs, and the weekend if necessary to fix it.
DeadStockWalking@reddit
Tuesday is the correct answer.
madknives23@reddit
Read only Friday is my motto, I push changes on Tuesdays
Stryker1-1@reddit
I've been on both sides of this.
As a previous field tech who used to do the physical cut overs and remove old gear and stack new gear a Friday night is best since most places dont operate on the weekend. This gave us the most time to do the cut over and address issues.
Now being on the IT side where I work Monday thru Friday I cringe any time someone wants to do a cut over on a Friday.
Nexthink_Quentin@reddit
Definitely early week for anything this big. If anything goes wrong you want full visibility, full staffing, and full vendor support. Over the weekend you’ll just have a skeleton crew and also won’t actually know how it’s affecting users until everyone logs in Monday. In my experience this just turns into a headache on Monday anyway.
A huge change like that could look fine over the weekend and then everyone logs in Monday and the whole thing falls apart.
CoffeeAcceptable_@reddit
If its after 10am on Friday, come back Monday..
SantasDog101@reddit
No change friday!
nuttertools@reddit
Those are the two days I would never allow a major change on. On the administrative side that sounds great, for users it’s a disaster waiting to happen.
rybread761@reddit
If you want to do the change on a Friday, you better be the one on call that weekend.
Alex4453@reddit
Neither one. Mondays and fridays are read only.
2c0@reddit
Fridays + weekends for big changes but I'm not coming in Monday or Tuesday.
NukaColaQuantum2077@reddit
I’m a Tuesday or Wednesday guy myself. Never on Monday’s as that is catch up day from the weekend and never on Friday’s because I like my weekends.
In the past, other IT admins have screwed up my weekends. I’ll never do that to myself or others because I love my personal time and respect others peoples time off.
BBO1007@reddit
Monday if there is an inkling of a chance you’ll need outside support.
Fridays if you hate not working weekends.
You are welcome.
Crazy-Rest5026@reddit
Only Fridays if you are compensated for weekend pay. Generally, I like Monday, For this exact reason. You’re getting paid to fix it during the hours you work.
If you are not being compensated on the weekend then fuck no. I don’t work for free boss man
Expensive_Plant_9530@reddit
We have a rule. No big changes or projects implemented on Monday or Friday.
We’ve had far too many instances where a Friday implementation resulted in chaos over the weekend and us scrambling on Monday to fix.
On Monday we are always short staffed due to Saturday IT coverage so it’s just a bad day to make changes too.
highroller038@reddit
Neither. Tuesday or Wednesday evenings are good. Mondays are too busy with break/fix and meetings.
QuietBookkeeper4712@reddit
Read-Only Friday
Dimzy5150@reddit
Mondays, maybe, Fridays are definitely a no-no unless its an emergency that cant wait until monday.
Rad_Dad6969@reddit
Tuesday.
Monday is for prep and finishing what you didnt on Friday.
vadavea@reddit
we'll do a lot of our changes Thursday, thinking being that many folks take Fridays off, so the blast radius is smaller, but also that other infra teams are available to assist on Friday (vice having to call them in on weekends, which definitely isn't good.) It's often the case that things go south because of some unrecognized dependency that needs involvement from (DNS/Firewalls/AD/etc...).
KerryFatAssBro@reddit
We have a rule of read only Friday's where I'm at but Monday is free game because we have the whole week to iron out any issues.
birchhead@reddit
ReadOnly Friday!
Primary_Excuse_7183@reddit
Monday.
Unless you hate your weekends
arslearsle@reddit
Never do major changes or changes at all on fridays… unless you like to have your weekend ruined.
Another story if your customers like to pay meals and double-triple pay for weekendwork - but you prob know thats not going to happen.
lowlybananas@reddit
Anyone who does big IT changes on a Friday is a psychopath
4wheels6pack@reddit
See this is where I deviate from the common consensus... I prefer doing changes on Friday, because IF things go sideways, I have the weekend to fix it before Monday comes rolling in.
If I do changes on Monday, unexpected cow-pie's have a much worse impact when people are actively trying to work.
loupgarou21@reddit
At my current job, where I don't work weekends and am not authorized for overtime, I rarely make big changes on Fridays. At my last job, where I worked for an MSP, my clients loved when I would do work after-hours on Fridays because it meant if something went wrong, I had all weekend to fix it and it wouldn't negatively effect their employees.
I don't typically plan big changes for Mondays, just because I end up with a lot of high priority things that pop up on Mondays.
serverhorror@reddit
Every day. But I prefer Friday and still leave for the weekend. Repeat this daily until your changes are stable and small enough to be confident that they go finish without any problems.
R0gu3tr4d3r@reddit
No changes on a Friday ever. Unless, it's a really big programme change, then we schedule a holiday weekend like Easter. It sucks, but that's why we get paid well in IT and we get the time back+. Source..I run CAB.
Competitive_Smoke948@reddit
NEVER EVER EVER TOUCH ANYTHING ON A FRIDAY!!! I have spent 20 + years with this rule... project managers very quickly STFU when I tell them I don't care HOW important something is or what the promised; I'm only coming in for double time & i'm not cancelling plans with the people who would actually visit me in hospital if I a work related heart attack.
unless it's booked in months in advance
shell_shocked_today@reddit
I vote against Friday. I've seen too many changes on Fridays botched because people were in a rush trying to finish so they can start their weekend.
davy_crockett_slayer@reddit
Monday. Never Friday unless you absolutely have to.
SousVideAndSmoke@reddit
Read Only Friday
rankinrez@reddit
Monday. But it all depends what your business is. Maybe if nothing is needed at weekends Friday makes sense.
Anywhere I’ve ever worked is 24/7/365 in terms of when it needs to work so Monday so people can respond.
Pub1ius@reddit
I typically don't do Friday OR Monday. Not Friday because I'm not spending the weekend fixing a botched deployment. Not Monday because inevitably something will have happened over the weekend that needs to be fixed immediately on Monday and\or the workload on a Monday tends to be higher.
Assuming the project is all planned, prerequisites checked, and we're ready to rock, I'd go for Tuesday. Then if shit hits the fan you fix it on Wednesday (with Thursday also there if you need it). That way I'm still out the door early on Friday to enjoy my weekend.
No_Yesterday_3260@reddit
Depends on the size and severity.
Small changes here and there, like a simple firewall update for 1 firewall, server windows updates - I prefer doing it so I have a normal work day.
But for MAJOR changes, I prefer doing it on weekend, so there's time to test, and impact production as little as possible.
midwestbikerider@reddit
Read-Only Friday is not a meme.
cbowers@reddit
It depends on the production criticality for the affected system. Best practice, it's a parallel system, not critical path. Then yeah, Monday. But if your outage window costs the business... if you're ever asked to justify your business model in the aftermath of an outage... hopefully you've already built that and got business buy-in. Most of my major outages (unplanned) mercifully picked the weekend on their own. We had some spectacular all-nighter/all-weekend saves where everyone came in 7am Monday morning to 100% systems up, and weren't affected at all. It would have been significantly more costly if they'd happened on a Monday.
To the point of cost of impact and business buy-in, and my last company, this process even drove major patching windows and reboots not even on Friday, but to Saturday morning. It was cheaper to pay on-call bonuses to on-prem and cloud IT teams than it was to have even a reboot outage during Monday-Friday.
boli99@reddit
If management see you working one weekend, then they'll push you to work all weekends.
...so best to do it during the week. Run a courtesy call round all departments to make sure they arent in the middle of an audit or tax report or something else time critical
then do the work during working hours. because that why they're called working hours.
3ShrimpTacos@reddit
We have a fairly strict "No Change Fridays" Rule here. Nobody wants to work the weekend if we don't have to.
Unable-Entrance3110@reddit
BMCBoid@reddit
No Major Changes On Fridays!
Let's say you break something bad. One, there goes your weekend. Two, over the weekend, you are going to only have the on call support for vendors available to help fix. Also let's say you do get it fixed Friday night....You can test, but you won't get the full load of production until Monday anyway.
cayosonia@reddit
We used to open Saturday morning so all risky changes would be Saturday afternoon. If it all went sideways you'd have Saturday afternoon to figure it out. If you couldn't figure it out you could roll back/restore Sunday
thedudeintx82@reddit
If I had to pick between those 2 options, Monday.
However, my preferred day was Thursday. The likelihood that what we did won't take more than a day to fix and we still keep our weekend.
For my 26 year career, we always did major changes on Thursday.
rosseloh@reddit
At my local plant it goes like this:
Makes the decision really easy for me.
Break2FixIT@reddit
I always lived by, Hair of the dog Mondays, patch Tuesdays, whiskey Wednesdays, thirsty Thursday's and fun Fridays (read only Fridays).. I don't make any changes... Ever
wrootlt@reddit
Even without considering real pros and cons i am naturally leaning towards Monday. And the older i get the more as i want my weekends to myself.
SportTawk@reddit
Obviously Friday, and take the following week off, that's how it seemed to go where I worked!
viking_linuxbrother@reddit
Never Friday if possible.
That working all weekend if it breaks is a fools game.
Its not worth it even if you think it'll spare people some issues because you are suffering in silence and no one will give you credit for how hard you really worked. You only inconvenienced yourself, for everyone else it didn't happen.
Bratwurst1981@reddit
Friday, with a “fire watch” team to repair what you don’t know was broke before users show up.
TireFryer426@reddit
repeat after me.
Read only Friday.
Blyatman95@reddit
Friday is for if your IT team is willing to work over the weekend to fix it. Monday is for if it’s not. You can come unstuck though if the vendor of the software doesn’t work weekends so you have no one to help if you’re royally boned.
If your company / customer is 24 hours screw it no time is good so you might as well do Monday at 10:00 when the most resources are available to help you fix it.
CFC1985@reddit
It's called "Don't break shit Friday" for a reason so I would strongly advise against any major changes late in the work week and especially not on a Friday.
Common_Suggestion266@reddit
Depends on how big or "likely" and issue is. Id avoid Friday evenings unless it is urgent and/or people power is availble Fri night, sat if needed or people to monitor. Again if you are expecting much downtime then yes weekends, early mornings or late nights. Like rewiring a network closet/data center. Also for any of these depends on the risk and how much testing has been done and I'm assuming positive results. 😁
mindtrix@reddit
Thursday nights
bukkithedd@reddit
I try not to schedule downtime on fridays/weekends, mostly because I don't want to work during those times. Monday is also a bad time to do such things as, since I'm usually busy with making sure nothing went tits-up over the weekend.
Thursday or wednesday are my picks for days it's OK to do major work on, just in case I have to call in vendor support for shit.
That being said, big upgrades in physical core infrastructure (network etc) are things I'll often schedule during the weekend, since that gives me time to figure out just what the absolute hell I was thinking when I cooked together the config needed.
vale-@reddit
Read-Only Fridays!!
Reedy_Whisper_45@reddit
I schedule outages for the weekend and let production know the time or timeframe of the outage. I don't EVER do anything on Friday if I haven't already scheduled the downtime (which includes my personal plans). I don't ever want to come in when I haven't planned it.
Other changes - not planning an outage, I try to schedule for Tuesday or Wednesday. That way I have support that I'm not paying extra for, and I'm not putting in unplanned hours. We moved all our file shares to sharepoint on a weeknight - close your stuff
aModernSage@reddit
IMO - Fear of Friday changes makes me think one may have fragile infrastructure. Additionally, FoF also reminds me of yesteryear when changes were barely if ever automated.
I've lived in both camps and I know that when your IaC, Ci/Cd, and config mgmt are not reliable - Fear of Fridays is very commonplace. It also tends to be common among the older generations who came up through the lack of those processes. EG: an automatic instinct.
However, when those processes exist, and have been thoroughly tested and vetted- then FoF is probably just instinctual and sometimes not based on facts and evidence.
As such, in my current role, I love Fridays because thats when I can make my big changes. Now, in the interest of transparency - my current org just happens to be fully cloud native, so there is that. However, my previous role was a massive hodge-podge of crap accumulated and sustained through decades of work. You name it, we probably some version of it. Even then, I routinely made changes on Fridays and worked to drive the FoF notion out of our org - with mixed results.
In the end, and to sum it all up - do what is reasonable, smart, and justifiable in your org. If the mould is set in stone, save yourself the stress and don't fight it too hard.
Accomplished_Sir_660@reddit
Sure you don't want to work weekends, but the ONLY answer is Friday after COB. That way you have the weekend (ya your weekend) to get it resolved before production on Monday.
Thats the ONLY answer. Don't believe me then ask the business owner who has to pay everyone on Tuesday, Wednesday, etc to do nothing because shit hit the fan on Monday COB.
Welcome to IT.
Mercdecember84@reddit
this depends on the requirements. If you work in financials or banking then you can really only do major work on Fridays
I worked at a medical place that was 24/7 in which all changes can only happen Sundays between 6pm and 8pm
Ztoffels@reddit
Lol wtf is u on? U rather eat shit yourself on a weekend than, fuck the company during their work time?
Thats why yall get fired then get mad the company did so…
CeC-P@reddit (OP)
Bro, right?! I value my time more than a company that doesn't respect me but not when it comes to experimental changes and updates that "should" work and then make us all look overpaid and incompetent to management.
CeC-P@reddit (OP)
My answer: OBVIOUSLY THE ANSWER IS FRIDAY! I want precisely zero outages during the work week. Everyone insisted on large updates going out Monday and kept breaking production systems for Tuesday. They kept insisting that Friday made no sense and left no time. We get overtime in case of emergencies!
I think these lazy former coworkers of mine just didn't want to work on a Saturday. Well, I don't want to work on a Monday night, even if I then get equivalent time off on Friday so I don't go over my hours. It's not about me and I know what job I signed up for! You know what's more important than my schedule? Not breaking the entire Autocad/testing lab/accounting departments for Tuesday and then having to hear about it from management!
accipitradea@reddit
The only argument for not-Friday is if your company doesn't pay you for your time or at the very least doesn't compensate you time-off. At which point you work for a shit company that deserves production downtime.
CeC-P@reddit (OP)
The last co I worked at started a new pay week on Sat so that influences it heavily.
The one before that, the overseas contractors would implement firewall changes mid-day because fuck it, and without a change order. And then do it wrong. And then cause an outage. At a hospital.
PhilsFanDrew@reddit
Sorry I don't agree with you. By Monday I'm already in work mode. I'd much rather have to chip in for a rollback or fix a rollout on a Monday evening into Tuesday morning than sacrifice any part of my weekend for work.
_crayons_@reddit
It really depends on who your customers are.
If it's a major upgrade with risks of production being down and costing business $$$ then it makes sense to do it on over the weekend and have time to fix it.
ParkerPWNT@reddit
"I think these lazy former coworkers of mine just didn't want to work on a Saturday"
No they just want do things properly. You lose so many escalation paths on the weekend.
mixduptransistor@reddit
Not sure if maybe you're outside the US, or where you are on the career ladder but most folks around here are going to be salaried Americans which means no overtime
Breaking things should be rare. Maintenance windows are a thing for a reason, but also if you're breaking something badly every time you do this work something is wrong
mixduptransistor@reddit
Not sure if maybe you're outside the US, or where you are on the career ladder but most folks around here are going to be salaried Americans which means no overtime
AZSystems@reddit
Tolerance to breakage is the metric. Keep the IF, Than, What mentality.
Size and outages should be planned, not carried out. Planning throughout the week as IT is available and caffeinated. I typically have had one weekend in which warning and always the same schedule Noon to Six, even if nothing to patch, expect downtime or malfunction. This was continued harmony between and respecting staff schedules, deadlines, etc.
StigaPower@reddit
Friday all the way 😎
SirLoremIpsum@reddit
With this caveat there is only one real option that reduces the genuine impact to Operating The Business.
Every answer about change windows for big changes is going depend entirely on what the change is and what the impact to the business is and that is personal and specific to your company.
There is no hard and fast rule because of it.
Switchnport@reddit
My team has a policy.
No changes to prod on Fridays unless it’s an absolute emergency.
ZoneEmbarrassed7697@reddit
Read-Only Fridays
Historical_Hunt846@reddit
We do it on Thursday. Friday allows time for the business to let us know if something is broken. If necessary, we have the weekend to fix it. A lot things can wait until Monday unless it is revenue or patient impacting.
kenfury@reddit
Monday is Meetings and any post work cleanup. Tue-Thur is business as usual planning and testing in Dev/UAT, Friday is final CAB signoff. Weekend is change. 1 week per month.
uptimefordays@reddit
It depends on the organization and operational maturity.
absentspace@reddit
My favorite is when a colleague makes a big change on Friday, emails everyone about it around 4:30 PM, then goes on leave for a week.
You know who you are.
ElectrifiedSword@reddit
I'm hourly so if something goes wrong on a Friday deployment, that's guaranteed overtime on the weekend.
Time and a half to fix something I fucked up while it doesn't affect users or management since they don't work on the weekend?
Hell yeah, Friday it is.
DDS-PBS@reddit
Friday Night - Your issue goes undiscovered until Monday Morning
Sunday Night - Same, but you have to work on Sunday
Monday Night - A hard to fix issue can take the business down for days
Thursday Night - Your issue is discovered Friday, you only take the business down for a day, and you have the weekend to fix it if it takes more than a day
Thursday night is optimal
JohnnyUtah41@reddit
I usually only schedule things around my gym and jiu jitsu schedule, Those come first, stupid job can eat it
2_Spicy_2_Impeach@reddit
Depends on what kind of change but if you’re worried about multi-day outages/degradation so you need the weekend, you’re doing it wrong.
With that said if you’re regularly stuff on Friday, you’re going to burn folks out when the fallout happens on the weekend.
Eons ago we did Tuesday and Thursday. Fridays always sucked.
paperellablu@reddit
it works, why changes? 🤪
juggy_11@reddit
After business hours during the week is the correct answer.
Warrlock608@reddit
Read-only Friday should be an industry wide standard.
borse2008@reddit
Have a back out plan revet firmware or changed due to circumstances. Just make sure you look at the risk levels or see if you can breakdown the risk by replacing so many one week and some another time. Make sure if your taking the network down communication is key to the end users. Ie working from home is unaffected if using cloud services. Just make sure it's communicated
bilingual-german@reddit
If these are the only options, I think it would depend on the impact on people working vs. the money, family, recreation on the weekend is important.
If people might be really impacted and it's paid, etc. I would do stuff on Friday. If only minor issues could pop up, I would rather do it on a Monday.
For migrations, etc I prefer Tuesday to Thurday mornings. I would also prefer weekend mornings over Friday evenings.
DirkyC@reddit
We are Read only Fridays except for major projects like server migrations or data centre moves.
snookajab@reddit
Read only Friday. Don’t want to lose your weekend.
-The-Babushka-@reddit
Ive worked at two companies, both large. One did it on fridays, one did it early In the week.
Take a guess as which company fosters a bad work/life balance, I’m sure it’s easy.
As stated by many other, you only commit big changes on Friday if you are prepared for your techs to work through the weekend.
Shineyjo0326@reddit
Care about the team doing the project? Monday is the answer.
Care about the companies bottom line? Friday is the answer.
Depends where you are in the pecking order.
bit0n@reddit
Switching to a new host or switching networking is overtime Saturday and Sunday so users are not impacted.
An upgrade to an app is a Tuesday morning thing with Vendor support on standby and a roll back plan.
Lekanswanson@reddit
Trust me your best testers are regular users. If there's a way to break something in the new config they will find it
Just make sure that you are able to revert everything if something goes terribly wrong.
bacon59@reddit
Eod monday. Im solo dept and salary, not giving up my weekend.
BlazeFireHorse76@reddit
No change Friday checking in
proudcanadianeh@reddit
For my environment, weekends are lower impact on day to day operations and are immediately double time. Weekdays if things break people get unhappy fast, and OT is only time and a half.
My preference is come in on the weekend, bank the double time then take a few days off elsewhere. Everybody wins that way.
Busy-Photograph4803@reddit
READ ONLY FRIDAYS. Embrace it
himji@reddit
I like to do it when I know there will be users around to tell me it's gone wrong. no matter how much testing you do, better to have users around to tell you when something critical is broken.
I avoid Mondays as that's usually most people's busy day so normally Tuesday evenings the best time to make changes
I_ride_ostriches@reddit
I think it depends on the nature of the change. Networking gear, typically don’t want to do that during peak hours/outside of a maintenance window. Mailbox configuration? Right in the middle of the day.
techtornado@reddit
Grasshopper, we have read only Friday as a mandatory thing
ludlology@reddit
Read only fridays my dude. If you get pushback from a PHB, tell them support is closed after hours
LeaningFaithward@reddit
I prefer Thursday so that we have a full work day to resolve issues or to rollback if needed. If the changes are problematic with major issues, we are rolling back after a few hours; not days.
Neither_Meaning_8354@reddit
Ich mach sowas immer freitags weil ich dann IN RUHE mein ganzes Wochenende habe um es zu beheben ohne dass mir nervöse User auf die Eierstöcke gehen.
kiddj1@reddit
If you pick option 2 you have no life
Waylander0719@reddit
It depends on the change and if live testing and validataion can be done soley by IT or needs users to bang on it to validate.
For example a server migration or network closet change I would do on the weekend. A push of a software update to end user workstations I would do on Monday.
mkaibear@reddit
Doing it on Fridays is fine if you've written off the whole weekend to fix it.
I've worked places where we've deliberately done this because it would impact law enforcement critical services if we did it in the week.
For anything not absolutely 100% business critical lose your weekend... No Change Fridays. Every. Single. Time.
Trust_8067@reddit
Why would you want to ruin your entire weekend?
Also, you shouldn't be doing "big IT changes" to being with, without making those changes in dev and staging first, so you know what the expected outcome in your environment is. Though I know most IT infrastructures are complete shit, and people push straight to prod.
Zerowig@reddit
OP is going to suggest Friday’s is the only time for organizations that don’t operate on weekends.
However, good luck getting the “A team” from 3rd party support organizations on weekends if something goes wrong.
ghjm@reddit
If IT workers can take Monday as their weekend day if they wind up working all day Saturday, then 2.
If the company expects weekend work to be just uncompensated hours, then fuck 'em and 1.
tito2323@reddit
Read only Fridays, patch servers on the weekend.
PoisonIvyToiletPaper@reddit
Thursdays, and in the afternoons/evenings. This way, we have 1 production day (Fridays) for any troubleshooting and usually everyone is a bit more relaxed if something did happen to break. We are also not-for-profit and don't make things, so there's less stress anyway.
ShutYourSwitchport@reddit
I would always choose EOD Monday going into early morning Tuesday. You keep your weekend, and upper keeps their busy Monday.
matroosoft@reddit
For the business it's better to do it on Fridays so you can fix broken stuff without downtime. For you it's better to do it on Mondays so you keep your weekend.
LowMight3045@reddit
Big stuff with user impact is always on the weekend. Not my choice . If I had a choice , Tuesday at 10 am . End users won’t be impacted
Affectionate-Cat-975@reddit
Make changes Thursday nights.
Mondays are Meetings
Tuesday through Thursday is work
Friday is clean up and prep for the weekend/time off.
Doing the change Thrusday night only risks F'ing up 1 day of production and you get real world impact of the change. If you change Friday night/weekend then you don't have the real world impact execution until Monday. Whereas if you jack up things on a Friday, some/most staff won't mind an easy coast in to the weekend.
caponewgp420@reddit
Read only Fridays for a reason.
olinwalnut@reddit
Read only Friday.
Possible-Shelter-800@reddit
Friday
CharlieTecho@reddit
Bank holiday is fair game as long as people are being paid and willing to work!
Advanced_Day8657@reddit
You wanna work all weekend for free? Wtf
8923ns671@reddit
We do it on weekdays overnight (or whatever off-peak hours are for that particular customer/application). I prefer that over losing my weekend.
Fritzo2162@reddit
NEVER Friday, unless you like tying up your weekends and people hating you Monday.
KillingTime1212@reddit
It’s an all a crapshoot. Do it whenever you want.
BCIT_Richard@reddit
Monday, Don't be like Cloudstrike and brick workstations on a Friday, lol
Forgotmyaccount1979@reddit
It depends on the business.
My place of work, our weekends are the busiest times for customers.
So we have no change Fridays/fuck off Fridays.
Personally, I like to schedule stuff for Tuesdays. It prevents weekend issues and Monday surprises from interrupting the planned work, but also gives plenty of week to tidy stuff up.
jsand2@reddit
Fridays are a no go for us. We dont want to work our whole weekend.
We replicate all our servers, so rollbacks are super simple if needed. We just do during business hours or late in the day.
fishymutt@reddit
Fridays are only for important bug fixes or big projects like migrations. If you ask me to do a release on a Friday you better have a good reason.
techdog19@reddit
Always avoid Friday who wants to get stuck working all weekend.
TheCrippledChicken@reddit
“Don’t F around Friday…” is preferred. I don’t want to spend all weekend fixing your fck ups.
tbone0785@reddit
All significant changes done on Tuesday nights. If you have to bring in coworkers from different tiers/groups, you don't want to do that on a weekend. Clear instructions for a rollback procedure if things go south. Pre-activity documentation of all configs.
Tex-Rob@reddit
Whenever possible, sort of neither, but sort of Monday with a big asterisk. Avoiding non-seamless transitions is the best way to go whenever you can, but often cost makes it not the choice your org will go with. A lot of what I did and was good at was these seamless transitions for clients.
JustlyDues@reddit
Monday is planning, verifying, and organizing for the week, changes are done Tues-Thurs, and documentation is double checked on Fridays for the week's work.
If your organization doesn't value IT work as much as the rest of operations, and thinks IT should scurry about in the night, then you probably work at a bad company.
With the amount of services offloaded to outside organizations like cloudflare, Microsoft, etc, you're likely going to face daytime outages anyways outside of your control. Why focus your life around work when others you depend on aren't?
jocke92@reddit
If you want to work on the weekend and get paid enough I don't think Fridays are read only. Usually business activities are low on the weekend. You might have all weekend fixing it before it's an issue. You might not have to rush and could resolve the issue relaxed.
If you break something Monday morning you have to rush getting the system back online. As the outage is causing issues for business.
One important/interesting thought when doing something that should be safe and not causing any issues. It might have unforeseen consequences. Something you or even anyone else could imagine. If it's on a Friday it might cause issues as the right technicians are offline rather than online on a weekday.
Wartz@reddit
See if you can figure out how to break down your big change into smaller less impactful changes.
TechIncarnate4@reddit
What do you mean "rest of the work week to fix it"? Your organization can afford to have issues or downtime all week while you fix something you broke on Monday night?
We have always done it on Fridays because there are less people working over the weekend that would be impacted. This gives us the weekend to fix anything or revert the changes before Monday. We typically avoid holidays so people can enjoy their time off.
AdventurousLayer8741@reddit
Read-only Fridays should be the rule. Hard to get tech support on weekends if needed, and I need some peaceful time off ffs. Ideally it would be read-only Thursdays because we only do 4 day workweeks.
Antique_Grapefruit_5@reddit
I run a hospital IT department and we do a lot of changes late Thursday night/Really Early Friday morning. I know it's a bit different for us because we never really shut down, but patient volumes are typically lower Fridays while still have the external support (and people to test) that we need. Typically my staff is in bed by 9am and starts their weekend early. It's the best of the evils we have... ..
Sufficient-House1722@reddit
One time i tried to do it on a Wednesday night after hours.
Took entire network down.
Couldn't fix it
didn't get it fixed until right before when i was supposed to work before the next day
worked until 11 AM making sure it was all good and took off the rest of the week.
Havnt made a major change since.
robotbeatrally@reddit
Friday if your team is willing to work the weekend. I hate working weekends. Monday for me. lol
Acheronian_Rose@reddit
Our team has done some projects/cut overs on a Friday to reduce the impact to production (we run a first and second shift every day but Friday, its first shift only).
However when we do this, we always make sure we have things like vendor support, power users to test said changes that evening or saturday morning, and a rollback plan so we don't have to lose the entire weekend if we encounter problems that cannot be solved relatively quickly.
Our O365 migration got pushed several times due to discovering issues, but we always had a rollback plan to swap back to our on premise server, so we only lost part of a Friday evening, which IMO is what it is for major projects time to time
rire0001@reddit
Never schedule a major update for a work day. Period. Dumbest thing on the planet. Unless you simply don't care about your customer base, and the organization's productivity, or the company's profits.
Start Saturday morning. Have manageable checkpoints. Set a fallback time - and have a valid fallback plan written out. Line up support people before hand - make sure critical infrastructure support is available. And testees from your business community. Order pizza. Make it a little party.
Failure to plan is planning to fail. I can't think of how many 'engineers' and vendors I've sent back to the drawing board when they want to shoot from the hip.
cptNarnia@reddit
Thursday PM, Fix it Friday
MikeJC411@reddit
Depending on the organization and what is provided by the services and how the company makes money. Customer impact should be the #1 consideration for maintenance and change windows. User impact and productivity is number #2. Number 1 has the biggest potential for financial impact. Does downtime affect the companies ability to produce income? The customer interfaces are up but the employees cant do anything with it like process an order?
The last thing is really the impact to IT and hours. It's a tough one to really come to terms with, but a reality of IT work. Additional hours of unplanned work from failed rollouts has alot of negative impact to IT productivity.
Do the changes when the impact is minimized. Plan plan plan and have a solid roll back plan with a hard window of acceptable downtime. A good strategy is redundant or load balanced systems where you can upgrade the passive side cut it in then update the other. Minimizes downtime if there are problems.
West_Acanthaceae5032@reddit
We deploy on Fridays...what could possibly go wrong?
DestinyForNone@reddit
Fridays for major infrastructure changes. Mondays for everything else.
Essentially, anything with an estimated outage window that won't cross into the next business day... Assuming things need to be reverted.
mariachiodin@reddit
Tuesday
matroosoft@reddit
For the business it's better to do it on Fridays so you can fix broken stuff without downtime. For you it's better to do it on Mondays so you keep your weekend.
matroosoft@reddit
For the business it's better to do it on Fridays so you can fix broken stuff without downtime. For you it's better to do it in Mondays so you keep your weekend.
The_Struggle_Man@reddit
Ready-only Friday. Not debate.
Warsum@reddit
NO. CHANGE. FRIDAY. (This also applies to a Thursday before holiday weekend. Fuck off.)
zedsdead79@reddit
Where I'm at, only changes allowed on a Friday (during business hours, 7am to 5pm) are for break/fix. Nada on the weekends. All other changes can be during business hours during the week if there is no impact due to redundancy or it's a very tiny impact, otherwise overnight all the way.
TheDawiWhisperer@reddit
Sometimes shit needs doing on a Friday.
Read only Friday is a myth perpetuated by people that don't work in the real world
mwskibumb@reddit
If you like working weekends fridays cool.
gwig9@reddit
My rule of thumb is no big changes on Friday... I want to have a weekend.
IF there is no other way and I HAVE to do a change on Friday, I plan for the weekend to be working. Always be prepared for the worst. That way you're never disappointed...
Thundahead@reddit
obviously 2,
if you do 1 and it goes wrong what are the business implications if everything is down
fnordhole@reddit
Divorce much?
ParkerPWNT@reddit
Hard disagree.
My business operates 24/7.
I have less support internal and from vendors on weekends.
Thundahead@reddit
I used to work in a 24/7 environment and these would be done on a quarterly powerdown weekend, if you core business is Monday to Friday which I guess from what the OP posts you don't trash production systems through the week
apandaze@reddit
i agree, on friday - whole weekend to fix it or roll back so on monday no one knows the difference
Due_Peak_6428@reddit
If you have weekend workers then Friday is ideal as you say
No_Dog9530@reddit
If a business has a solid DR and can sustain with downtime of the device, then it’s absolutely fine.
Competitive-Round-90@reddit
Rolling over to DR for an outage due to a planned change sounds like a bad BCP. There should be other rollbacks and control measures that could be utilized before activating DR. Routine maintenance shouldn’t create a disaster event.
apandaze@reddit
if the change is planned correctly DR will never be needed. even at a small scale, test several times, perform the change on the weekend without end users, so the following week is normal. spending a full 24 hours at work is miserable.
Endlesstrash1337@reddit
Never ever major changes on a Friday. I have shit I need to do on the weekends.
mallanson22@reddit
Definitely read only Friday in my mind.
cheapcologne@reddit
When I started a CAB for IT and project management, I put in the rules "No changes on Mondays or Fridays." I don't want me or my team to work a Friday night. And I really don't want to fuck things up on a Monday. That's bad juju for the whole week.
Fridays are read-only. And no changes during the weeks of major holidays.
pdp10@reddit
Monday, so that the maximum amount of staff are immediately available if things go wrong, and because the maximum amount of end-users are available to look at the thing and notice issues that weren't caught by automated post-change testing.
Nothing worse than Monday-morning emergencies because an end-user noticed something wrong after a Saturday change-window that passed post-change tests.
Ssakaa@reddit
Tuesday, allows cleaning up all the BS Monday churn first, last second reminder comms, etc. Noone's blindsided... but has the same benefits of the Monday deploy.
AdeptFelix@reddit
Depends on impact. If it's important, no. If no one will care, sure. I'm fine with Monday changes, just make sure to do an extra look over before committing until the coffee kicks in.
rush-2049@reddit
We do them Thursday night to get a day of real load on it so we know we don’t have a ticking time bomb on Monday.
KingSlareXIV@reddit
If you have guaranteed access to all the needed resources to test all relevant use cases and fix any issues should your change go bad, and the business is slow or closed over the weekend, Fridays are great.
If you can't guarantee that over the weekend, and lets face it unless you are a small IT shop and you do literally everything yourselves with no need for any support, weekday changes are the way to go.
Brilliant-Bat7063@reddit
Thursdays are what we do. That leaves Friday for us if shit goes wrong
Substantial_Tough289@reddit
Friday, it allows time to recover if things go bad.
If on a 24/7 operation things need to be coordinated, usually Wednesday night is good.
NoradIV@reddit
IMO, the best moment is monday night or tuesday morning before people come in. I would come at 5AM and setup everything. If I need maintenance or admin help to unfuck their shit or test, I can get everyone there on shift at their worktime.
cinta@reddit
Mondays because if you fuck something up that requires getting vendors or other departments involved, doing that over the weekend is not ideal to say the least.
ConsistentCoat5608@reddit
Infrastructure on Friday's, so you can use N+1 to keep operations alive if something goes wrong.
Monday for any major software on Monday which gives you access to higher teir support and developer access for troubleshooting.
b4k4ni@reddit
It depends. Really.
We never do changes on Monday, as the users usually have a full plate after the weekend and Mondays suck anyway. Garfield and I are on the same page in this regard.
So Tuesday or Friday and here it depends what we need to do. New User configs, management changes, GPOs, Updates, ERP or other application changes or updates and so on - workweek time, because even if it works for us while testing, there might be some users having issues, so it makes more sense this way.
If we need to make migrations or hardware exchanges and similar, we do this after work Fridays and/or over the weekend. This also goes for Updates or other changes, that can't be done on workdays, because of production or something like this. So a business critical system is needed all the time.
For ourselves, we also have a read only friday (or other days before long holidays), so we won't change firewalls or anything that could go wrong over the weekend, if the change is not needed, aside from a sysadmin with an itchy finger. This is basically to protect us from weekend work - especially the on call colleague. :)
SpicyPanda23@reddit
Almost exclusively
Key-Brilliant9376@reddit
Thursday is the day for major changes.
retrogamer-999@reddit
No changes on Friday and Saturday is a rule of thumb. Sunday only if absolutely necessary.
Monday to Thursday is free game
sole-it@reddit
Always Wed or Thursday. If anything breaks, we still have Friday to fix it and if it's been snowballing, we still have the weekend.
WorldlinessUsual4528@reddit
Heavily depends on what the changes are. For some things, I'll do it on a Friday and anticipate working through the weekend but it's only for changes where I'll be the one fixing it, or the other people involved also want to do it over the weekend. I won't make a change that requires an emergency call out to other teams.
If others have to be involved and they don't do weekends, usually a Monday or Tuesday but again, depends on the potential impact.
We have one guy who always does major changes the day before his weekday off lol. If it breaks, people wait till he's back.
equinox6k@reddit
It's a weird question, and there's no straightforward answer. It always depends on the service, your ability to fallback to previous version, how long it will take to restore a fallback scenario, the availability of vendors and partners, your willingness to work overtime and at weekends, whether colleagues are available to cover for you the next day in the worst-case scenario, and of course the impact on core business services. Never start work at 08:00 AM and work until very late; tiredness leads to a lack of patience, which leads to stupid, unthought-out decisions. Communication and transparency are key priorities. Announce incoming changes, communicate if something goes wrong, and keep posting updates for affected personnel. CEOs really hate surprises! ;)
Sensitive_Scar_1800@reddit
i think that organizations that are afraid to push changes on a specific "day of the week" need to work to improve their documentation, standardization, and automation practices.
I push production level changes as needed, including fridays, because I know that our processes are mature enough to handle an error/problem/issue.
do things break? Yes. Do we have a processes to fall back? yes. Do we take the time to assess and analyze the problem using Post-Mortems and Collaboration so its treated as a learning event? Yes.
PDQ_Brockstar@reddit
Depends on the project size and the likelyhood of major complications. I'm normally a read-only Friday guy, but if the scale of the project is too big, you take off Tues, Wed, & Thurs and start work Friday at end of business. If things go better than expected, you still get some weekend. If not, then you had half the week off to compensate.
DJDoubleDave@reddit
Typically working on weekends means it will be just you, or at best a skeleton crew. That may or may not be acceptable depending on the org and the change.
If you're running 24/7 services or otherwise do not tolerate weekend outages, then do them some weekday other than Friday.
If it's a large org, where big problems might need to include a separate team for firewall, directory, networking, etc. then probably avoid Friday. If things break, then how many people do you need to call on the weekend? Will all those teams have members available? Best to do it when you know people will be working.
Friday makes sense only when both of these things are true: A) weekend downtime doesn't matter, and B) you can reasonably expect to fix or roll back the change by yourself (or with a small team with confirmed weekend availability)
Binky390@reddit
Not helpful to your question but I work at a private school that's partial boarding so there is no "after hours." It's one of the most challenging parts of the job.
Admiral_Ackbar_1325@reddit
Tuesdays. Always Tuesdays. Mondays are busy, no one wants anything to be down on a Monday, Tuesday gives you the rest of the week to fix or rollback if something goes wrong.
TheOGTachyon@reddit
I always choose Friday and assume my weekend is shot too. Then if it isn't... pleasant surprise for me.
os2mac@reddit
Institute a “Don’t Fuck With It Friday” policy or the more politically correct “Documentation Friday”
Ideally. Always three(environments) there shall be . Dev, test ( or user acceptability ) and production . Patch dev first. Let it cook a week, debug and then do test cook a week then prod. I try to do prod patching on a Wednesday to give me a couple of days for change management and feedback and to give a couple days before the week end for fall out remediation. Then you have a nice quiet weekend.
weHaveThoughts@reddit
Fridays are cursed. “No Change Fridays” is a saying for a reason.
Thursday nights if weekends are not an option.
G8racingfool@reddit
Neither. Do it later on a Thursday. Friday's are traditionally the least productive day of the week anyway, so downtime isn't going to be a huge hit and gives you at least a day to fix/roll back and then you can work on fixing the issues that caused the deployment to fail after the weekend.
discgman@reddit
Friday before a holiday weekend. Be a man (or woman)
MochnessLonster@reddit
Big changes after hours on a Monday/Tuesday.
Read-only Fridays is law here.
BlotchyBaboon@reddit
This is very business dependent and ultimately up to you. Generally the idea with the big stuff is to obviously have the least amount of impact.
Here's something interesting from last year though that we did in an office of 75 people -
I had a firewall change out that I needed to coordinate with an ISP. Technically there was a way I could start on a Friday night, but the support I was going to get from the ISP was probably going to be awful if I needed it.
Instead, I went to the CFO and I slightly made a joke that it would be great if everyone just worked from home on Friday. He loved the idea - I was shocked how receptive he was. So yeah, problem solved.
(And as I hoped, I was done by noon that day, so I just went and had beers to pat myself on the back.)
ABlankwindow@reddit
It depends. This isn't a right or wrong. Ut really depends on the context.
Generally speaking our maintenance windows are wed overnight these days. Very few people working but also still have two thur/ fri for DOH vendor support if need be. As well as if we can roll back Friday before the weekend if need be.
That being said plenty of things get done at other days and times.
wed would be mmore normal preference but I've always worked in 24/7/365 industries and weekends don't slow down.
So tue 1030 am to 1230 pm and wed 11 p to thur 2 am tend to be my preferred maintenance windows.
The Tuesday am being cases where we haven't been able to test something in dev environment and we are deploying to live and I want the vendor to be available should bullshit happen that I cant triage inside 5 minutes.
largos7289@reddit
I mean if your paying me to work on the weekend, then i wouldn't mind it so much, along with flex time to take off during the week when it's fixed. However in IT and salaried, it's expected so that's why i don't do changes on Friday.
FearlessFloyd91@reddit
I usually do changes on Tuesday or Wednesday nights after hours. I prefer not to work on weekends if I don't have to. I plan, do dry runs and have backup plans in place in case anything goes wrong after hours during the changes. Only time we did something planned over the weekend is when we physically moved our entire datacenter from one building to another.
-lazyhustler-@reddit
Thursday, then you aren’t screwed for a week by blowing up on Monday but still have Friday to dispose of with an off ramp into the weekend if you need more time.
Arudinne@reddit
Man, I try to not even work on Fridays. /s mostly.
It really depends on what's being changed.
Competitive-Round-90@reddit
Really comes down to the business need. If you run a business that has peak usage during a weekend, may be e-commerce or online gaming infrastructure, you shouldn’t be doing updates during that time. If you are in a 9-5 m-f kind of shop like a school then you have options for patch times. If you are in a surgical department and there are no surgery’s that happen during the weekend, probably want to shoot for late Friday evening.
badaz06@reddit
I've always been a late Friday person for major upgrades and changes, fully understanding that my weekend and probably Monday and Tuesday might be shot, but expecting time off later for the extra effort.
IMHO the business relies on the tools and services we provide, and my job is to make sure those are available during normal business hours.
I know folks in the UK and Europe don't generally have the same work schedules, but I'd hate to have the CEO over my shoulder asking "When will you be done?" every 5 minutes.
Laser_Fish@reddit
If we are thinking something might go down we do it at 6pm during the week. If we are sure it will go down, we will do it Friday evening and put out a notice for downtime. We rarely need more than an evening but it happens.
greendookie69@reddit
We're a 24/6 shop. Saturday is the only day we don't work, so Friday afternoon is the only time we really get any break to make changes. Any other time is too disruptive to business for us.
So it all depends on your business I guess.
Master-IT-All@reddit
The correct answer is neither.
The correct answer is: You need to develop a change management process that correctly identifies risks, avoiding them or accepting them. And then scheduling change windows where production is not able to use a system.
So if an update to Windows Server is available, and you want to patch and restart. You can request to schedule that any evening. The expected downtime is under 2 hours, and troubleshooting is another 2 hours.
A replacement of core switching that may take eight hours, with troubleshooting and QA taking another eight? Well that's obviously needing to be scheduled to start after business on Friday.
Absolute_Bob@reddit
There's not really a correct consistent answer for anyone. It's all about your individual circumstances around production hours and maintenance windows.
hitman133295@reddit
Do biggest risk on Friday or weekend. Medium to low risk on Tuesday or Wednesday
t_whales@reddit
The day of the week doesn’t matter. The change request and communication is more vital in my opinion.
dchape93@reddit
We make our changes on Thursday. That way we have Thursday evening and Friday to fix if things go south.
BamaTony64@reddit
The problem with the Friday theory is that your users will not be there to test. Then you show up on Monday, and everyone is not just mad, but Monday mad. I am solidly on the never touch it on Friday rule. Midday Tuesday is my go-to unless it is after hours on Monday.
Bright_Arm8782@reddit
Monday. If things go wrong I'm a good team player and want the whole team around whoever was doing the work.
dard12@reddit
Anyone saying Friday is insane.
Monday is superior in every way. All of the IT staff and business users are available to help test and troubleshoot.
Always have a rollback plan.
ParkerPWNT@reddit
Monday, Vendors won't have the same level of support over the weekend.
mrrichiet@reddit
Depends on the change exactly. For small things, you'd generally have "no change Fridays" but if it's big project work (like I think you're describing) then you'd do it on Friday\at the weekend so as not to impact Prod.
Phx86@reddit
Read-only Friday.
mdervin@reddit
Implement Friday, (company pays for dinner)
Confirm & User Confirm Saturday (company pays for Breakfast & Lunch & God Forbid Dinner & two drinks)
Rollback Sunday (company pays for Breakfast & Lunch).
for the next few weeks, I get to come in whenever the hell I want to, and I get to leave whenever I want to, and then I take a comp day whenever the hell I want.
explosivelemons@reddit
We coordinate maintenance hours for after hours, pending how long it'll take it'll be Fridays or whatever works for that member environment, have the weekend to work OT if needed, and then users can let us know first thing Monday morning if they're impacted.
Jeff-J777@reddit
I would do big changes like that on a Friday over a Monday any day. I work at a place that was 24x5 so we can only do stuff on the weekends.
But other places I worked were 24x7 but every 3rd Saturday at night was blocked off for IT to do work. We had scheduled maintenance windows where we could do various IT tasks from rebooting servers to replacing firewalls.
The only exception to all of this is if a critical vulnerability is found, and a patch needs be applied. Then depending on the business impact is when we patch that hole.
cantstandmyownfeed@reddit
I always push for Sunday evening.
On a Sunday night, I'm out of weekend mode, but not tired from working all day. If something goes south and we need an all nighter, that's makes a difference.
Friday is second choice. Ruins the weekend, but no way am I getting yelled at all day on a Monday due to an outage.
ReNiC@reddit
no thx. tues-thurs.
ljr55555@reddit
Depends on the usage. If your system is heavily used on weekends for whatever reason, then yeah it makes sense to have a week to recover. As an example - if I supported a platform that did America football related stuff, I'd need the platform up Sunday no matter what. So Monday night would make sense to start a change - after the Monday night game. Got until at least Thursday.
We were pretty surprised when we profiled our online bill payment platform's usage. Peak at about 9:30 pm for each time zone (kids went to bed, time to pay the bills) and pretty much all weekend. For that system, updates at 2am on a weekday actually made sense. If we ran over and were offline until 11am, minimal impact. Noon had a small peak we would rather not disrupt. 6p had a small peak too. But we really need to recover by 9p.
Most businesses apps? High usage is 7-6 M-F, so recovering on Sat and Sun has the least impact. Saying "bummer, app should be up by Wednesday" isn't gonna go well.
FuturePath6357@reddit
if you want to be bothered on the weekend, do it friday.
illarionds@reddit
Really depends on how you view the weekend.
If you'd be thinking "great - at least I've got the weekend to fix it!" - then Friday is good.
If it's "fuck, now I have to work all weekend" - Monday is better ;)
Assuming people are willing and able to work over the weekend though, Friday seems inarguably better for the company.
ghostnodesec@reddit
Funny, depends on the amount of time needed for the change. We do Thursday nights, Friday is typically a light day, less impactful, Monday would be a disaster, as those are the busiest days. Huge change, that is more than what you can do at night. Then weekend. And is it just me, but time moves slower at night, 2 hour change 8pm, no problems usually done by 9pm. Try the same thing at 5am, wham its 10am and your name is mud.
MDParagon@reddit
It's bad jeebies to do it on a friday
jpnd123@reddit
I've only worked on 24/7 type of places and we just do all our major changes on Tuesday or Wednesday during off peak hours unless it's an emergency...I wouldn't sign up for Friday, I like weekends
the_doughboy@reddit
I know people say don’t do it on Friday. But if you do it on Friday you have 2 days to fix it.
NirvanaFan01234@reddit
I prefer some day other than Friday, usually. I'd rather give up a Tuesday night than a Friday night. I don't really want to give up my weekend if something goes wrong. But, I don't want to impact production either.
So, I usually decide based on risk. I'll do riskier things on Friday and then, if I need to work over the weekend, I'll claim comp time and get back those hours sometime over the next week or two.
jason9045@reddit
Late Mondays. You've still got enough time to fix or roll back your changes before the next work day and you're not sacrificing your weekend, and there's more of a chance vendor support will be available than at 2AM on a Sunday.