What's the rule of thumb for rebooting a production server?
Posted by Mediocre-Cobbler5016@reddit | sysadmin | View on Reddit | 274 comments
Just started at a small company and got access to our production server for the first time. Ran uptime and got back:
up 659 days, 2:02
Is that...normal? Also noticed there's an apt-get update process that's been running since January. Not sure if that's related.
What's the standard reboot cadence for prod: every 6 months? Once a year?
Thanks!
SenTedStevens@reddit
We reboot all production servers every month during patch cycles. For out of band machines, it requires an INC ticket by the system owner or approved CHG ticket depending on what server it is.
DeadbeatHoneyBadger@reddit
I typically reboot as I have kernel updates
zakabog@reddit
Reboot as necessary.
Mediocre-Cobbler5016@reddit (OP)
It's our web-facing server. Does that change the cadence?
konoo@reddit
We run quarterly patching and reboot unless and high severity vulnerability hits then we just get it done after hours.
BrentNewland@reddit
We do monthly on our Windows servers with forced reboots.
Linux servers.. no idea.
usa_reddit@reddit
You don't need to reboot Linux.
fargenable@reddit
When the kernel and glibc packages are updated it requires a reboot, for the updates to take effect. When applications are updated, like nginx, Apache httpd, MariaDB, etc, the binary needs to be restarted, that is usually accomplished by restarting the systemd service that controls the binary.
usa_reddit@reddit
Updating glibc does not require a system reboot; it only requires restarting the processes that are currently using the old version of the library in memory.
fargenable@reddit
Systemd (pid 1) is that linked to glibc. So requires a reboot.
konoo@reddit
There are absolutely times when you should reboot linux...
Calyx76@reddit
If it's not a hardware update. You shouldn't ever need to reboot Linux.
Interesting_Word99@reddit
How often you patching your kernels then?
DontGiveThemYourName@reddit
Copy fail has entered the chat
konoo@reddit
Followed closely by Dirty Frag (CVE-2026-43284)
Reading the comments from the "NEVAR REBOOT!" kids here it's no wonder that these exploits are so pervasive...
altodor@reddit
And the one from today.
https://github.com/0xdeadbeefnetwork/ssh-keysign-pwn
No_Resolution_9252@reddit
Never ObViOuSlY iTs LiNuX
No_Resolution_9252@reddit
So you aren't patching then
konoo@reddit
even with kernelcare you still should reboot every once in a while. The idea is that it allows you to patch then reboot when you can schedule a maintenance window not that you never ever reboot the computer again. That's kind of a crazy take.
usa_reddit@reddit
Spoken like a true Windows Admin, With a little bit of skill you can avoid the majority of reboots on Linux.
While rebooting a Linux server is the simplest way to ensure all kernel and library updates take effect, modern high-availability systems utilize Kernel Live Patching, targeted service restarts via tools like needrestart, and binary hot-upgrades to apply core system and application updates without ever requiring a system reboot or causing downtime.
Stonewalled9999@reddit
kernel updates even with hotpatching often need a reboot to load the new kernel
altodor@reddit
You sound like the reason why when I got to my current job everywhere I looked I found 10 to 15-year-old Linux installs where almost every package came off the install ISO.
chuckmilam@reddit
Me, the incredulous compliance auditor guy over here. How in the hell?
baasje92@reddit
Our company has a Linux server that hosts a send emails safe plugin from Outlook that reboots weekly in the weekends. The server has auto updates configured and after every update the service that runs the plugin won't start properly unless you reboot. It's a known issue for the plugin sadly which they can't figure out. So weekly reboot is the workaround.
NUTTA_BUSTAH@reddit
Doesn't kernel updates (Linux updates) still require a reboot on majority of systems?
T_Thriller_T@reddit
Sometimes, some of them. Not always.
Not entirely sure when and why. Current DirtyFrag requires a reboot always as far as I caught.
Others don't.
Both facts make effective vulnerability management for Linux quite a bit harder, which is why I know but have no further specifics.
usa_reddit@reddit
Or you could unload and blacklist models.
Flerbizky@reddit
Not necessarily. Live kernel patching exists.
deepasleep@reddit
When is the last time there hasn’t been at least one high severity CVE in a given month? I can’t remember, maybe 2016?
chandleya@reddit
Their org has really been sleeping on their opportunity to be part of a news cycle
chandleya@reddit
Which month in the past 7 years hasn’t had a high severity vulnerability?
altodor@reddit
We have weekly windows. Updates should be routine and boring, no way to guarantee that like making them frequent.
Altruistic-Ad-4090@reddit
So a web facing server with questionable patch status. Ouch.
ljr55555@reddit
Unless you were looking at a mainframe or something, the days of the uptime competition are over. Some, if not most, OS patches require a reboot. What percentage varies by OS (it's a rare windows patch that doesn't say 'reboot me!'). Over the course of two years, though, there were absolutely patches that need a reboot to really be applied.
On anything internet facing, I update weekly during a scheduled maintenance window. Now, I work for a larger company and we've got load balanced servers so there's no noticable outage to rebooting web servers. It was pretty easy to get the business to agree to a weekly risk of something going wrong with a patch and reducing capacity.
Internal stuff we do monthly - rebooting if needed.
Lucky_Foam@reddit
Is it broken? Are there patches that need a reboot?
Reboot your server when it's needed.
If it's up and running with no issues. Then leave it alone.
nullbyte420@reddit
.. You only have one?
notarealaccount223@reddit
It's also the dev server.
dadgenes@reddit
All companies have a dev and prod environment. Good companies have them separated.
notarealaccount223@reddit
I've heard it as "Everyone has dev, some people have prod as well."
Blues-Mariner@reddit
😅😅😅 “Devops”
T_Thriller_T@reddit
Thanks that got me to smile and chuckle!
intoned@reddit
Lol
midy-dk@reddit
“Oh we totally have a test-environment. We just don’t have a separate production environment” - to quote a client.
MagicBoyUK@reddit
😬 Yikes.
chefkoch_@reddit
And stores the backups.
Mediocre-Cobbler5016@reddit (OP)
yeah just the one. I'm starting to realize that's not standard?
Ssakaa@reddit
What does an outage cost per minute? When was the last time it was off and you validated the drives would spin back up? If a ups fails deadly and cooks the mobo, what do you lose? What's your rpo/rto on your backups? Are those tested? If your ISP has a hiccup, how long are you down? How many CVE 10s is that running kernel and libc pair sitting on? Have the services at least been restarted on patching? Have you at least been patching the runtime part you can live-ish? Do you know no catastrophic breaking chnges have come out in patches that'll stop services from coming back on a reboot? Can you reliably identify the change that broke your setup on your 2 year long patch cycle to revert it to limp along on operational services while you find a proper fix?
garynuman9@reddit
Wow this post is an entire education in a paragraph.
Well said.
Ssakaa@reddit
Paragraphs have structure... that has... it's about the density of a tungsten cube. That's what we get out of me deciding to address the "that's not standard?" standing in an elevator browsing reddit on my phone...
CeldonShooper@reddit
You should write an invoice for this to OP.
NotMedicine420@reddit
For a small business it can be.
Five_Guys@reddit
Depends if you like troubleshooting your single point of failure while you try to play Halo
FalconDriver85@reddit
Only if your web server runs Fedora and the sales dude has the icons on the desktop arranged as a p*nis
Dandyman1994@reddit
I hope that's a web dude Vs sales guy reference!
Purple-Path-7842@reddit
Same lmao
McGarnacIe@reddit
I'm guessing you're cloud only and all your user accounts live in a public cloud tenancy? What about company data and employee data?
Kinamya@reddit
Correct, depending on your size, 2 or more. Fail over to one, like point the webserver lb to one, then update the other, then failover and do the same. That way there is minimum downtime if any
neopod9000@reddit
Is it an important web-facing server?
Maybe run a quick non-intrusive vulnerability scan on that bad boy, so you can compare the risks of giving it a reboot yourself against the risk of someone else doing it for you.
MrExCEO@reddit
U need to backup and patch that server asap. Obviously it hasn’t been patched for years.
T_Thriller_T@reddit
Not from a sysadmin, but from a security point of view, you may want to rethink and check your system here - and maybe get yourself slightly educated.
My country's equivalent of the CISA has a good "basic cybersecurity knowledge" course and I think the CISA may have, too.
In the end, a server can go without restart for very long, but it should be a conscious decision (because it's not ideal). As it does not seem to be, you may want to get yourself a bit of education to see what your tech staff should cover.
None of this is the end of the world, but it's also not good and easier to know plan in expenses etc then fo it later on.
Concepts you want to consider (get a coarse grasp yourself, then eith your tech folks) from my POV, which may be incomplete, wrongly ordered or stress the wrong things - again I recommend checking CISA or similar
Bigger concepts, but very important, too:
Mephistopplz@reddit
I assume it’s in a DMZ?
redoxburner@reddit
One of your multiple, load-balanced, web facing servers?
If there is only one web facing server then 24/7 availability clearly isn't a consideration, so reboot at will at a low traffic time.
If it's one of multiple load-balanced servers then you have an HA environment where each server can handle the load for a while, so reboot at will at a low traffic time.
BlackV@reddit
Yes it becomes weekly instead
jibbits61@reddit
Company wide we do monthly patches, windows and Linux. Finance systems that ‘can’t be brought down’ are done quarterly.
I’d say if it’s web facing as well, patch monthly and get it and the other servers scanned for vulnerabilities. You’ll be busy for a while. Make sure you have good backups!
mike9874@reddit
If you're talking cloud you should be thinking of something like an azure app service plan - people need to stop putting everything on an IaaS VM in the cloud.
The benefits of the cloud are that they'll manage the underlying server bits, you just tell it you want a web app and they make it happen - simplified management & reduced attack surface
Opposite_Bag_7434@reddit
Seems about right. Especially if the business depends on the web server being up.
stephenph@reddit
And if your biz depends on the web server being up, it should have a redundant server of some sort. In which case a reboot is not going to cause an issue.
rose_gold_glitter@reddit
Web facing server that has not been updated in nearly 2 years. That is .... a choice. From a security perspective, this is downright suicidal.
EmperorGeek@reddit
Sounds like you need a load balancer in front of it so you can reboot the back end servers as needed after patching.
I would be terrified to have an unpatched server facing the Internet!
snarlywino@reddit
On a web facing server, with today’s Windows vulnerabilities? Yesterday!
stupv@reddit
It means it should be getting patched regularly, which should include regular reboots
zakabog@reddit
Is it bare metal or a VM? Is it a scalable service it hosts or just one behemoth application?
GullibleDetective@reddit
Setup a WAF now!
kaiserh808@reddit
Wait, what? You’ve got an unpatched kernel exposed to the internet for two years? do this now: sudo apt update && sudo apt upgrade -y sudo reboot
Pls_submit_a_ticket@reddit
Might as well take a snapshot and run sudo apt dist-upgrade with it being unpatched/rebooted in that long. Bold to assume there are snapshots or checkpoints though i guess
whatdoido8383@reddit
Based on their responses, they're more than likely already compromised and they don't know it.
Lucky_Foam@reddit
This is the answer. Only reboot as needed.
owzleee@reddit
Alternate weeks live and DR so you also make sure failover works. Test your failover regularly!
Mediocre-Cobbler5016@reddit (OP)
Just woke up. Did not expect this. Read through everything. Main things I'm taking:
- check backups first (multiple times)
- find a maintenance window
- ask about redundancy
- monthly cadence going forward
Thank you all! This was way more than I expected.
ContributionFit1243@reddit
Webapps should have clusters w/ failover, and reboot regularly.
If you don’t trust your system to come back up after a reboot, you should fix that.
Altruistic-Ad-4090@reddit
Schedule a maintenance weekend once am a month. If there is stuff that can't wait due to security reasons, change control and notification, but again, off business hours if possible.
Endlesstrash1337@reddit
Before you even think about rebooting that thing make sure your backups are working. Not just backing up but that you can restore stuff too.
pangapingus@reddit
And if a VM snapshot for good measure (or if you have a BDRaaS appliance make sure it boots)
ludlology@reddit
but dont forget to remove the snapshots, or you will be learning some very arcane tricks at midnight in three years after you realize an ancient snapshot chain is corrupted
BeasleyMusic@reddit
You just brought back nightmares of discovering month old orphaned Veeam snapshots 😭😭😭
shadhzaman@reddit
Linux or Windows?
Windows will get rebooted once a week post updates, in a set window that is announced - that's the golden rule, set a window least likely to disrupt something, announce it and make the people aware. Doesn't matter if you are on a federal site or bank, you've seen almost all of them go down for maintenance on odd hours of a saturday (like 11PM).
On the other side of this, if you are not doing post update reboots or updates (lets say its a linux) then at least one to two months depending on the usage and type of the server, but announcing it to the people who use it first.
savageXent-Tr00blxx7@reddit
asking if its OK to do now or ask for a servicewindow.
alfred81596@reddit
We reboot once a week at the same time we do security updates (Windows and Linux both). Servers are divided into 3 update rings across 3 different days; anything that requires high availability is divided.
IMO, I'd rather reboot more than technically necessary and not be afraid to kick a server if need be. I always know it's gonna come back :)
Calyx76@reddit
You said apt-get. So it's a Linux server. Rebooting is for hardware changes if that's the case. Linux doesn't need to be rebooted normally. If you need to do anything it's restart the services. And that is a huge IF. Most of the time you don't need to do anything just leave it be. Run updates regularly and call it a day.
dmoisan@reddit
Or if apt-get pulls down a kernel update.
Most likely, someone was running apt-get and forgot about it and closed the shell or ssh connection. Many times apt-get will install an update and run into a config file that it prompts you to update. This will wait forever if the session got closed.
Remedy is to reboot, perform dpkg-reconfigure and apt-get --fix-missing
Calyx76@reddit
Okay yes you are right. I forgot about kernel updates.
RavenousTitan818@reddit
Depends a lot on the server, but we run updates monthly and reboot them during maintenance windows.
Affectionate-Cat-975@reddit
During maintenance windows
Kahless_2K@reddit
Standard reboot cadance is at minimum, when there is a critical vulnerability that needs patched.
Hopefully you will also patching for less important ones at least once per month.
Years of uptime is a really bad sign.
rocuspeter@reddit
We do monthly on windows servers.
socksonachicken@reddit
That's highly dependant on snapshots, backups, load balancing, what the server is for...
Fuck it, send it.
symtexxd@reddit
Backup restore and a hot standby server
Suitable-Hand-1059@reddit
If it’s a Windows server, rebooting is a necessary step in updating in many cases. I have ours scheduled to reboot in a staggered sequence once a week so that they can complete updates and to avoid causing power surges, etc.
andrew_joy@reddit
If you shout " oh no i tripped over the power cable" you can then reboot anything.
Top_Boysenberry_7784@reddit
I'm curious if OPs environment is a complete shit show or it's just the Linux servers that are the issue.
I have walked into many companies where the windows side isn't perfect but they are doing ok but no one knows shit about Linux so the admins don't even log on to those. They just have Linux servers a previous employee setup or was something a vendor/contractor built for them at one time.
regszurob@reddit
If high availability is required, you move the services over to the peer node and then perform the update and restart. If high availability is not required, you request a sufficiently large maintenance window so there is enough time to recover in case something goes wrong during startup. The updates themselves determine how often a reboot is required. On Debian-based systems, you can monitor /var/run/reboot-required to see when a reboot is needed. In such cases, it naturally makes sense to perform this during the next suitable maintenance window (for example, the following weekend). If you update both the OS and the firmware frequently enough, reboots will end up being required more often than you actually have the opportunity to perform them.
Inn0centSinner@reddit
Ask your boss why they haven't been keeping up with patching.
TheFluffiestRedditor@reddit
If apt hung during a cron run, that will have blocked all future apt runs, existing process and all.
That server is going to need to be inspected.
T_Thriller_T@reddit
Wait, what?
I'm not sure if this is logical and my brain knotted itself while processing or if I did not know yet.
Could you explain with a few more words?
frymaster@reddit
apt is run periodically by cron, and sets a lock file so ensure only one apt process is running at a time. If it's hung, then any subsequent runs of apt will notice the lock and immediately exit. So that hanging process needs to be killed, and also it sounds like the server needs some TLC
T_Thriller_T@reddit
Very good to know, thanks.
Never heard about the problem so far, but I will remember!
Trust_8067@reddit
Most patching doesn't require a reboot for Linux.
xargling_breau@reddit
All Linux kernel changes require a reboot unless you use software like KernelCare for live patching. Even then, some patches may still require a reboot.
T_Thriller_T@reddit
Do you happen to now what makes a reboot required with KernelCare or potentially other live patching?
I lately had the discussion, was asked just that and then could not say anything but "i know some do, I know some don't"
JoePatowski@reddit
we’ve been using kernelcare for two years and we haven’t had to reboot because of a patch
Adorable_Wolf_8387@reddit
it's possible to do an in-place kernel upgrade without resetting uptime. Of course I am incredibly skeptical they are doing this, since there's not really a good reason to at a small business.
Inn0centSinner@reddit
Thanks. I've haven't touched Linux since CentOS when I had to configure OpenVPN. This was more than a decade ago.
T_Thriller_T@reddit
Absolutely possible.
Absolutely possible to ne the opposite:
Update installed, but kernel running the old version until reboot.
Makes finding out what kernel is operative really tedious.
T_Thriller_T@reddit
Depends who build the server and who supplies.
I think some Linux business "suppliers" offer it as a standard / at least a standard config. (Especially with LTS as far as I'm aware - which is limited, haven't done market research specifically).
So doubtful, but possible!
automounter@reddit
Maybe they're live patching.
itishowitisanditbad@reddit
Thats the sort of thing you say optimistically and then you and your coworker side eye each other, both knowing that'd be a miracle.
dinominant@reddit
If it's a Windows server, the update might cause more harm than running it in a secure and monitored environment.
UserProv_Minotaur@reddit
Are you not patching the server?
Substantial_Tough289@reddit
Windows, every 30 to 40 days. The exception being when applying updates which we try to schedule for the same day as the reboot.
Linux, only when patched/updated.
The question is, if the machine is a Hyper-V host, do you automate the restart or do it manually stopping all guests before restarting and the starting the guests?
frymaster@reddit
it gets wedged every now and again and, annoyingly, that will stop any newer update checks from running as it'll be holding the lock. Just kill that process
guydogg@reddit
Friday only, no heads up, 15 mins before you go offline.
Our non-windows servers are rebooted bi-monthly. Everything else follows a monthly schedule. About 2500 servers.
qkdsm7@reddit
Ubuntu lts prod web uptime was ~250 days--- only recently had relevant, discovered (hah) kernel patches that required reboot, and could pay the subscription for the option of live kernel update without reboot. Yes there were documented mitigations that could have held it off but easier to just update and reboot, than have to discuss it.
So... not necessarily crazy with that uptime but suspect in your instance, it probably isn't great. The apt update issue is certainly a red flag to me, our policy is 2x/month, in practice productiin can get through apt/upgrade more like 3-5 times a month.
xXxLinuxUserxXx@reddit
We have apticron installed which sends us mail regarding open security updates. If critical we patch according to urgency (rce -> as fast as we can e.g. right away or during the night).
Else we try to only have 10-20 open updates (quiet common with all the php extensions being own packages). I would say we at least patch once a month sometimes biweekly.
And yes we have more than one server so we just update them one by one (during the same timeslot) and verify that the application still works.
Dev systems install updates with unattenden_upgrades everyday at around 6 o`clock.
TheBigBeardedGeek@reddit
We do everything monthly as part of our patching cycle. Even with a *Nix box it's good to reboot post updates for services, kernel, etc.
johnyb6633@reddit
Just do it! And if anyone complains, get pissed also. Like “yeah damn windows update just rebooting shit whenever it wants”.
automounter@reddit
Microsoft boys be rebooting once a month. Linux be rebooting once a year.
To be honest it's more likely we rebuild a computer than reboot it.
Hoggs@reddit
So you leave your kernel running unpatched for a whole year? Yikes.
Ssakaa@reddit
Given the past few weeks of cves.... wheeeeee.
automounter@reddit
The past few weeks of CVEs are an exception. And I will say MOST of our fleet is rebuilt 3x a year. No patching just rebuild it and blue/green.
Ssakaa@reddit
So the kernel project changed how they record/report things a few years ago, massively increasing the numbers, but still, letting Google do the summary for me here...
Now, even if 90% of those are in driver code for modules you don't load, that's still a fairly big chunk of things to seriously consider patching for at least on the same monthly schedule you see for Windows.
R4PT0RGaming@reddit
That’s wild a year ago…
BisonThunderclap@reddit
Hell, as a Microsoft shop we're patching once a week on those servers and they can reboot if they need.
My philosophy is that I'd rather know all the servers can reboot without issue for when something does go drastically south.
stephenph@reddit
Yep, my goal is at least monthly patching /reboot, with an optional two week to catch something like dirty frag.
There are arguments for frequent vs rare reboots, but with todays speed and reliability I would rather take a four hour window monthly then risk a multi day outage.
notickeynoworky@reddit
Why are you patching once a week when MS rolls out patches monthly? Well, not counting the out of bands to fix what the patch broke and the ones to fix that.
No wait. Weekly adds up.
Loudergood@reddit
Yeah, if you're afraid of rebooting any of your infrastructure at an scheduled time, you've got a problem.
DegaussedMixtape@reddit
Primarily Microsoft boy checking in. 1 week or 2 weeks max, gotta keep those patches rolling. Linux, as needed, potentially never.
StrangeInspector7387@reddit
Man, you gotta patch your Linux servers.
Opposite_Bag_7434@reddit
Yea, “never” is a famous last word when it comes to Linux patches.
LazyInLA@reddit
There are cases where that is normalized, sure, but just because it can run forever doesn't mean it's a good idea. I don't think a good sysadmin should ignore such a significant aspect of their practice just because the OS doesn't sometimes get schizophrenic after seeing the moon too many times.
Gummyrabbit@reddit
Reboot it remotely one hour before you go on vacation.
theoriginalzads@reddit
Do it in production… oh. Wait.
Nikumba@reddit
All our servers reboot monthly with patching, our RDS servers reboot weekly but that more a performance clean up than anything.
xzl830@reddit
When I worked at KableTown our PPV servers were up for 5 years.
ArcaneTraceRoute@reddit
Once a month during patching, typically 3rd Friday Night/Saturday morning for monthly patching.
Schizoidman007@reddit
I'd ask why it hasn't been updated or reboot in all that time, does it not come back up correctly?, is the OS no longer supported?
cwm13@reddit
Unless there are very special circumstances, all 2,000+ of ours get restarted monthly for updates. They have a varied schedule they can be assigned to, test group earliest, then some a week later, some 2 weeks + 2 days later, etc. Some very specific ones are manual restart after automated install, but those also are generally restarted every month.
T_Thriller_T@reddit
I think most of where I work the rule was somewhere between 2 weeks and monthly - and since I went into security we were often not happy with the monthly ones.
(Not because it's not okay, we were totally fine when it was for rolling test groups or servers with extremely high uptime requirements, but because those had no real reason to be monthly in comparison to others).
Nice to know this actually seems to be a standard!
Medium_Banana4074@reddit
Before rebooting such a neglected server:
BrainScaping@reddit
Do it early, do it often.
CharlieTecho@reddit
Don't lol
anikansk@reddit
I schedule reboot every server once a month. There is nothing worse, especially with windows, to be facing "Applying Updates. Please do not turn off your computer" when its urgent and you dont know what update or reg hack from 236 days ago is being applied.
Grant I was small, only 180 servers, but every windows server was rebooted every 30 days and linux server every 90 days either via schedule or patch window. We had an agreed window within the business, a four hour slot UTC time where no-one was working globally on Saturday evening.
3rd_CultureKid@reddit
Don’t do it in the day 🤣
ThimMerrilyn@reddit
Tbh I never reboot anything manually unless I have to but monthly patching sometimes causes reboots.
LuckyWriter1292@reddit
Reboot outside core hours as needed
dk_DB@reddit
Windows - monthly a few days after patchday + oob updates when needed.
Linux - depends on the type. Many are appliances for us, so depending on the update/maintenance cycle of the manufacturer Internal machines - along with ms updates: monthly as i don't need to make it harder for everyone to plan. 99% are clean enough to just roll the updates daily + reboot along the windows servers.
For your web facing machine. Ideally you want two of the same - assuming you're relying on those being on 100% of the timw - and that thing should not live in the internet... It's place is behind an WAF
Ohmystory@reddit
Backup backup backup … verify that you have all data ( if database, database needed to be
Quietest-ed or use a database backup facilities )
BlackV@reddit
323onp@reddit
You forgot to delete your snap and your vmfs store got full and fell over ....
BlackV@reddit
Ha valid
xargling_breau@reddit
What kind of server is it? is it Linux or Windows? As someone who worked at a very large webhosting company that specialized in shared hosting on linux, reboots only happened when necessary. All kernel patching happened via kernelcare and the only time a reboot happened was if it was a patch that could not be applied via kcare.
frissonaut@reddit
You should read what OP said. Wouldn't need to ask if it is linux or windows.
xargling_breau@reddit
You mean the one part that I skimmed where there is one mention of apt-get, not anything else.
satsuke@reddit
I do every other week or so, but I built enough redundancy and failover in the it doesnt impact my users.
gumbrilla@reddit
OK, so Linux. Good, but unless you are running something that does in place upgrades your machine will be badly in need of patching.
Depending on your distribution, but check if the following file exists.. /var/run/reboot-required... So something like 'cat /var/run/reboot-required'
If that is there it's telling you that you should reboot for updates to take effect. Reboot generally only required when that is set. For a singleton Web server I'd say once a month, assuming it's got other layers of protection, and the Web serving software is also updated, best configure unattended upgrade
You're going to have to fix the apt-get process, I'd kill it, and then run apt update and then apt upgrade manually, see what errors you get, probably some keys are no longer recognised.
SeniorSwordfish636@reddit
Post the url of the website here and all your questions will be dutifully answered.
/sarcasm- don’t do this!
MormonDew@reddit
It better be every few months for updates at least.
davy_crockett_slayer@reddit
Verify data integrity, and all essential services come back on.
DheeradjS@reddit
Our VMHosts are quarterly. These have no internet connectivity otherwise.
VMs go monthly at the most.
helphunting@reddit
When rebooting blade servers make sure you press the button on the correct one.
grepaly@reddit
It is not about the uptime. New kernel installed, reboot. Unless you have live-patching.
TimTimmaeh@reddit
Monthly, after patching.
doctorjbeam@reddit
Prepare three envelopes and full send it
nrm94@reddit
Always on a Friday afternoon, and make sure you are remote when doing it.
thatgeekfromthere@reddit
In the old days, use to run servers like uptime as high score, I even got to 1000+ days at one point. These days we reboot monthly when doing updates to servers.
SAugsburger@reddit
1000 days? In some past jobs I found switches with uptime into 7+ years. Kinda crazy.
webnestify@reddit
I have cron to reboot every Sunday at 4 AM (local server time). I know it's an overkill, but I really like to do it on all of my devices and managed servers 🙂.
chesser45@reddit
That much uptime? If it’s a windows server… plan an outage window and pray? /s kinda
fdeyso@reddit
Judging by the apt package manager it is not.
chesser45@reddit
My excuse you say? I don’t know how to read!
fdeyso@reddit
It’s some sort of linux.
Parity99@reddit
Get a solid backup on that mofo before you do anything. And test restore it.
EmperorGeek@reddit
At some point in that period there was AT LEAST ONE OS patch that required a Reboot. Even Linux has to reboot to patch its kernel. Normal reboot cadence where I work is monthly reboots, either automatically after patching or manually after patching (some environments have to be rebooted in a specific sequence and functionality needs to be confirmed at each step.
ptinsley@reddit
On a Friday right before you leave on vacation
Pure_Fox9415@reddit
It's bad, BUT DO NOT REBOOT ANYTHING JUST FOR REBOOT ITSELF. Ensure you know everything running on this system, and how to get it back if it wouldn't return on reboot. Make a full and per-component backup of everything.
ZestycloseStorage4@reddit
Just Send It, and if anyone complains just blame the ISP
habratto@reddit
I have one server with bugged software made by a big company who is too proud to admit their software is bugged. I'm rebooting this weekly. Other works endlessly.
suburbanplankton@reddit
Every server (we have a mixture of Windows and RHEL) gets patched and rebooted once a month.
Stinky--Whizzleteats@reddit
Don't look at it, but don't let it know you're not looking at it. When there's a progress bar don't touch it, don't breathe on it, but don't act nervous either because they can sense your fear.
Prepare the holy oils and say a prayer to the machine spirit.
But seriously though, an apt-get being stuck like that needs to be resolved with a reboot.
deafphate@reddit
How old is the hardware? Had a server up for well over five years because the hardware was so old, we didn't think it'd come back up. Ran an old version of HPUX on ancient hardware, yet it ran a critical clinical application used by ONE doctor.
No_Resolution_9252@reddit
Longest uptime I ever saw was an NT4 box that ran a bowling alley that had been up 17 years. Probably the most impressive was a Windows 2000 server running SQL 7, legacy C++ apps and a web server that was up almost 11 years before we shut it down to move it to hyper-v.
If it doesn't have any security exposure and your apps aren't garbage, you may never need to update it. But security exposure can pop in things like cnc machines, hvac machines, MRI machines, etc and should probably have their software, drivers and firmware updated periodically
tomthecomputerguy@reddit
Just do a scream test. At 3 pm. On a Friday.
What's the worst that could happen.
Annh1234@reddit
Depends on what's on it. Most stuff can be updated without restarts. But sometimes you don't know if everything will come back back if you restart it...
Fatel28@reddit
If you have a machine that cannot survive a reboot that's a totally different problem. Servers should be able to reboot when needed and come back up without intervention
Annh1234@reddit
Agreed, but also had machines with 3k days uptime lol
Ummgh23@reddit
Just send it
phoenix823@reddit
If you are not patching and rebooting monthly you're stacking up risk when you shouldn't be.
Sagail@reddit
Yes agree except when you have enclaves with a diff patch cycle
arbedub@reddit
Only on a Friday afternoon.
stephenph@reddit
On a three day weekend. Even better if the entire Linux IT admin department is meeting at Joe's for a bbq/beer bash.
semi-@reddit
frequently enough that nobody depends on or is impacted by it being rebooted. Monthly is a good target. Bonus points if you are fully reprovisioning it automatically and not relying on it carrying state over so you know that nothing is depending on anything not defined in whatever IaC your environment uses
degoba@reddit
We patch a d reboot nightly or in the case of our docker clusters we tear down and rebuild the servers nightly.
LazyInLA@reddit
IMO normal is different for every environment but I need to know my critical machines are going to come back up so I have a window scheduled for once a month if updates haven't forced it sooner. I'd be nervous as hell rebooting a critical box that I've become responsible for, not having touched it before, with that kind of uptime.
Mediocre-Cobbler5016@reddit (OP)
Should I just leave it and not reboot? Don't fix what's not broken type of thing?
konoo@reddit
You should make damn sure you have good backups then patch and reboot, probably several times.
stephenph@reddit
I would strongly consider a full rebuild, or at least have that discussion.
LazyInLA@reddit
Yep, this. Although with oddities like a hung apt-update query, I would want to see a clean reboot before attempting updates again.
Trust_8067@reddit
Correct, you literally have no reason to reboot it other than you're new to IT so it's something different you haven't seen before. For a Linux server, that's not even a long uptime.
stupv@reddit
Be aware of the tremendous risk of rebooting (and patching) when it hasn't happened in a long time. There can be a million skeletons in that closet just waiting to fall out
TooOldForThis81@reddit
You need to speak to the other admins first (if any), or your direct supervisor. Going and do your own thing at this point can be dangerous. Also, document the results of that discussion. CYA, always!
MentalCaramel7640@reddit
What is the impact if it doesn't restart? How easily (if it goes horribly wrong) could you rebuild it from scratch and reinstall it's app/recover the app from backup? That should tell you if you should be putting a failover in place (and testing it and cutting over to it) before you reboot it or not.
Whatever happens you have a ticking time bomb in an unknown state somewhere before phew, it was perfectly fine to well, that's a mess.
randomugh1@reddit
Bootup your Backup in a lab and see if it works
stephenph@reddit
For a few months MCI was running one of its secondary web servers on a Solaris 5 box under the devs desk. Every so often he would just reboot it and cause an outage
jameseatsworld@reddit
I moved our servers to the cloud so I can just blame Azure when we restart middle of the day /s
lacymooretx@reddit
Should be rebooted daily during mid afternoon
sleepeezz@reddit
It depends on the SLA, business requirements, and available resources. We operate in a 24/7 environment where we committed to zero downtime. Because of that, we could not reboot servers freely, as even a restart would cause service interruption.
Over time, we encountered issues with outdated OS patches, especially when deploying endpoint security solutions that required newer updates or system restarts. Some endpoint security installations also required mandatory reboots. From these experiences, we learned the importance of planning maintenance windows and scheduled downtime, such as once every six months or annually, to keep operating systems and security components up to date.
As mentioned earlier, it ultimately depends on the business requirements. For internal servers, we have more flexibility and can manage maintenance activities more freely.
Opposite_Bag_7434@reddit
Depends on the server, what it is used for and the company.
I’ve had a couple of friends that worked for hospitals. The blood banks (this was a while back so high availability was a bit different than today) could be down like a couple of times a year for a crazy short maintenance window.
I worked for a major pharmaceutical and we had tons of servers that had very short maintenance windows, but fortunately most were virtual running on top of pretty serious clusters. So individual servers could be brought down or even replaced.
OP I would schedule time outside of when the server will be in use to perform maintenance. Cleaning filters, inspections on the physical hardware, patching and restarts. It’s cool to see huge uptimes but they should have occasional attention. I like quarterly myself with at least monthly for more critical patching. This way it is never left for a long time.
lelio98@reddit
Rebooting should not be a concerning event. If it is, you lack the adequate infrastructure and management for the impact that rebooting would have.
If you are running a single, on premise web server and rebooting would cause problems, then you need rework how you manage your webserver(s).
The actual answer is, reboot as necessary (except Fridays). You shouldn’t be worried about a reboot.
A7XfoREVer15@reddit
Given the uptime and the size of the org you described, I would verify backups before doing any reboots.
uptimefordays@reddit
Reboot with patches as required.
HurdyWordyBurdy@reddit
Preferred? No warning at 8pm Friday.
OptimalCynic@reddit
Depends how stable your power is
mollythepug@reddit
Only on Fridays before the long weekend, when you’re off on vacation the next week. 😈
aringa@reddit
At least once per month for patching.
weHaveThoughts@reddit
“Just started”, I’d wait a bit before restarting a Linux box, probably another couple of months. But that’s me.
omn1p073n7@reddit
3 times, always reboot it 3 times.
Medical-Ask7149@reddit
Keep automatic updates on and allow it to restart after updates. Always a good idea…
PDQ_Brockstar@reddit
Typically monthly, but with Windows new hotpatch feature, that might change.
theoverseerer@reddit
Not generally, as many patches require a reboot, but hard to know, without really speaking about OS, type of server etc. I think most folks try to follow a monthly/quarterly type system, if the system is mission critical, run clusters and do rolling restarts, some systems of course support live patching, so it could be patched, but guessing not.
hamburgler26@reddit
By rule I like to use my index finger to initiate a reboot. I find it is more accurate than, by rule, using my thumb.
Flashy_Try4769@reddit
Windows sever every month after MS security patches are installed. Non-Window, never.
valar12@reddit
Remove disk snapshots. Ensure a clean backup is performed. Rebootio!
BuffaloRedshark@reddit
At least monthly when they get patched
Nearby-Pattern8644@reddit
I would back it up then mirror it .
Before any reboots
gac64k56@reddit
If you have a change control process, put in a change ticket
Make sure backups are current
We do weekly reboots and patching, all automated, testing in dev / UAT first during the weekdays (testing for bad patches and / or bad code) and on the weekends for PROD.
Trust_8067@reddit
As long as it's getting security patches it's fine. 659 days of uptime is recently rebooted for a Linux server. My last job at a F500, we had several with over 6 years uptime.
Steve_at_Werk@reddit
Reboot after patching; so, monthly for most stuff
brownacid@reddit
So no patches that have required a reboot in almost two years? What’s this thing running?
ben_zachary@reddit
Reboot it after hours once you know your safety net. If anyone asks about it , just plead innocent heh
Excellent-Program333@reddit
Reboot Only Friday?
QPC414@reddit
Yikes that is some serious uptime, regardless of the OS.
Take a snapshot of the VM. And as many others have said. BACKUP and test the restore.
snappywombatt@reddit
IllIntroduction8499@reddit
Tell everyone, then yolo.
MyPhotographyReddit@reddit
I'm tired boss.
KareemPie81@reddit
Bend over, grab your ankles and just wait for what’s about to happen
Liquidennis@reddit
With an updated resume in hand.
KareemPie81@reddit
Oh, you know this dance I assume
StrangeInspector7387@reddit
At least monthly during a defined maintenance window, usually after business hours. Use that time to apply OS and app patches along with BIOS, firmware and driver updates.
d00ber@reddit
If you're just updating security patches, it might not require reboot.
Check cat /var/run/reboot-required to see if one is necessary, but it really depends on release..etc
Single-Virus4935@reddit
Restart when needed for security or kernel updates. W/o security updates at least every 30/day because a server with way over 30+ days may have accumulated possible local changes and the upgrade paths over multiple versions isnt guarantueed to be tested by the maintainer. 700d is almost a guaranteed unplanned major outage when the server is rebooted
BisonThunderclap@reddit
Take a backup. Test restore the backup.
Reboot the machine.
After everything is figured out, put in a monthly reboot.
MoshizZ@reddit
Yes!! This!! Test the backup before you reboot.
I’ve been in a terrible situation where I had an issue with a server not booting and all the backups that I had also had the blue screen upon boot.
That was an absolute shit few days.
yamsyamsya@reddit
At least once a quarter. I'm not rebooting to fix any random issues, and at least with Linux, I'm not rebooting because the server needs it for some reason. I reboot to make sure it comes back up properly.
whiteycnbr@reddit
Reboot when required by patching, if it's hot patching then no reboot. Reboot if issue with app running, otherwise don't reboot.
Coldsmoke888@reddit
Monthly restarts and updates are mandates in my environment.
I have a couple sites upset by it because they have annoyingly complicated automation systems with SCADA and hundreds of PLCs and all that terrible shit. We just schedule same day, same time, 2-3 hours downtime to allow for troubleshooting. It’s fine.
Always back up before you do this regardless of the time since last.
Asleep_Spray274@reddit
As long as it's not running a life support machine, just bounce it
Meadbreath@reddit
Yank and pray
Ssakaa@reddit
Don't threaten me with a good time.
hkusp45css@reddit
I'll bounce a server on a whim. We do it all the time, here. We have plenty of redundancy, so it's not it costs us productivity or anything.
Other than that, we reboot them when needed for updates and such. I have some nix servers with over 600 days of uptime on them, but that's just because you never really have to reboot nix.
Attack-Chihuahua-85@reddit
My personal web server: weekly. Company prod boxes: quarterly, unless there’s a major cve then immediately.
Also is this a physical box, or vm? Do you have a load balancer or proxy in front of it?
Master-IT-All@reddit
Monthly for updates.
Altruistic-Map5605@reddit
Just do it. Preferably after hours and with a memo.
Soggy-Attempt@reddit
There are two trains of thought. 1) old school, only reboot when absolutely necessary. Ive seen Unix systems have an uptime over 1,300 days. Of the 1,000+ the company had probably 50% were greater than 500 2) patch/reboot on a schedule. Usually monthly/quarterly/yearly. More secure but more downtime.
discgman@reddit
When necessary.
RoxoRoxo@reddit
my servers running very um fragile software just 10 ply software we reboot at the max every 30 days mostly out of safety. no idea what would happen if we didnt ive been here for 3 years and its just been the process but ive heard stories of it freaking out in the past so we just do this
Acrobatic_Cycle_6631@reddit
That’s going to be the least of your problems. Allow a lot of time for the reboot to take place, ensure you have a contingency plan if anything goes wrong.
I’ve been through this, the server took a few hours to reboot as it was applying windows updates etc.
the_doughboy@reddit
Active/passive (at least) so you can do it whenever you want.
whetu@reddit
It depends on your patching policy. Where I work, we patch monthly and reboot if the system indicates it needs a reboot, which is usually the case. This is all automated because we have the redundancy and architecture to support that - and we're a small company too.
In our context, a host being up for more than 60ish days (i.e. two patch cycles) would be worthy of an investigation.
coolbeaNs92@reddit
Some of the comments here are pretty shocking.
theEvilQuesadilla@reddit
Do you have redundancy? If so, send a warning if necessary then do it after X hours.
If no redundancy, I hope you know of any and all dependency servers or services.
dotnofoolin@reddit
If possible, I usually spin up a new server, patch/update it, get the app running on it in parallel, and nuke the old server. Minimal downtime and a guarantee you can get the app running first, during normal business hours (not in a middle of the night panic).
This assumes a virtual/cloud situation. Different story for an on-prem physical server.
Mehere_64@reddit
In the case of this server, I would highly recommend you testing your backups prior to any reboot or patching. You do have backups right?
ez151@reddit
After hours and at least a day away from being put back in production just in case tshtf
Clean_Picture2756@reddit
Guessing not a fully patched windows server then, as they update and restart every month on our sites .
zakabog@reddit
From running
uptimeandapt-get updateI would assume it's a Linux server running a Debian based distro (likely Ubuntu if it's not Debian directly.)