Release upgrade, or start fresh?
Posted by N5tp4nts@reddit | linuxadmin | View on Reddit | 19 comments
Every couple of years, one of my systems reaches end of life. For example; my system that runs Nextcloud and a webserver is on ubuntu 20.x LTS and needs to be upgraded. If I do the release upgrade, a bunch of things will break and need sorting out. In 20 years I've probably never had a painless release upgrade, regardless of the distro.
What's the general consensus? Start fresh, or do a release upgrade and spend a bunch of time chasing demons?
I should probably be containerizing the things I need so the host can be ephemeral, I know, I know.
badtux99@reddit
Servers should only do one thing, and one thing only. That way if a release upgrade breaks something, it will break the one and only thing that the server does. It is when you have lots of stuff on a server that it becomes a huge problem.
I have basically two kinds of physical servers in my infrastructure: Compute nodes, and storage nodes. Everything else is a virtual machine or container running on those servers. If it is a virtual machine it does one and only one thing to make it easy to upgrade. I can do a release upgrade without much fear of disrupting things. I migrate a compute node's workload to another compute node before upgrading it, if the upgrade fails people aren't disrupted. Storage nodes are harder to transparently upgrade, but because they are just serving vanilla NFS to the compute nodes and NFS never seems to get broken by upgrades, I've never had an upgrade fail there.
Herds, not pets.
insertwittyhndle@reddit
Are you running a hypervisor of some sort for these machines?
The easiest solution is to take a snapshot, or clone the VM outright, and upgrade it. If you use a snapshot you can roll back. If you use a clone, no big deal if you mess that up either.
Ideally you’ll also have backups of course.
MikeZ-FSU@reddit
I've done tons of "do-release-upgrade"s on ubuntu LTS over the years. The only troubles I've had were when I failed to look at the version numbers of important packages. To me, that's a totally self-inflicted wound, and not ubuntu's fault.
Virtual_BlackBelt@reddit
I always recommend clean installs for servers. Labs (home or other), sandbox, desktop... go ahead and upgrade. Just remember to keep your data separate and to try using as much IaC and automation as is reasonably possible to reduce those conflicts that you're talking about.
shaolinmaru@reddit
And you still insists?
N5tp4nts@reddit (OP)
I had time. Took a snapshot and tried the upgrade.
vogelke@reddit
Every time I've tried to do an in-place upgrade, I got bitten in the badoingas. The only "upgrades" I do might be a kernel patch for a security problem; otherwise, get a new box with a fresh install and start moving things over.
michaelpaoli@reddit
"It depends". There are pros and cons either way. And some distros/versions upgrade much more easily than others. E.g. I've been doing Debian upgrades for more than a quarter century - never hit any major issues with it. Doesn't mean it's trivial, and there won't be the occasional glitch, but it's often (typically) much easier than reinstalling - particularly when one needs, with a new install, to do all the installation of specific software, configuration, reloading of data, etc. But some distros/versions suck at upgrading. E.g., used to be the case that Red Hat's upgrade process was so horrid that all their official documentation highly advised against it, and instead highly recommended doing a fresh install - that was some fair while back ... but still. And yes, some distros/version still quite suck at upgrades.
As far as the *buntus go, going from one LTS to next, should work pretty well - upgrade to latest point release on the old first, then do the LTS version upgrade ... but I don't have nearly as much experience with that, so I'll let others comment on how (not) well that works with the *buntus.
Also, even if one generally goes the upgrade route, one still needs backups, and in general the means to reinstall/redeploy to get things back to the state they were, or to more-or-less replicate to make quite similar system(s) - so going the upgrade route isn't an excuse for lacking the proper infrastructure to do those other things too. So, yeah, if the system goes kablooey - from failed upgrade, or anything else, one should well have the means to get it back to decent state at/around how it quite recently was - e.g. before starting the upgrade attempt, or your server room had a fire, or whatever.
Ok_Fault_8321@reddit
Major releases tend to recommend a fresh install.
iamsrsuguys@reddit
What kind of things go wrong? I am upgrading two servers again in a few days and haven't had problems yet (knocks on wood)
N5tp4nts@reddit (OP)
In my example, apache/php making version jumps causes all kinds of config file conflicts and results in a webserver that wont webserve.
StatementOwn4896@reddit
Yeaaaa php and Apache mm mm mm mm mm toasty
josemcornynetoperek@reddit
On desktop: upgrade. On server: clean install, because I can check if all what is running on it is really needed.
hunta2097@reddit
I've upgraded my lab server in-place for years. Never had a problem, Ubuntu is pretty solid.
Backup your OS disk if you really need to.
gmuslera@reddit
One of the advantages of containerization is to decouple the updates of those systems from the base OS. You just need the volumes somewhere, and from there on you have a lot of flexibility on what you do or how to evolve your infrastructure. Ubuntu have snaps (and you can install flaxpak or other alternatives) that somewhat does a bit of that decoupling/isolation, but they are more oriented to desktop applications and probably docker/podman will be a better fit for the things you mentioned.
Gendalph@reddit
I've containerized everything I could.
As for your problem specifically: interpreters and associated libraries have always caused pain. For PHP I recommend not using distro-provided packages, but instead go for Ondrej Sury's PHP PPAs. Other languages have their own solutions.
megared17@reddit
Install fresh on a new drive, then mount old drive somewhere convenient to copy data, settings, etc as needed
coffee-loop@reddit
I want to clear the air on this. Containers are a great solution, but containers still have a base image that needs upgrading as well.
N5tp4nts@reddit (OP)
True; I wouldn't be making my own. The things I use, someone else maintains them.