Why I can never be a sysadmin; or, Why is software like this?
Posted by OnTheEdgeOfFreedom@reddit | sysadmin | View on Reddit | 85 comments
This is not a very serious post; I'm just screaming into the void and hoping a few laughs and nods echo back; though there is a serious question at the end of it all. Below is an email I sent to my friends at 5am, after I spent all night getting a linux laptop running again. Of note: I know what I'm doing when I write code, but I'm completely useless at systems administration. My palms sweat if I need sudo for anything. I cringe at touching config files. dpkg? I don't do drugs, man, keep that hard stuff out of my life...
Without google I'd never be able to maintain anything. So when my laptop boots and there's not even an option to connect to the network... I'm sure you guys all nod and know exactly what happened, but I didn't, and while there's humor in trying to resurrect a laptop on Easter morning, it's not the kind of humor I like at 3am.
My email to my friends follows. Intended for humor but please consider the question at the end: why is it even like this? We've has OSes for 50+ years, and this happens?
---
I remember an old "Peanuts" quote: I love humanity, it's people I can't stand.
While I agree with that, I have my own version: I love programming, it's computer systems I can't stand.
I bought a new cell phone recently, because if you live in Costa Rica you need a Costa Rican phone number to do anything, and I didn't want to give up my US number, so yeah. I got something Samsung/Android based, cleaned off all the crapware games that immediately started nagging me to play them, got it all set up... the very next day, it died. Black screen no matter what I tried, but I could still wave the phone to turn on the flashlight so I knew something in there was working. I just couldn't use it. On new hardware? Why?
Tonight I thought I'd wind down from the game with some music, and fired up my laptop because for just music I don't need the full tower system.
Hm, no internet. Starlink glitched again?
But Starlink was working fine... hm, no list of available wifi. In fact no option to show the available wifis.
What?
I plugged in the ethernet cable. Nothing. I plugged in the apple phone for a hotspot over USB.
Nothing. How is this possible? The laptop's been working fine for days. I didn't do an update. How can so much hardware fail at once?
Google time (on the tower system because the laptop clearly wasn't going there). lsusb, lspci... the hardware is there. Searching for other causes.. no, I'm sure the drivers are fine, I didn't update anything.
Wait. Where did the drivers go?
Modprobe. Nothing.
Half the system is missing. Disk failure? I mean my wife's tower has a dying disk, maybe it's contagious. Run badblocks. Crunch crunch crunch...
Disks are fine. My personal files are all there. The disks are ok, so...?
More google. All it's coming up with is some sort of failed update. Which I know I didn't do because I have an unholy dread of updates. Ok, let's look...
The last update happened... 3 days ago?! Without telling me!? And based on the file sizes, it ran without completing, probably when the battery died, because initrd is a fraction of the size of the last good version.
Try to reboot into grub so see if there's an option to boot into the previous version. There should be. Maybe there is, I'll never know. It's about impossible to time the keypress right to get into grub, and when you do get in it freezes as you type commands. Mid-command, before you hit return. Ten or so cycles of reboots, nope...
I'm not sure why there's not a simple command to say "I don't care, delete the current OS and go back to the previous one." But apt wasn't working, and it's now 3am. Google kept lying. Fail. Fail. Fail.
In the end I had to make a rescue disk. It turns out that rescue disks don't have a tidy command to move the OS back either. More Google. You have to mount a handful of different directories, and what is chroot anyway, and then modify root's path, and in the end apt-install purge still doesn't work and you end up taking a sledgehammer to things with dpkg --remove --force-all. And don't forget to reconfigure grub because dpkg isn't your nanny, even if I need one.
Finally, reboot... oh look the internet is back. 5am. I can see the pre-dawn light out my window.
I've been using Linux for years. I remember the untimely birth of Windows, 40 years ago. And I know the horrid truth about them: Neither of them are yet ready for primetime.
Fundamentally, no system should ever boot into an incomplete install. There should be a pointer to the active install and it shouldn't be moved to a new one until the install finishes cleanly and passes some sort of self check. Roughly speaking, the failed updated was like putting a pie in the oven before you put the pie together; it makes no sense. But no, grub just looks for the highest version number and has no idea what's valid or invalid. Oh, it doesn't work and the commands to change things fail? Sucks to be you, pathetic userland victim.
So now I've discovered the unattended-update daemon and taken a sledgehammer to that too, because I never want a machine doing stuff behind my back.
WHY is it like this? 50+ years of OS development and all we have is systems that can't survive a low battery?
I'm going to bed, annoyed.
CommanderKnull@reddit
Look into Fedora Atomic or similar ummutable distros where the root file system is read-only, you can easily rollback when something fails and have to make a very conscious effort to make changes root-wise.
SirLoremIpsum@reddit
That is a huge amount of the job.
Sorry. But we all use Google all the time.
Someone asked me to update a hosts file to test something on Fri and I couldn't remember where it lived lol.
GuessSecure4640@reddit
Do folks not realize we use internal documentation too? OMG I fixed this bizarre issue 2.5 years ago, surely I'll remember the solution 5 years from now since I did it once...or, I'll document it and have my own "Google" specifically for this environment
Secret_Account07@reddit
I just had to add a static route and update a host file the other day. Something I’ve done a million times but don’t anymore because it’s horrible practice. Haven’t done in years.
Had to google because I couldn’t remember
accidentlife@reddit
One of the neat things about Linux is the host file is very easy to find: /etc/hosts
tajetaje@reddit
Also super easy to add temporary DNS overrides in /etc/resolv.conf
VernapatorCur@reddit
It's also easy to find on windows, but it's a couple folders deep
SenTedStevens@reddit
The HOSTS file path is burned into my brain because I had to deal with incompetent devs or people who refused to believe the issue was DNS-related. But it is in a weird location.
VernapatorCur@reddit
I had to add manual entries to the hosts file for several users due to an issue with our VPN. We finally changed VPN providers and I had to go through and clear the file on every workstation I'd modified
jaydizzleforshizzle@reddit
I’ll be honest the LLM are great for not having to remember vague syntax for everything, assuming you know what you want out of it.
Gsxing@reddit
Anybody that works in IT and claims they don’t use Google to solve their problems is a gods damned liar 😤
JEnduriumK@reddit
I solved problems like printers not staying installed in Windows 3.1.¹
This was back before Google existed, or internet was ubiquitously available.
I definitely use Google to solve problems today. But not all of them.
(But most of stuff today is just too complex to not use Google. You can't know everything.)
¹ It was a drafting printer. The printer drivers would install. They'd work. The next day they'd be gone. The problem? It turns out that if you boot directly into a program such as Automenu (no command prompt), and then launch Windows from there, and then select
Exit Windows, theSave Settings on Exitoption wouldn't actually save the settings until you also exited Automenu and got to a command prompt. The owner of the PC was leaving Windows, reaching Automenu, and powering the PC off without exiting Automenu as well.Exiting Automenu a single time saved the driver installation permanently. Removing Automenu from
AUTOEXEC.BATprevented future issues.GuessSecure4640@reddit
FluffyMcFluffs@reddit
I swear, IT is just learning how to Google properly.
transer42@reddit
And the experience to know how to apply what you've googled
GuessSecure4640@reddit
Well...and to know what to apply...people will do anything without vetting it first if they think it'll fix their issue
stackjr@reddit
Yeah, this is key. Knowing how to find the solution is one thing, knowing how to implement the solution is a whole different ballgame.
IAmMarwood@reddit
Was helping a user the other week with a real obscure issue and after failing to help her she got back to me a few hours later saying she’d found the answer buried deep on the internet somewhere.
Told her not only congrats on fixing her problem but thsts she’s learnt the dirty open secret to a significant portion of my job, it’s just advanced Googling. 😂
Smiles_OBrien@reddit
I prefer the term "Professional Guessing"
SpectralCoding@reddit
These days I use ChatGPT more than Google. Gets you to answers faster and in a way you don’t have to hunt through a bunch of ads and junk. “How do I refresh an external data source in Excel?” Answer in about 8-10sec, vs 30sec+ on Google.
pugs_in_a_basket@reddit
How about RTFM? This is the problem, answers, no context. How could you ever evaluate the reliability of the answers you are given when you know nothing to begin with.
SpectralCoding@reddit
Well a lot of the time the answer either works or doesn’t. If it doesn’t often (but not always) there is no impact to a wrong answer. For example, it should be pretty obvious in the Excel example whether or not the instructions are accurate… It tells you to press a button that doesn’t exist? Ask a clarifying question… It tells you the way to update an external data source is by closing and opening the document… Yet the data doesn’t get updated? No harm. It tells you to delete the data source, maybe that raises a flag for you… I’m not an idiot that doesn’t know anything about Excel, I just don’t know how to do that specific thing.
Same thing with the 10,000 other topics I know to a 100-200 level but temporarily need 300-500 level knowledge. Programming. Car repair. Medical advice. I’m not going to take 3000mg of Tylenol because ChatGPT tells me to. TBH, I’d be surprised if you could get it to advise you to take 3000mg these days.
Also, I’m likely to get an answer on Google from StackOverflow or Reddit or some random YouTuber before the Microsoft docs. Is that really any better than ChatGPT?
pugs_in_a_basket@reddit
And how is any of this better than reading documentation?
childishDemocrat@reddit
My advice if you use an LLM is.... Be sure you have enough base knowledge to know what it's doing. Especially with Linux where there are different flavors and versions and the LLMs will mix up instructions for two different flavors or versions in the same answer. Being able to figure out when it is lying to you can be the key to not having it lead you down a 2 hour path to messed up OS hell.
Oh and the comment about not booting into a fully updated system is spot on. We can do it for firmware (though not all devices do) why not OSes... It's not like people run on limited disk space much any more.
SpeakerToLampposts@reddit
Entirely true... but it's true of looking through Google (/DuckDuckGo/whatever) results as well. You have to know enough to sort out which results are actually relevant to what you want, often how to modify them to your particular situation, etc. There's no substitute for actual knowledge.
childishDemocrat@reddit
I think that those sources generally have better context though. Don't get me wrong I use LLMs frequently. But the mixing of sources in the same answer is only right about 60 percent of the time. Checking the original links to get context can help. As can good prompting. But none of the LLMs obey constraints like "always use current version documentation" for example
awful_at_internet@reddit
Yeah, but search results don't glaze you into thinking youre the best dev to ever set eyes on code.
pdp10@reddit
Subject-matter experts get the most productivity out of LLMs, but the hoi polloi aren't yet willing to fully admit that.
Whether the LLM is spitting out a foreign language, legal briefs, or computer code, not closely understanding the output, is playing with fire.
Cyhawk@reddit
Its the same with Google. I can still even though Google went into the shitter results to answer my questions quickly, because I know exactly what I'm asking for and how to find it.
Have you seen how regular people use google? Yeesh.
Kumorigoe@reddit
Because then they would have to admit they don't know what they're doing.
SpectralCoding@reddit
I just log into prod servers, ‘brew install copilot-cli’ and then run it as root with ‘--yolo’, paste the error message and let it fix it.
/s, but it would be cool to try!
Jarasmut@reddit
We do it for Android and Apple reinvented the zfs wheel with apfs so they do something to the same effect as well. Windows of course doesn't and Linux is predominantly used for servers where this functionality just isn't relevant. You have your linux up and running and when you want to update it, you simply don't. Instead some orchestration tool grabs a new linux install that already comes with the upgrades applied, configures it to your liking and whatever has been running on the old linux is automatically moved over. Then the old linux is destroyed. Update? Never heard of her.
And for LLM you are exactly right, you need to already have the expert knowledge to know when it's giving you bs.
Dreilala@reddit
I wonder if google and forums were enshitified on purpose so that looking at them without the use of AI becomes torture.
I distinctly remember the good old times with way less ads, way less cookie banners, way less paywalls and just people helping people cause they enjoy doing so.
DaChieftainOfThirsk@reddit
Same method, different wrapper. There will be ads in llm's soon enough...
OrdyNZ@reddit
Search / Ai yeah. Haven't touched google except maps for years.
OnTheEdgeOfFreedom@reddit (OP)
I've known some brilliant sysadmins who could practically close their eyes, lay their hand on a box and tell you what was wrong. I'll never be one. And trusting an AI to tell me what sudo command to try next is the worst feeling in the world. I know how often AIs serve up hallucinations and out of date info.
I still believe it shouldn't be like this. I know just enough about software design to know it doesn't HAVE to be like this. And yet, phones crash and latops screw themselves up with no human help needed....
Win_Sys@reddit
It’s not strictly brilliance, it’s mostly experience. If you work with an OS or software stack long enough, you can get to a point where you intuitively know where to at least start looking based on the error or behavior. I do networking now but when I first started in networking it felt like switches and routers were a blackbox as to what was happening under the hood and to an extent it is but having full access to the OS wouldn’t have helped me in 99% of cases anyway. At first I would just be guessing as to what the issue might be but over the years I taught myself, learned from others or fixed issues through trial and error. I now know several network operating systems extremely well, mostly through experience. I can usually pin point issues very quickly or configure entire switches without having to lookup commands. I have had people tell me I am a genius but I can promise you I am not. I can do it because of my years of experience and the countless hours of banging my head against the wall because I can’t figure what the issue is.
awful_at_internet@reddit
AI is helpful in that it can give you a new approach to searching for answers that you hadnt thought of. Helps a lot with X-Y problems, for example.
wowsomuchempty@reddit
DDG (& increasingly Claude..)
aeiouLizard@reddit
Which is literally nobody.
tankerkiller125real@reddit
I don't use Google... I use Kagi or hell even bing. I have found Google to be hell on earth to try and get answers out of over the last 4-5 years.
jfoust2@reddit
Once upon a time it all came from magazines. You'd need to read 'em all.
NeXtDracool@reddit
IMHO the ability to effectively search for information is the most important skill of any technical job. It's simply impossible to know all the information necessary for every task and problem.
badnamemaker@reddit
Yeah literally at this point even when I’m troubleshooting with my very experienced IT manager we will both be asking gemini what to do next 😂
Shadax@reddit
This is true for every industry. Your doctor googles things. It comes down to how the results are interpreted and put in practice b
autojack@reddit
I can’t count how many times I’ve been on with a vendor or VMWare support and half the call is us throwing back and forth pages we’ve found while Googling.
Impossible_IT@reddit
27 years in IT, Google is my best friend. LOL!
Jaack18@reddit
Yeah this is why i don’t use Linux on everyday devices. I don’t know enough to fix it easily
pdp10@reddit
To play devil's advocate, I'm not confident that a skilled user of macOS or Windows would choose to fix an analogous problem without a reinstall, which would also suffice for Linux.
Or would the famous
dism.exefix a non-booting disk?Frothyleet@reddit
It can, although it's usually more convenient to use a rescue disk to repair the install.
Cyhawk@reddit
Thats why you should use it. Break it, fix it, learn it.
Plus we have ChatGPT/LLMs now and you can run them locally. Even if not correct initially, it'll point you in the right direction.
flummox1234@reddit
but ... it stays consistent. Learn once, same forever. Whereas Windows changes with each release and when you get into the guts of configuration it gets "holy shit where is the kb for this" a heckuva lot quicker vs linux's man pages.
Jarasmut@reddit
Powering off a system accidentally during updates can bork most desktop operating systems. Grub does manage operating systems but what you reverted was merely the kernel version, the OS stays the same so that's why that didn't help.
What you said with pointer and not booting into an unfinished system is how Android works, the OS is essentially installed twice and updates are applied to the inactive OS. Only when the update is done and it's safe to boot into the updated version does it switch over. And if the first boot doesn't succeed it automatically switches back. Android will then come up like normal and tell you that the update failed.
This Android update process is called the Android A/B partitioning. We don't really do that with a desktop OS. You could absolutely do it and in fact Apple uses something for macOS where it creates a new OS "out of thin air" every time you update and keeps the previous one around to switch back to if the new OS fails to boot. (Their file system similarly to zfs is able to reference on the block level allowing to create logical structures called snapshots that allow states to be frozen in time by keeping track of the data on the block level. Essentially where other operating systems with their file systems would be too slow and old for this Apple has reinvented the wheel that's zfs to make upgrades more robust.)
You can use linux with zfs/btrfs as well and get this type of functionality on many linux distributions but this adds complexity and might give you a harder time to set up and maintain. In fact I avoid it because in my experience it's more likely to break than a regular linux install.
Linux is driven more by server use than personal computer use and linux servers are all about automation and scaling servers up and down based on load. What we have been doing for a while now is treat Linux operating systems as disposable. You have your linux up and running and when you want to update it, you simply don't. Instead some orchestration tool grabs a new linux install that already comes with the upgrades applied, configures it to your liking and whatever has been running on the old linux is automatically moved over. Then the old linux is destroyed. Update? Never heard of her.
This obviously doesn't work for a simple desktop computer installation.
It's funny to me because I am not a programmer and configuring linux and all that comes easy to me but code? That's nightmare stuff to me.
Frothyleet@reddit
Windows essentially does A/B when it comes to feature updates, for what it's worth. C:\Windows is kept around with ".old" appended.
Froggypwns@reddit
The now canceled Windows 10X was going to use A/B type partitioning. I hate that it never made it into production.
pdp10@reddit
It's not necessary for devs to know all this, but I'm putting you in charge of all of them, anyway.
The Steam Deck's SteamOS version of Linux uses dual, immutable, monolithic OS installs. OpenWrt uses a monolithic immutable install with overlays, and it's an option on Alpine Linux. We don't use it on general-purpose pets because it has more disadvantages there, than in an appliance or cattle install.
Windows doesn't have the option because Microsoft doesn't let their customers the OEMs, customize the OS in any significant way. WinCE ("wince!") was monolothic, and probably had the option. No idea with Windows 10 Enterprise IoT Edition, but I doubt it.
All this said, a truncated
initrdis the most common total system failure mode we record with Linux, hands down. On non-battery machines, it can happen when/bootis too small or overfilled.Nomaddo@reddit
I think it does.
https://learn.microsoft.com/en-us/windows/configuration/unified-write-filter/
The Unified Write Filter (UWF) protects the contents of a volume by intercepting write attempts to a protected volume and redirects those write attempts to a virtual overlay.
OnTheEdgeOfFreedom@reddit (OP)
|It's not necessary for devs to know all this, but I'm putting you in charge of all of them, anyway.
I'm retired. You can't make me. If nominated I will not interview, if hired I will not serve. I'm not going back to development hell. Do you hear me? Hell no, I won't go! DEATH FIRST!
Sorry, little panic attack there. I retired to a farm in Costa Rica because I donwanna tech no more.
(And I didn't know ANY of that stuff. Luckily, Gemini did.)
Speeddymon@reddit
Looking at this like an engineer: You have 2 root causes. 1: You thought your updates were disabled but they weren't. 2: You let your laptop battery die.
One of these you can actually solve for: disable the updates for real; but that doesn't solve the power issue or the pointer thing you mentioned. You're essentially talking about using a shadow copy of the OS installed, updates happen on the shadow copy and you reboot to activate the shadow copy. Then if there is any issue you can go back to the old release, revert the shadow and try again.
It's something I believe is in the works but it's still a few years out.
burdalane@reddit
I've been a hybrid sysadmin/developer for 20 years, and without Google, I would never have been able to do system administration at all, and I still wouldn't be able to do it now. I had very little IT experience when I was hired as a Linux admin. I had installed Linux once with all default settings and run a web server, and I could compile code on the command line. Everything else I learned by searching while on the job.
NeatRuin7406@reddit
this is the universal sysadmin experience. software that works differently depending on which version, which patch level, which OS version, whether the moon is waxing or waning. the specific thing that kills me is that the behavior you're describing isn't even a bug in most cases -- it's an intentional design decision made by someone who had a very specific scenario in mind and never considered yours. documentation tells you what the software does when it's happy. it has nothing to say about what it does when it's encountering your edge case. i've been doing this long enough that my first response to any weird behavior is no longer "that must be a bug" but "what was the developer assuming that i'm not satisfying." usually that framing gets me closer to the root cause faster than reading the docs again.
Froggypwns@reddit
For what it is worth, Windows these days tends to handle power loss while updating fairly gracefully. If anything interrupts the update, including unexpected loss of power, next time it boots up it will rollback the changes.
childishDemocrat@reddit
Yeah remind me of that the next time I have to enter my bitlocker key for the 50th time just so it will boot lol. It has improved. But it's still entirely possible to hose your machine. Especially if it's in the middle of recovering from a hosed install and gets reinterrupted by an inpatient user ....
Halen_@reddit
Yeah this has been a solved problem on that platform for a while. I'm guessing OP just hasn't spent a lot of time with it.
JEnduriumK@reddit
Android does/did this.
https://source.android.com/docs/core/ota/ab
1776-2001@reddit
- Linus. Coincidence, or a glitch in the Matrix?
OnTheEdgeOfFreedom@reddit (OP)
I quoted that deliberately. I'm old enough to remember when that was first published.
1776-2001@reddit
"This line was spoken by Linus Van Pelt in the November 12, 1959 comic strip of Peanuts, written and drawn by Charles Schulz (1950-2000)." [source]
OnTheEdgeOfFreedom@reddit (OP)
What's your point, youngster..
TechSupportIgit@reddit
Maybe they can't conceive of greybeards using Reddit.
In the 80s and 90s, would you just be doing a lot of company funded training with vendors while heads in the manuals out in the field? Kinda curious how the IT landscape looked like back then.
skat_in_the_hat@reddit
WE HAVENT HAD OSes FOR 50....Oh... shit... i just realized im old now.
Mrhiddenlotus@reddit
I don't want to be a hater, but this is the kind of thing sysadmins normally complain about when it comes to devs.
NotMedicine420@reddit
The answer to your question is in your post.
Maybe you do, but 95% of programmers don't. Even though they may think they do. Hence crapware everywhere, bloated, slow and bugged.
OnTheEdgeOfFreedom@reddit (OP)
I am very proud of my track record in the software industry. 40+ years, I wrote many hundreds of thousands of lines of code in several different companies. In my professional lifetime I got back two bug reports, one from my first week at my first job and the other one was caught internally.
The secret of my success? I never did anything web based; web=eventual failure. It was all low level C++ with dashes of python. I avoided dependencies other than the C++ runtime like the plague. (Dependencies have bugs. They just do.) I absolutely hate it when anyone can prove me at fault for anything so I made very sure it never happened. All my stuff was backwards-compatible and portable with a simple recompile because I didn't want someone coming back in 5 years with an issue. My cubical was mecca when other people had problems.
(And I will admit that my record with private projects, stuff for me and my friends, is nowhere near that good. But no PHB ever caught me out.)
But sysadmin? I can't. My problem is I know how it should work, any programmer with a brain cell would have provided a tool to do X and check Y, and when that tool doesn't exist my brain shorts out. Of course it exists. Google is just lying. It's not possible that you can't just fark the foobar and have it just work.
Programming is "easy" - you just learn certain immutable patterns and never-evers and the compiler catches your typos. Sysadmin has no such safety net, no reliable patterns that are simply always true. There be dragons.
NotMedicine420@reddit
I have deep respect for C/C++ coders especially in this era. Devs like you are like unicorns.
OnTheEdgeOfFreedom@reddit (OP)
Devs like me are retired and not coming back. :)
I saw so much bad code in my career... and I've watched the decline in skill over decades. I cut my teeth on assembler on a PDP-10 (google it) and gained a deep respect for error handling and efficiency. You HAD to know what you were doing. There was no internet to look things up on. You RTFM and memorized it, you learned what shortcuts were going to bite you in the end...
Those skills translated to C and then to C++. This, young Jedi, is a raw pointer - an elegant weapon from a bygone era. Master it and the world is yours. From misuse comes suffering...
I watched as people started copying and pasting from the net - from examples where error handling was "neglected for simplicity." I watched as people could hide incompetence in group projects (not a thing when I started out - the project was yours start to end).
In the end I took a job in defense and I won't even try to describe that that was like. Young cheap kids trained on web technology put to work on ancient tech and languages they'd never imagined. It was a horror show. Armies of extensive testers, because nothing ever worked the first time. I retired the first moment I could - the angst of seeing what we going down was killing me.
Yeah, it's the end of an era and vibe coding will kill us all.
sully213@reddit
Just because the program compiles successfully does not mean it's good code 🙃
flummox1234@reddit
this guy compiles ^ 😜
q123459@reddit
rant
since halting problem exists most such heuristic system can achieve is a checklist.
and because breakage can happen anywhere you would need to have llm that can self bootsrap on your set of hardware Or you would need to have degree in computer science + all source files so you can code yourself all needed components that were not installed.
since modern cpus are not fast enough for llm to complete such task in reasonable time (under 1 year) there is only second option available /rant
flummox1234@reddit
As a greybeard programmer who started when you still had to do the sysadmin side yourself, you basically just experienced the reason why everything moved to "the cloud" and why so many are willing to pay exorbitant prices for it, aka, why the profession of "devops" exists. For me I'm more amazed at home much people will pay to avoid having to do the system side of things.
mdervin@reddit
TLDR: Congratulations or I'm sorry that happened to you.
AdeptFelix@reddit
Story makes me appreciate DNF's history and rollback features in RHEL-based systems. Super easy to revert updates, and holds 2 prior kernel versions before removing old ones.
Escanut@reddit
I remember having some type of mid (I'm in my 20s so early?) life crisis when setting up wsl2 and doing some kubernetes tests.
I was pissed alright, but then I remembered when I didn't understand how to install Gta VC on my dads computer and GOOGLED the answer and how good it felt.
Maybe you just need a career break or just read some books or something.
H8FULPENGUIN@reddit
Feels like half my time these days is spent talking through issues with Perplexity.
Immediate-Panda2359@reddit
Any UNIX-like system made after 1990 and lacking "ifconfig" in the default install is a tool of the devil. "ip a" my shiny metal ass. Sincerely, a guy who started as a sysadmin in 1987.