Linux veterans, what is your most "my system is destroyed"-type of story? And if you did, how did you eventually identify and fix it?

[-]

_usually_wrong_@reddit

Some root process used 100% of the partition with the root filesystem...

Which was formatted as ReiserFS. Kerbango, instant corruption. It was a moment of satori: When the big kernel dudes throw shade on something, best to pay attention.

I can't recall what year that was. Something like 20 years ago.

No issues since then, and I've managed as many as 600 physical nodes in a cluster.

[-]

AvGaadM5@reddit

Memories fade.

It's an old DEC Alpha computer via cluster picked up from NASA years+ ago, ~97.

The firmware was the boot media. When you turned it up, it immediately went to the firmware looking for your command to boot Linux.

Linux on a DEC Alpha was pure dreamy. It was screaming fast, quick. When there was snow in the winter months, like November to February, the computer would keep the room warm. It was a bit ridiculous though a powerful machine.

The firmware image was sensitive. There was no recourse or general knowledge of what to do if it failed.

Well, the ECC ram, SCSI spinning iron, 300 mHz RISC - somehow the firmware image didn't take.

There's not much to do with a low level brick. I still have the 86nm chip on my desk. That particular RAM should've been sold years ago.

Somehow, learning the pain of rm -rf accidentally is trumped by a firmware image missed write.

[-]

Unaidedbutton86@reddit

I did chown -R apache:apache / instead of /mount/x, there wasn't anything important without a backup though so I reflashed that testing server

[-]

Coffandro@reddit

I currently have an arch install that is, for reasons unkown to me, slowly killing itself lol.

Random applications just will not without me restarting at random times.

[-]

iBN3qk@reddit

sudo chown www-data:web -R /

Oh shit I meant . not /

Not destroyed, but I had to call an adult.

I ctrl + c’d before it got to my bosses home dir.

Just the dev server…

[-]

mattzog@reddit

A classic story... an errant space...

rm -rf / directory/name

I killed power when I realized and had to reinstall from scratch.

Always keep backups!

[-]

Demortus@reddit

Holy cow, I always wondered how someone could accidentally rm -rf their system.. Good to know!

[-]

iBN3qk@reddit

I thought I was getting helpful advice in a forum 🙄

I hear that rm -rf / throws a warning now…

[-]

AnEagleisnotme@reddit

To be fair that can't happen with modern RM, but RM -rf ~ /something would

[-]

Good-Throwaway@reddit

Replace rm with chown -r someuser:somegrp

or chmod -r 750

If run as root the effect is disastrous and no way to recover from other than restore a whole file system from snapshot.

[-]

alkatori@reddit

The other would be a script that has an empty variable for a directory

[-]

Accomplished-Sun9107@reddit

I remember having that aliased to echo "are you fucking kidding me!?" - you know, just in case..

[-]

DarkestStar77@reddit

Been there. For me I meant to remove a sub folder in my home folder, but ended up deleting my home folder. Really hard lesson, and one I have but forgotten over the past 20 years.

I remember the panic 5 seconds in when the command hadn't returned yet, and it should have. Killed the power, but way too late.

[-]

harrywwc@reddit

yeah, did that once as well - although it was a FreeBSD 2.x system, not Linux - so there's that I guess ;)

[-]

No_Shirt9277@reddit

It happened to me something similar but worse once. I used to have my password manager database file stored at ~/ (yeah, I know) and I did something like

rm ~/Downloads/ *

from ~/ deleting all files in home (including the password manager database) and getting a bunch of the usual warnings saying that directory is not a regular file and can't be deleted like that.

[-]

sunkenrocks@reddit

With an equally horrific prefix of sudo or doas as well, I'm sure.

[-]

mattzog@reddit

but of course

[-]

sunkenrocks@reddit

Probably one you didn't even need either but it's annoying to see an error and type sudo !!, and I've never made a mistake before. Why now?

[-]

Uppapappalappa@reddit

upps, that sounds damn realistic.

[-]

PlayerIO-@reddit

Thankfully on most distros you need to use —no-preserve-root now

[-]

throwaway234f32423df@reddit

wasn't --no-preserve-root (and default root-preserving behavior) added in 2006? was this before then?

[-]

mattzog@reddit

Sometime around 2005 - 2006 when I was a total linux noob

[-]

mattzog@reddit

I was using the laptop at the time, trying to delete a USB drive. I went back to doing my normal stuff while it worked and then things began to just strangely... stop... working...

[-]

mattzog@reddit

I was using the laptop at the time, trying to delete a USB drive. I went back to doing my normal stuff while it worked and then things began to just strangely... stop... working...

[-]

NotScrollsApparently@reddit

I really don't get why they haven't put in same safeties for this by now. Just ask me one more time to make sure if I'm trying to rm f or chown r or anything like that on /, cuz I probably don't!

[-]

Unsigned_enby@reddit

When trying (successfully, technically) to rm all of the files off my system from a given package, I failed to remember that pacman includes shared directories (such as /usr/bin) in the list of files owned by the package. So I ran rm -rf /usr/bin on arch-linux, which stores essentially all binaries in /usr/bin. I got suspicious when it it took more than a few seconds, but by the time I realized what was happening, it was too late. All hope, and all my executables, were lost.

[-]

stubob@reddit

Yep, I didn't do it, but I saw it happen. We were building a Solaris box (so you can guess the timeframe) and one of the scripts didn't 'cd' correctly and ended up doing rm -rf /. The box crashed before things could get too bad, but he still spend a week rebuilding that box inside a SCIF.

[-]

wasabichicken@reddit

I did something similar, except with the home directory.

What saved my ass was the home partition being ZFS with daily snapshots. Restored it from yesterday, hardly lost a days work. Could've gone a lot worse.

[-]

dlrc93@reddit

I have a Windows 11 / Linux Arch dual boot. Windows only for gaming. I have 1 HDD and 1 SSD. HDD for data and SSD for the OS bootfiles.

Had a Windows 11 update. SSD with the bootfiles corrupted. Couldn't login into either of the systems. No problem. Just reinstall windows on the bootfiles SSD onto the partition with windows by reformatting the SSD and install Windows and Linux into their respective partitions. I didn't know I was actually installing windows onto the data HDD. Meaning I've reformatted the data HDD and and overwrited data on the disk with Windows. Data was then gone for good. But I had a backup. So happy end.

[-]

Sync1211@reddit

I installed the Nvidia driver on a laptop with Optimus graphics back in 2017. Xorg was bricked even after removing the packages, so I needed to reinstall.
I tried to fork APT (wanted a nicer progress bar) and ran sudo make install, ended up with 2 apts and promptly uninstalled half my system during the next upgrade.
I resized an LVM volume (Proxmox pve/root) without the --resizefs parameter which corrupted my rootfs.
I moved a LVM volume to another drive and used lvremove instead of lvreduce which promptly removed most of my LVM volumes. (Fortunately, I was able to restore it from LVMs automatic metadata backups)
On a jailbroken Kindle: Moved files into /bin without checking for conflicts, overwrote system files with incompatible ones. Still bricked as I can't get into fastboot.
Not system-breaking but stupid nonetheless: I wrote Rasbian to my USB stick instead of the SD card. Lost 10GB of archived projects from my IT class. (Not devastating, but they would have come in handy quite a few times.)
(Bonus) I started my Linux journey by daily-driving Kali. It wasn't very reliable, but I've learned a lot by just fixing the issues I've had with it. Switched to Debian a few months later.

[-]

Any_Extreme_6106@reddit

After an update in Arch Linux, went back to Debian and never look back again.

[-]

-_Mad_Man_-@reddit

not a veteran, but I do have a story still

Quite a few years ago I was dual booting windows with Pop!OS, and had windows first on the drive

One day I decided that I wanted to get rid of windows and only run linux, but since windows was first in the partition table, I wasn't able to expand the main linux drive to the left, since the bootloader/grub was first

Simple, I thought. I remove the bootloader to expand the main partition and them remake the boot partition

However I royally messed up and ended up with a bricked system and couldn't boot

Since I wasn't familiar with what to do in this situation, I just ended up reinstalling the system, learned to not mess with these types of thing haha

[-]

mdins1980@reddit

Many years ago I was working on a buildscript to install a package. I was piecing it together manually to test something, in the directory was a subdirectory called etc. I wasn't paying attention and I ran "rm -rf /etc" instead of "rm -rf etc". And as other can see there was no recovering from that. I nuked the machine and had to reinstall. Nowadays I would do this in a VM to avoid these types of screw ups but this was pre VM's. You live and you learn.

[-]

kremata@reddit

I'm on Linux for 7 years now. Never had a crash big enough that my system was destroyed. The worst thing that happened was that I made so many mods that the system became very slow and I lost patience troubleshooting so I just reinstalled the whole thing.

BUT...

On Windows, one time I was working and suddenly realised the systems overload. Ransom virus was in the process of encrypting all my files. Luckily for me it started in a folder containing my old websites PNG files. So I turned off the computer, saved my data on another disk and reinstalled. This is what pushed me to switch to Linux

[-]

SeyAssociation38@reddit

I once ran out of both ram and swap space on windows and the system froze with a gray screen. I have also had several BSoDs when trying to do stuff while the system is heavily loaded or when using USB tethering from my phone to connect my computer to the internet

[-]

Gangsir@reddit

How did you go from "oh my system is overloaded" to noticing that it was a virus?

Did you see the virus in task manager? See the files being converted to encrypted versions?

[-]

kremata@reddit

When I noticed my cpu was running full pin I started to check a couple of things and in my search I saw a strange .txt file on my secondary disk. This file was created by ransomware to leave instructions on how to pay them. So I immediately turned off the system. I probably could see it in task manager because how else could I know where to look. But I don't remember this part.

[-]

PhillyBassSF@reddit

I have experienced more than once my uefi bootloaders missing. Fixed it.

[-]

daemonpenguin@reddit

Two come to mind.

Once, back in the early Linux days, I was upgrading my C library (the library virtually every application links to in order to work) and I accidentally typed "mv old-version old-version.backup" instead of "cp old-version old-version.backup" prior to putting the new copy in place.

Everything stopped working immediately. No command line programs, no desktop programs, everything except my current shell was immediately dead in the water.

I booted off my distro's live media, mounted my hard drive, and copied the original C library over to my hard drive, then rebooted. Everything worked again.

On a more physical side of things, once I had a hard drive start to burn a hole through itself. It would heat up, start to smoke, and stop spinning. I had backups, but I still wanted to get fresh copies of everything off the drive to preserve recent work and my configuration to another drive. A friend came over with a spare hard drive. We plugged it into one port and booted the computer to try to transfer files over.

When the drive got too hot it would smoke and stop, and this happened pretty quickly, making file transfers difficult. We ended up packing the drive in ice which kept it spinning long enough to transfer everything off to the new drive.

[-]

HurricaneFloyd@reddit

There was no live media in the early Linux days. You must have a different idea of what early Linux days means.

[-]

feckdespez@reddit

Get out of here with your gate keeping...

Live Bootable distros have been a thing for nearly 25 years at this point (Knoppix released in 2000).

Linux made his famous post in 1991. 9 years to the first "live" bootable distro. 24 years from then until now.

Another way to think about it is... more than twice the amount of time has passed since the first live bootable distro was released compared to the amount of time passed between Linus's first post about Linux and the release of the first live bootable distro.

[-]

RealUlli@reddit

What OP describes is very close to what I experienced. Except I didn't make a backup of the file. Caused every command to segfault, I had to power cycle the machine, after which it recovered.

Pure luck. It happened in 1995 or 1996, no live Linux then...

[-]

JL2210@reddit

echo -n > old-library < old-library.backup ;)

[-]

daemonpenguin@reddit

Imagine if that actually worked? That would be pretty cool. It won't, I think because as soon as echo hits a null byte or EOF indicator it'll stop out putting, and you'll end up with an empty or mostly-empty file. But it's a neat idea.

[-]

JL2210@reddit

I think the bigger problem would be that the file isn't created with executable permissions

[-]

daemonpenguin@reddit

Not sure the permissions would be a bigger problem than having an empty file..

Especially since libraries do not need to be executable on Linux to work. The user just needs read permission.

[-]

is_reddit_useful@reddit

Once, back in the early Linux days, I was upgrading my C library (the library virtually every application links to in order to work) and I accidentally typed "mv old-version old-version.backup" instead of "cp old-version old-version.backup" prior to putting the new copy in place.

That is an example of why it is good to keep statically linked rescue tools installed. One example is busybox-static in Debian and Ubuntu.

[-]

SlimlineVan@reddit

That's gold! It reminds me of an old clean joke my dad told me as a kid.

Q: What do you do if your girlfriend starts smoking in bed?

A: slow down and use more lube

Seriously though, I've had my fair share of crappy hard drives but that is too much even for me. Data recovery literally was a ft job back than

[-]

Simple_Anteater_5825@reddit

That's similar to a line from a film that went something like:

"You guys are pretty old, why do you still use a condom?"

"Because I love the smell of burning rubber!"

[-]

SlimlineVan@reddit

That's probably more apt now too than back in the day ie just as we now know IC engines are superseded by EVs, burning rubber was probably part of their world when burning any carbon is bad nowadays. Old timers burn rubbers even when they're not stuffing the planet!

[-]

SpaceCadet2000@reddit

In the early days I remember installing a glibc rpm from Redhat 6.0 on my Redhat 5.2 system. I was trying to get some program or other working that required a newer glibc, and I naively thought I could just install the Redhat 6 version and it would be okay.

The system never recovered from that, I had to reinstall everything. Nowadays I could probably get it back up, but I was just learning Linux at the time.

[-]

vlaada7@reddit

Oooh, I’ve done the very same thing myself! The other one was overwriting my grub with a u-boot intended for beaglebone black. Check your disks before using dd 🫤

[-]

E-Mobile@reddit

I can imagine doing something like that in the shop and then charging the client for ice 🤣

[-]

lordofthedrones@reddit

Zip lock bag and freezer. It worked way too many times back in the day

[-]

kudlitan@reddit

I'm not afraid to reinstall since I have /home on a separate partition.

[-]

AntaBatata@reddit

There are a thousand different ways to still destroy /home.

[-]

Either-Trash-2165@reddit

Fired Ethernet controller of a motherboard... The CPU was a i7 extrem edition who was 2 years old a the time, specific socket so at my company since they don't have anything to trouble shoot the issue I got it dor free. I just desoldier the ship who short circuited thé motherbaord. Input it in my EXi cluster at home.

[-]

LordSkummel@reddit

I screwed up my X server 15-16 years ago so that I couldn't get into the gui of my ubuntu install at the time. I think I just reinstalled and was happy that I had /home on its own partition.

[-]

Moo-Crumpus@reddit

Only once. A lot of hidden configuration files had accumulated, which I wanted to delete recursively. Hidden files are marked with a leading dot, aren't they?

So the expression .* finds all hidden files, I thought, great. And since there were also files that other users had left behind, I gave myself superpowers.

sudo rm -R .*

I got myself a cup of coffee, deleting the files should be quick. I smoked a cigarette, drank my coffee and came back. The hard disks were still rattling - and I almost wet my pants at the horrible realization:

... also fulfills the pattern .* and I was about to recursively delete all the system's data.

Worse still, my backup consisted of DAT tapes that I had neglected for years and actually wanted to rewrite after cleaning up. Hahaha. Double dumb.

[-]

met365784@reddit

Mine isn't too outrageous, but my main system utilizes multiple hard drives, and I added entries to fstab for each of them. Everything worked fine, until one of the drives had failed, so I wasn't able to boot into the system. Used a live distro to boot, and made sure to add the nofail option for all the extra drives. It was a simple fix, but something I am more mindful of nowadays

[-]

AnnieBruce@reddit

That caused me problems when my RAID got corrupted somehow. Thankfully once I got nofail in there and booted I was able to fsck it and it got all the data back.

[-]

AnnieBruce@reddit

A couple weeks ago I tried to install the Mesa update from debian backports.

It did not go well. At all. It took me some time to massage grub into booting to a command line, from which I had to go through several rounds of removing every trace of Mesa via Aptitude, then reinstalling XFCE, and after a few hours I was booting normally to my desktop. Overall 3-4 hours of work, and I had a few applications get uninstalled because Mesa touches so many things that getting rid of it nuked some dependencies.

Oddly enough, after this, some system instability issues I've been having for a while went away, like, Firefox stopped crashing 3 seconds after launching. My best guess is that some other thing I did(I am not a strict follower of the don't break debian guide, at least I admit I broke it and it's not debian being bad) screwed up a dependency somewhere, and this whole process got that dependency fixed.

[-]

NewmanOnGaming@reddit

There was this one time when I used arch and an update to timeshift somehow deleted a backup and blew out a partition. Needless to say it was an unexpected issue with the initial timeshift update that was pushed and I had to completely rebuild my previous arch install.

Second one was due to a borked update from another AUR package that caused me 9 kinds of grief. Fortunately I never rely on data stating primary on my local system and keep offloaded backups.

After about a year of AUR breakage occurring off and on I eventually moved away from Arch and stuck to a different distribution. Been using it since.

[-]

Clottersbur@reddit

Uhh... So, one time out of no-where I couldn't play World of Warcraft because it would just black screen. It was working fine on Arch then one day just broke. Couldn't figure it out for months.

So, anyway I installed another distro and it worked. Went back to Arch and it still works.

I didn't run any updates or change anything whatsoever between working and not working.

My mind still is blown.

[-]

CompetitiveAlgae4247@reddit

i dont think this counts but i bricked my pc trying to install windows 10 fron arch linux

[-]

CompetitiveAlgae4247@reddit

to be fair i used balena etcher which doesnt work on windows iso files

[-]

throttlemeister@reddit

Recompile the kernel to include a needed device driver, happily rebooting only to be greeted by a black screen showing LI with a blinking cursor because I forgot to include another critical driver that was in the default kernel but not marked yes by default in the makefile

[-]

JoeLikesGreen@reddit

Been there bro

[-]

JL2210@reddit

I usually copy the config from the running kernel and use oldconfig to make sure I don't break anything. But you also have to rebuild dkms modules if you have any of those

[-]

throttlemeister@reddit

It went over your head I think. 😁 Just teasing. But I did just dated myself. The LI blinking cursor aka a boot problem with LILO predates dkms by a long, long time. Life was a little less... Polished... Back then. Like that the default config for the source tree of a distro was not necessarily the same as the installed binary for that distro. It was typically just the extracted tar.gz straight from kernel.org without modifications. And you had to know exactly what hardware was present in your system to be able compile a working kernel. And no you couldn't just turn on everything, as that would create a file that was too big. LILO would crap out on you. Also a time where help wasn't given, it was earned. You had to do the work, or else rtfm. Little patience for stupid questions you could have easily found the answer to by reading. Tough love but I still remember most of the stuff I learned back then. We're talking before windows 95 here.

[-]

dve-@reddit

Quite recently I was working on my old school Thinkpad, you know the ones that still have a removable battery pack.

So what happened was that somehow the battery pack was loose and it lost contact while I was in the middle of editing my nix.config

For those who don't know: the perk of NixOS is that it is purely declarative in a way that you define almost EVERYTHING in that one file, instead of installing and configuring things by commands. Well, that one file somehow was gone/empty after I lost power.

I didn't make a backup or used a git report like you are supposed to. Good lesson.

[-]

1008oh@reddit

On an ubuntu work laptop, I managed to break the system python so hard when I tried to update it that there was no other recovery than reimaging the entire computer...

[-]

Arneb1729@reddit

Fucked up Ubuntu installations are the main reason why the Python people accepted PEP 668 which basically blanket bans `sudo pip install`.

[-]

sunkenrocks@reddit

Oh, yeah, I've done that one too.

[-]

Synthetic451@reddit

I had an Ubuntu machine, I think it was version 10.xx or 11.xx, that just completely borked itself during an upgrade to the next major release. I got pissed off and just switched to another distro after that. Didn't even bother to figure out what was wrong.

Another time, I accidentally dd'ed into my external storage drive and wiped all my media files. Took years to rebuild that collection T_T

Besides that, not really any major fuckups.

[-]

Outrageous_Trade_303@reddit

sudo chown myusername -R /

I had an identical system so I was able to restore the ownership

[-]

skuterpikk@reddit

Forgetting the encryption passkey imediatly after a server was fully configured and ready to go, it rebooted into oblivion.
And of course the classic ignorance induced "Franken-Debian"

[-]

freenullptr@reddit

15 minutes after reading this post, my laptop forcefully shut down (extremely low battery, there's some issue with calibration) in the middle of a full system upgrade (Arch). On the next boot, only Windows showed up in rEFInd.

I booted into Windows, got an Arch ISO and Rufus and found a random USB stick around the house (one of those fake ones, this one advertises itself as 64GiB to the OS and is in reality 8GiB - at least it's enough for a live medium). I burned the image, got into UEFI settings, disabled secure boot and set boot order correctly then booted into the live system.

The next issue I was faced with was that I had no network device other than loopback. Seems like the live image is missing some driver that I have installed on my actual system. I connected my phone through USB, set it to USB tethering and thankfully got an ethernet device to appear.

I then mounted the system into `/mnt` and chrooted into it. Thankfully, before I ran the system update I had to clear out the package manager cache (to get back some space), and so every single package that was in that cache was a package that should have been updated. Using some `sed`, I managed to grab a list of all package names that I then fed into `pacman -Sy` to reinstall. Had to add an `--overwrite '*'` in there too, because installing failed on most packages due to some files that were installed onto the filesystem, but hadn't been recorded as belonging to a package.

I ran the reinstall twice on the live system (some post-installation commands failed both times, but fewer the second time). Then, I rebooted directly into UEFI settings with `systemctl reboot --firmware-setup` (very useful command), set the boot order as it was before, re-enabled secure boot and finally managed to get back into a working system.

It's funny how life works. This kind of thing hasn't happened to me in years

[-]

M3n747@reddit

I once pulled a sudo deluser on the system's only sudoer.

[-]

jon-henderson-clark@reddit

Unix, but I forgot to end the passwd file w/ a return & left for the morning. A restore from backup was done in my absence & we lost some data.

[-]

Kendos-Kenlen@reddit

I destroyed my partition table. I was able to fix it by creating the same partitions with exact same size.

[-]

is_reddit_useful@reddit

https://www.cgsecurity.org/wiki/TestDisk is good for that. It can search for partitions and automatically create the partition table.

[-]

s_elhana@reddit

"sudo chown user:user -R ./" on a wrong terminal window that was in /, then after some moments ctrl-c, but damage was done.

Lots of stuff stops working and it is easier to just reinstall rather than trying to fix it.

[-]

is_reddit_useful@reddit

Nowadays GNU chown also has --preserve-root, like rm, but unlike rm, --no-preserve-root is the default. One can alias chown="chown --preserve-root"

[-]

NotScrollsApparently@reddit

I did this few days ago because in a moment of rush I forgot the dot. I canceled it in a second and everything seems to still work fine? But I'm eyeing it suspiciously all the time now...

[-]

Talistech@reddit

It's time to start checking logs :-D

[-]

NotScrollsApparently@reddit

This is very much a "he would be upset if he knew how to read" situation because I'm new to linux and I have no idea where the logs are or how to interpret them 😁

In my defense during that second I saw a lot of permission errors about being unable to change owner despite sudo so I just assumed fedora has some built-in root protection after all

[-]

joeblough@reddit

Probably close to 20 years ago, I put Mandriva(?) on a second partition of my work laptop for dual-booting ... however, the kernel didn't recognize the motherboard / fans ... and as a result, Mandriva booted, but fans NEVER came on ... laptop got alarmingly hot to the touch and finally just quit ... it took some time for it to cool down and boot back into Windows ... ever since then, even today, when I install linux on a computer (laptop or desktop) I make sure the fans come on as expected! Even if they're driven by the mobo ... I still don't breath easy until I hear a fan hum.

I've never had that problem since ... so probably a one-off, but that's my "funny" "I killed a system" story.

[-]

is_reddit_useful@reddit

Even if the fans are supposed to be controlled on the motherboard, it may be possible for the OS to override that. So, it is good to check.

[-]

HurricaneFloyd@reddit

Your fault for overriding your BIOS hardware fan control.

[-]

stormdelta@reddit

One of the nice things about UEFI now is that it seems like fans default to being controlled by it if the OS doesn't do anything.

[-]

inevitabledeath3@reddit

That applied even in the BIOS days. It's quite uncommon for your OS to have to implement fan control. Normally it just has the option to tweak or override the existing fan control provided by firmware.

[-]

mmcgrath@reddit

I was called to look at a system that was "behaving poorly". This was a production system so we needed to investigate, etc. After logging into the system I immediately got errors, basic commands like ls were failing but the shell was working. Instead of ls I switched to using "echo *". cat was failing but I think I used "gzip -c ./file | zcat" which was working somehow.

Long story short, a more junior associate was trying to install a package they had downloaded but for a different version of Fedora than was on the system. It required a newer glibc and they managed to find the version.. from SUSE..., get it installed or mostly installed with an rpm --force. Then the system started failing. Knowing we weren't attacked, and that there wasn't a bug or hardware failure, we rebuilt the system and met with the associate to explain why that's a bad idea and the difference between a production and development system.

[-]

is_reddit_useful@reddit

It is good to keep some statically linked rescue tools installed, like busybox-static in Debian and Ubuntu.

[-]

Asystole@reddit

Man, that's the definition of "just enough know-how to be dangerous" right there.

[-]

atomicxblue@reddit

I've had to rebuild my rpm table from scratch. It was not fun, which is why I moved to Debian based systems.

[-]

_-Kr4t0s-_@reddit

JFC. I hope that since then you guys have adopted Ansible and CI/CD or some other tooling to help you with that problem.

[-]

lonelypenguin20@reddit

it sounds a lot like it was before all that fancy stuff. u know, when servers were actual computers, not virtual machines inside a cluster of containers or some shit

[-]

mmcgrath@reddit

This was at Orbitz.com (travel site in the US) many lifetimes ago :)

[-]

_-Kr4t0s-_@reddit

Oh. I don’t know if I can say what they do or don’t have, but I know the answer. They (well, Expedia as a whole) used to be my customers 😁

[-]

fleshofgods0@reddit

This almost sounds like the dependency hell that I ran into on Linux Mandrake when I was 10. It left a bad taste in my mouth and dissuaded me from using Linux until I installed this brand new distribution that had JUST been released called Ubuntu. Once I borked my WinXP install by trying to change the Start menu panel's Explorer.exe for a cool new alternative I found (and could no longer boot into XP AT ALL), I finally stuck with Linux because of how you can ALWAYS troubleshoot, look under the hood, and find the actual source of your problem and resolve it.

[-]

Yondercypres@reddit

I removed Root account on a system once by accident, not even sure how. It would boot sometimes, and I was able to log in as a user and get my data off of it, but anything that required elevated permissions at all didn't work. WiFi, GPU drivers, browsers, etc.

[-]

ShijoKingo33@reddit

most recent one, I was testing ollama in my computer with python virtualenv and decided to move from python 3.10 to 3.11 and it broke my cinamon desktop completely even though both were installed and reference was python 3.10, so reinstall and docker was the solution to avoid that.

[-]

GavUK@reddit

About 15 years ago my PC's hard disk started to develop bad sectors so, having a few disks laying around, I decided to use dd (or ddrescue, I forget) to copy everything from the failing disk onto a larger one. However I mixed up the source and destination, and only realised when I got a error about running out of disk space - I'd completely wiped my data on the failing disk by overwriting it with whatever it read off of the one I'd intended to copy to, and hadn't backed it up for months...

[-]

venquessa@reddit

My system was destroyed.

It was a brand new Linux From Scratch experiencing it's first boot. Unfortunately the VM I had been using to build it ... had the classic VMWare clock skewing issues and for some reason all my files had future dates.

After having bash freak out at me repeatedly for files that don't exist (yet) I decided it was not worth trying to fix and did the unthinkable...

rm -fr /

A few seconds later I seen a permission denied, then another, then a bunch. IMMEDIATELY began spamming CTRL+C

Too late.

My "shared media" drive mounts with no root squash as I put my dev files on that disk.

So the rm -fr / without any restraint on "partition" was happily working it's way through the network drive as well.

I did lose a bunch of stuff, including half my MP3s. I learnt a valuable lesson about mounting things with no root squash.

[-]

YeOldePoop@reddit (OP)

Hey man, doing LFS and even booting is itself a victory though. Kudos.

[-]

erodedstonestatue@reddit

messed around with arch linux, accidently formatted my drive with windows on it, too broke to buy a new license. guess I'm stuck with linux from now on

[-]

EtanSivad@reddit

A decade ago, I was running Ubuntu 14 as a home file server and trying to get FreeNX to play nicely with the SSH connection I had setup to my work computer. It was also the first time working with Linux 64 bit.

I forget the exact command, but I had told it to uninstall python-minimal and python2.7, and it ended up installing the 32bit versions of the libraries. What I do remember (even took screenshot) because it was the first time I'd ever seen apt-get say:

"You are about to do something potentially hormful.

To continue type in the phrase 'Yes, do as I say!"

And I had to literally type in "Yes, do as I say!" not just "Y".

Spoilers, I didn't want to do that because the 32 bit libraries just blew up when trying to load.

It was pretty easy to fix. I got to a command line (Forget if it was through grub recovery or just an alternate terminal.)

I uninstalled the 32bit libraries, then did

sudo apt-get install gnome unity

and it reinstalled itself from scratch. This was my file server at the time, and the desktop was never quite right after that. Ended up wiping the drive and reinstalling a few months later.

[-]

Antinomial@reddit

Installing an OS to the wrong partition, overwriting existing OS and data.

[-]

esdraelon@reddit

I was given the specs of the programmable north bridge frequency divider (this set the front side bus for the system) hanging off SMBUS and told to see if I could write a kernel driver that would modify the FSB from the command line.

Next day, I show up with my trusty gigahertz frequency counter and the server I borrowed and wanted to show off. Senior engineer wants to see the work.

I believe the FSB at the time was 1 GHz. I carefully type in 1.05 GHz, sure enough, the frequency counter scoots up to 1.05.

Now, mind you, this is a $40,000 server.

So he comes back with the equivalent of "well, how high will it go?" I'm saying well ... 1.05 Ghz. See? Pointing to frequency counter.

But he's a senior engineer. He claims we have the best mortherboard guys in the country and he want to see what happens closer to 1 GHz.

So I start walking it up 100 Mhz at a time. Somwhere around 1.6 or 1.7 GHz, I see an amber LED start glowing on the motherboad. I basically said something like "uh-oh, we got a warning indicator" to which he responds "that's a resistor" and then that server never really worked again.

I have no idea what became of that code, but he was giddy with joy and drove back to main campus with it.

[-]

Inevitable_Score1164@reddit

I tried to apply two years worth of patches to a SLES server with a zypper lock on kernel-default. Tried to play the zypper "choose a solution" mini-game. I did not, in fact, choose a solution. Rather, I broke the system quite thoroughly. I found out when I still couldn't SSH into it 30 minutes after reboot. Luckily, it was a VM.

[-]

dudeness_boy@reddit

Not me, but my friend was running fedora. He went on and tried to use gparted, which was installed, but the system couldn't find it. He tried "sudo dnf install gparted", and it gave the error "sudo: dnf: command not found". he decided to shut down and try again in the morning, but when he typed "sudo shutdown -h now" it said "Permission denied" even though he used sudo.

[-]

pppZero@reddit

killed a few installs in my years. usually caused by being on some kind of bleeding-edge distro, not updating for a few months, then seeing it wants to update (everything) and just sending it. Sometimes it works, sometimes you're getting a fresh install. Destroyed a number of gentoo installs trying to do things with CFLAGS.

the worst one was an OCZ SSD i had in a laptop I took on holiday with me once. Filled it with photos and videos documenting a couple of months of excursions around the countryside, and it died the night before I got home to my NAS. Turns out that model drive had a firmware bug where sometimes after going to sleep, it'd never wake up.

[-]

strohkoenig@reddit

When my parents needed a new PC, I did the select and buy the hardware as well as installed Manjaro and chose KDE as DE. Biggest reason was that Manjaro was said to be rather user friendly at that time and it also is rolling release so - in theory - I can just install updates from time to time when I'm at home and don't need to bother with major version updates.

While I had installed it in a VM beforehand, I hadn't tested the VM over a longer time span because I was quite busy at that time. I'm also not living at my parents place anymore but several hours away, visiting them once every few weekends.

When I installed it, everything went fine. Then after some weeks, I came home again and updated the system, everything went just as expected.

One week later they called me: They've tried to boot the PC for the first time since I was home and it didn't work. It was quite difficult to reschedule everything I had planned for the weekend but I managed to do that, went home and spent the entire weekend trying to fix the issue. After two days of not being able to boot the system, I finally found a way to get into the OS again. First attempt to hopefully solve the issue by updating the system again failed - apparently the previous update had destroyed the updater, it didn't work anymore. I did more research on the internet and by pure chance found out how to fix the issue: for some reason, activating the automatic login setting so that my parents don't have to enter a password at boot broke SDDM. Disabling the setting allowed KDE to get back up but now my parents had to enter a password to login.

Sadly it was already the end of weekend and I didn't have any more time to solve any other issue so I told them the password and asked them to deal with it for the time being. They didn't like it but oh well, it couldn't be changed.

I initially planned on getting them a new OS as soon as possible but with a lot of stuff happening, it was more than one year til I got to actually do it. Now they're running Fedora KDE and it works like a charm. I now run all my computers on Fedora KDE and I'm quite happy how stable the system is, while still delivering new versions quite fast.

I've not used Manjaro ever since. Dunno what caused the issue but apparently they hold back updates for a week before releasing them or something? That's what I've heard so maybe incompatible versions could've caused it.

[-]

strohkoenig@reddit

Another thing happend years earlier, when I was around 15 years old and tried out linux for the very first time. I got it from a Computer Magazine which was popular at that time in Germany. For 2009, they were already quite clickbaity, packed Kubuntu 9.04 or 9.10 on it and advertised it as "the new Windows now that Vista ruins Windows" or something.

Considering I didn't have a good internet connection at that time to do some research online about what Linux is and what their idea is and also have only used Windows beforehand, I decided to test it out and immediately followed the instructions in the magazine of how to install it via dual boot.

...

I mean, it looked better than Vista I guess. However, all my good hopes faded away when I wanted to insert a CD-ROM into my disc drive to install a game... As it turned out, Linux doesn't really run .exe files, especially in 2009. So well, considering that wasn't possible, I was buffled on what to do with this OS. Since I didn't find any use and I was annoyed by GRUB always selecting Linux first and Windows second, I decided to simply remove Linux again by deleting all non-Windows paritions and merging the created free space back into the Windows parition.

So I went on, deleted the XX GB Linux partition, merged the free space into the Windows partition and rebooted the PC. Huh.. GRUB was still there but now when I didn't input anything, the preselected Linux would not boot anymore. Man I just want my PC to boot into Windows Vista again automatically... But hey, in the Partition manager I had seen another weird partition. It was only 100 MB big and called something something MBR? Dunno what it is, so back into the Windows partition it goes!

...

My PC is dead.

Panic.

What do I do, what do I do, WHAT DO I DO???

Well, I was able to boot the Live CD with Kubuntu on it again so I installed Kubuntu AGAIN and hoped for the best. Luckily, GRUB was back and I could boot again. Sadly, Kubuntu is also back again!

Still, I didn't have the courage to touch the system again for quite some time.

At some point, I found my Windows rescue disc which was delivered with the Laptop and could basically completely remove Kubuntu and GRUB from the PC using it.

Still, it was probably the worst start I could've possibly had with Linux. Over the next few years, I've tried a lot of different distros and DEs but none worked well enough. Either I didn't like DE or Linux or I liked them but they bricked after some weeks. I especially dodged any KDE 4 plasma workspaces distro as that failed Kubuntu approach has really scared me away. Ironically, at some point almost 10 years later, I was testing Linux Mint, liked their three DEs (Cinnamon, XFCE, MATE) but all of them bricked after some weeks of use. I had given up on using Mint at that point so out of fun, I installed the KDE edition which existed back then. It was a blast, KDE Plasma 5 was beautiful, fast, worked like a charm and didn't brick!

Linux Mint stopped supporting the KDE version soon after, but I've used KDE 5 ever since. It's my favorite DE by far and it's ironic how KDE 4 was initially my most hated part about Linux.

[-]

hailsatyr666@reddit

Tons of stories. Got a park of 40k+ embedded Linux machines. Whenever we do a SW upgrade, a bunch of them always go down and you gotta go with a serial cable to check what's going on. Majority of those cases had an unexpected reboot happen during SW upgrade. Depending on when the reboot happened, the damage and symptoms may differ. Common scenarios are grub2 config goes missing - in that case I need to manually enter config or use a backup and rebuild the original config. Another scenario is either initramfs or a new kernel installation is cut off in the middle. In that case you boot from rescue kernel and reinstall kernel or rebuild initramfs with dracut. Another common case is duplicate packages left after interrupted yum transaction.

[-]

Inevitable-Series879@reddit

I realized that I was trying to make Arch too minimal and removed a bunch of packages that broke the system, this happened countless times until I realized I’m the problem and now I just make neovim minimal

[-]

gaggzi@reddit

rm -rf / by mistake on my friends server back in 99 or something

[-]

nicothekiller@reddit

One time, I accidentally deleted the entire efi system partition. Predictably, the pc didn't boot. I managed to fix it with chroot, reinstalled grub, and it worked.

[-]

fsover2@reddit

Obligatory XKCD success

[-]

linuxlib@reddit

48 hours:

I am surreptitiously upgrading the AI cluster in the Nuclear Facility Control Room. In just 15 minutes I will be able to play MineCraft while I monitor the reactors on the edge of supercriticality to avoid being bored out of my mind!

48.5 hours:

Well, that didn't w

[-]

_DraXX@reddit

r/redditsniper

[-]

raptor7197@reddit

r/ofcoursethatsasub

[-]

Swizzel-Stixx@reddit

Didn’t think I would see one of them here lol

[-]

Uppapappalappa@reddit

didn't know this one :)

[-]

purefan@reddit

`nixos-rebuild --build-host 192.168.1.45 switch` same arch, original configuration.nix too... yet when it finished, not even the mouse moved, couldnt even get the bootloader up... had to format and reinstall. Since then I ssh, its kind of tedious but I enjoy Nix and NixOS

[-]

loblowpoopydoop@reddit

Killed root access totally. Thank god I use multi partition for everything so copied out my usr partition reinstalled deb with my original credentials and put the usr partition back and got access to the system again. This was early days using Linux and I didn't have much to loose except my 10s of hours customising qt to look how I wanted, still run exact same config to this day visual and partition wise. I think I tried running a Kali install from within my deb install or some such malarkey and botched all root access.

[-]

Zero_Karma_Guy@reddit

It was when I got my new laptop. I booted it and it had Windows on it and I thought "my system is destroyed" bit then I put Fedora on it and it was fine.

[-]

daveysprockett@reddit

Do not remove /dev/zero from a Solaris system.

Almost everything dies, but there was the vestiges of a rescue shell.

In an attempt to restore it, I noted /dev/zero is an interface into the kernel and from another system determined its major and minor device numbers.

mknod $major $minor /dev/zero mknod: can't open /dev/zero

mknod used /dev/zero as its source of initialised bytes.

OK, we're re installing.

[-]

dagbrown@reddit

I bet the -r boot flag would’ve sorted that out easily. That’s just a reboot, much less traumatic than a reinstall.

[-]

daveysprockett@reddit

Don't recall what we tried, but it was a VERY sick puppy, with multiple attempts at reboot. I don’t even recall why we were so keen to restore, though possibly there were issues getting a tape drive onto it. Also don't recall from 30 years ago what -r might have done. You must have a better memory for old systems than I do.

[-]

sunkenrocks@reddit

You also just might not have been aware of -r lol.

[-]

dagbrown@reddit

Really, it takes a special kind of nerd to remember Solaris boot flags and what they mean, decades after such knowledge became irrelevant.

[-]

dagbrown@reddit

The “-r” flag stands for “reconfigure”. You normally use it if you’ve added disks or something, but it forces a complete rebuild of /dev from the boot archive (that’s Solaris-ese for “initramfs”) which is why it came right to mind.

[-]

daveysprockett@reddit

Thanks. I'll remember that for next time.

More amusement about that incident was that I fetched a boot floppy (or at least something on a floppy drive), pushed it in to the slot and heard the crash of it falling onto the case floor. No drive included, as it was a "secure" machine.

[-]

Spiderfffun@reddit

I havent been using linux for a while but once I rm -rf-ed all my python packages on arch by accident

[-]

FilipoPoland@reddit

Not much of a veteran but just felt like sharing.

Well on my first install of arch where I had to deal with nvidia I think that I messed up the hooks for it. Well one day, when I was updating which I do practicly everytime I boot my computer, the update seemed to freeze. I waited. The screen went black but I could see my fan rotating so I kept waiting. However after solid two hours or more I decided to use brute force with the computer and just turn it off.

When I booted it up again I was greeted with nothing. I tryied to fix it with a live USB I identified that my initramfs was missing.

At the end of the day I decided that a careful reinstall is probably a better idea. To this day I still do not know for sure what happened.

The lesson for me was do not ever disrupt pacman and make sure to have the right hooks for nvidia.

[-]

Capt_Picard1@reddit

Just a few months ago, upgrade your Ubuntu 24.04.1 crashed my system. Would when boot anymore. But was able to fix it by reinstalling grub into the boot partition

[-]

Thor-x86_128@reddit

Do not use dd while drunk, it's a big mistake

[-]

computer-machine@reddit

I'm running a Debian server via a bunch of Docker containers, where the volumes and config live on a btrfs raid1.

At one point, after switching my pihole to manage DNS and then back again (hadn't worked out; didn't have time to fiddle), every boot had a race condition that would cause the CPU to lock.

Sometimes I could manage to cd to the docker-config.yml location and stop it before the CPU hit 100% and the system became unresponsive forever, but that stopped being a thing. I tried loading into a super minimal shell, which worked fine, but as soon as I loaded enough modules to mount the raid, it would lock up, so I couldn't get to the docker file to change anything to not load automatically at start.

Eventually I just backed up /etc and reinstalled, and everything worked fine from there.

[-]

PotatoNukeMk1@reddit

One time i prepared a few old hard disks for selling them. I used dd to write noise and zeros in multiple runs.

Then i accidentally i used the wrong device for dd and deleted the systems ssd. Not also the data were gone, no, also because of excessive write to the ssd it got destroyed.

[-]

DocToska@reddit

I think sooner or later THAT seems to happen to everyone. At least I say that to myself to feel less stupid about my own related muck-up.:p

I once toasted my work laptop while trying to wipe data from an attached USB disk. I'd love to say that nothing of importance was lost, but it was. This was freshly after emigrating to another continent. I lost the only licensed copy of my billing, accounting and tax software and the DVD of that was in storage 10.000km away. And I still needed to declare my end-of-year business tax for the country I had just left behind. The vendor of the software refused to send me a download link as I was no longer within my support period. So this turned into a costly endeavor. :p

[-]

_-Kr4t0s-_@reddit

Protip: always use the flag ‘bs=4M’ with dd. On hard drives it will run faster than the default because it’ll take better advantage of NCQ and sequential write speeds, and on SSDs it won’t burn them since it will line up with the flash page size instead of performing multiple write cycles per page.

Of course a SECURE_ERASE would be better for erasing SSDs but the dd trick still applies if you’re trying to flash an image to one.

[-]

michaelpaoli@reddit

Not my system, but here's interesting(/harrowing) story of someone's Linux system that got significantly broken, and how I got it working again: How linuxmafia.com got back to being operational again

Also had a fair number of case, different employers and work environments, and yes, very much including production, and various *nix flavors ... drives mirrored ... one dies - folks are asleep at the switch, not monitoring/checking, not paying attention ... then the 2nd drive fails. That's when they hand it to me, and tell me also that there aren't any backups and it's critical production data. Yeah, gotten a fair number of systems like that operational again. And many such drives, I only got to spin up one more time - just enough to back 'em up or mirror them to a good drive ... and then those drives would often never spin again after that.

Various other "interesting" stories, but that'll get you started, at least.

[-]

Upstairs-Comb1631@reddit

Updating LIBC or rm -rf /. :D

[-]

lunakoa@reddit

I once did a dd if=/dev/zero of=/dev/sda bs=1M count=100 on the wrong machine. Instead of rebooting and hoping for the best I was able to do a backup dump on some of the file systems and /boot a well. Noted things like partition size, labels etc and was able to recover the system.

[-]

DocToska@reddit

A few years ago my Linux workstation was also serving as a file server and had a lot of internal HDDs and external USB disks attached. In the process of moving those disks (and in some cases just their data) to the new dedicated file server. I had pried an external USB enclosure open to get the the HDD out to directly connect it to SATA, as moving 1TB over USB would be much slower. In the process of that I dropped the HDD onto the tiled floor. /facepalm

And sure enough: Afterwards it couldn't be mounted anymore.

The disk contained no critical business data, but a whole heap of mostly irreplaceable personal documents and digitized family memorabilia. I resisted the urge to experiment with the disk itself. So the first thing I did was to pull a disk image and made all my rescue attempts with that image instead.

Diagnostics on the image showed that the partition table was bad and so were all copies of it. I tried a whole heap of rescue tools (some freeware, some commercial) and they all totally misjudged the partition size and some were even wrong about the file system type. I got *something* recovered when I tried a tool called "Foremost". It can can recover certain file types such as images, videos and some documents, but the recovered files have no name (other than numbers) and are structured into directories of the same file type. JPEGs go into a /jpeg directory, AVIs into an /avi directory and so on. Which wasn't particularly useful.

Unable to recover anything in a usable fashion, 16-17 months passed. And with each passing month I got even more painfully aware of what a treasure trove I had lost and pondered if and how it would be possible to recover at least some of that.

Eventually I heard about a free tool called "TestDisk" and gave it a whirl on the disk image. On the first attempt and without any hassles it managed to recover about 98-97% of the data. A few directories and files were garbled, but nothing important. \o/

[-]

ben2talk@reddit

I was using Windows Vista, I encountered a blue screen.

I fixed it with a CD, and installed a brown screen (Hardy Heron I think).

[-]

The_Real_Grand_Nagus@reddit

I suppose the worst thing that ever happened to me was a mobo failure. Any hardware issue (mobo, sata cable, memory error, etc.) that results in inconsistent behavior are the worst because in the beginning you're not even sure if there really is a problem. That being said, I think what you really want to hear are any "lessons learned" running Linux (or anything else really) for years.

Mine is: always have backups. I don't mean you have to have a super slick backup system, just have something that works for you, keep copies of all your important data on a separate drive (preferably separate location), and develop a method to consistently make and track these copies. After a few hard drive failures in the 90's/early 2000's where I had to use tools to recover as much data from drives as possible, put them in freezers to get them working again, etc. I finally bit the bullet and just bought an extra drive. I ended up eventually rolling my own backup system using rsync and hard links (think r-snapshot style thing), but you could use something like back-in-time which can do the same thing and has a lot of options.

Of course, I'm not fully on SSDs yet. I've been able to keep my spinning disks alive for a really long time. I swear, life-time hours matters, and just turning my machines off over night seems to keep the disks healthy for a lot longer.

[-]

susosusosuso@reddit

You fix it by reinstalling Linux 👹

[-]

remic_0726@reddit

I needed to install Linux, and during the installation Linux erased the firmware of the CD player. After the second drive died, I searched the net, only to find that this damn distri had a bug like this.

[-]

undying_k@reddit

Somehow I got a job where the main distribution was Gentoo. I was glad because I used it myself. On the first working day, I was given the task to update the database, Postgresql. To update it, I had to update a lot of libraries, so I just decided to rebuild the world. However, something went wrong in the process. The old glibc has been deleted, but the new one has not been installed yet. As a result, the entire system and database crashed. It was a pretty stressful first day at work, because it was the main and only GEO database in the company.

[-]

QuickSilver010@reddit

I tried setting up fingerprint without properly following instructions 3 years ago. I got permanently locked out of sudo

[-]

K-Stelao@reddit

Not a veteran, but a few months ago my laptop kernel-panicked while upgrading the system (arch, btw). Some important packages were being upgraded (iirc, the kernel and linux-headers among them).

When I rebooted, the files of the packages being upgraded were there, but all of them were empty. Pacman treated the packages as if they were already updated, so upgrading again didn't work. Reinstalling again didn't work either, as pacman didn't want to overwrite the corrupt files unless every path was explicitly specified.

So, I had to force-reinstall each corrupt package involved in the upgrade, specifying every empty file that had to be overwritten. It took a while, but everything worked as usual after that.

[-]

Owndampu@reddit

Ive had the same, twice you can actually do pacman -S --overwrite="*" packages to force overwrite all the files of those packages. I borked my glibc one time so I couldnt even arch-chroot, had to pacstrap my system into working again lol

[-]

tiny_humble_guy@reddit

Not a veteran, but let me share. About a week ago, I made my system broken due to libc musl failure upgrade. Can't be saved and need to be reinstalled. It kind of make me feel bad because it's not ordinary distro, it's source based distro.

[-]

patopansir@reddit

the first time I used Linux it was Ubuntu, and it was the first time I had to install an operating system. At the time, either I didn't see the warning that it would delete the data or I misunderstood it

I was a child so backups are not an option. I don't have money. I could store a few things thanks to USB drives

Well. All my anime was gone. I had it all so neatly organized and every folder itself was a guide on the order of watching them, it was a lot of it. 500GBs maybe? 100GBs? Thankfully, I was able to at least someone recover a list with some of the anime, but I still have to download it. I still have that list and I don't want to redownload it. I don't care anymore.

I remember I had the entire Sword Art Online. Maybe I had Lupin and Detective Conan. I had everything from Devilman Crybaby but I was able to recover most of it. I have no idea how I recovered it, all I know is that I still have it.

[-]

bigredradio@reddit

I purchased a used IBM Power system to run AIX and Linux. It wasn't cheap. When it came in I decided to update the firmware. I thought it hung, so I power cycled the box. Instant $$$ paperweight. Because it was used, no IBM support contract. Eventually found a way to fix it via a serial connection and floppy disks.

[-]

HelicopterUpbeat5199@reddit

Not most, but definitely funniest. Back in the early 90s I was installing Slackware from a shoe box full of floppies, like you did. After all the floppy disk nonsense, felt like hours, I got to pick the font for the terminal. One of the options was Klingon. I thought that would be cool, so I picked it. Turns out Klingon is not a font you can print English in. And this is the terminal in the installer. There was no way to undo without learning Klingon. I gave up and started over. Which meant feeding in .. all ..the ..floppies ..again.

[-]

ObjectiveGuava3113@reddit

I ran pacman -Syu once

[-]

-Happyx@reddit

used debian netinst, fking bootlooped my comp. solutionis go to repair store

[-]

raydleemsc@reddit

You know how close '.' and '/' are on the keyboard? One operates on the working folder, the other on the root of the file system. OK, now watch out for that when sudoing in root, because practically any update or hail mary on a local folder to recover an app or user can seriously mess with the system when under pressure. Abandon hope all ye who ENTER here. Thinking of the consequences is well worth the time to rebuild the server (from backups, but still).

[-]

symcbean@reddit

Reading a lot of war stories isn't to make you a Linux expert / is not going to make your computer more robust.

It doesn't matter how much you know about Linux or computers, you're machine will get trashed at some point. Tested backups are how this becomes an irritation instead of a disaster.

[-]

YeOldePoop@reddit (OP)

Oh, for sure. I fully expect to make mistakes and even some that have been mentioned here. I just enjoy seeing how some people approach issues and solve them as opposed to how I would, it's interesting. I also value the experience of veteran users.

[-]

No_Shirt9277@reddit

I thought: Kali is just Debian with some extra repositories. Let's just replace my Debian's /etc/sources.list with Kali's, update, upgrade and reboot and see what happens. So... It didn't reboot.

[-]

pikecat@reddit

See below for flaming notebook.

I've developed a good maintenance/backup procedure to guarantee my uptime.

It involves using my root system on 3 partitions, ideally on different drives. With each significant upgrade I rotate the active partition, the one the I boot into and use.

Each one has an entry in grub, so if any one fails, I can immediately boot into another one and be working.

The rotation goes like this:

reboot into the 2nd newest one.
delete the oldest
copy the newest onto the newly free partition
fix grub and fstab
reboot into the new copy
now perform upgrade. The former newest is now 2nd newest, the backup.

==

My best destruction and recovery stories are mostly hardware related. I forget the borked systems because it's been too long since any.

Once my notebook literally caught fire while using it. Flames were coming out above the keyboard near the screen. There was that distinctive smell of burning electronics.

Obviously I shut the power off as fast as possible.

It turned out to be the power board for the screen. I managed to buy a new part, installed it and my notebook worked perfectly, despite the melted and partially burnt plastic cover plate.

[-]

sophimoo@reddit

few days ago i ran rm /* on my debian server, idk if anything broke because i had to give permission and even though i say y it still didn't work. maybe it's borken but ssh and everything still works

[-]

turdmaxpro@reddit

I'm no veteran, but early on, early 2000s, I was always breaking everything lol. I didn't understand any of what I was doing, and had a Red Hat disc, liked it but starting installing apps that had missing dependencies. Googled and found missing dependencies that also had missing dependencies. On and on until it was broken . Then I tried Ubuntu which was a lot easier, but would break everything by installing wine and following random tutorials to get all the Windows apps I was trying to bring with me.

Flatpak is what convinced me to try Linux again, and I'm very thankful for it.

[-]

JerryRiceOfOhio2@reddit

i git into dependency hell a couple of times, just reinstalled, no biggie

[-]

Eubank31@reddit

Not even a veteran but I 100% removed python from my fedora system that requires python to install packages (which includes python....)

[-]

hugh_jorgyn@reddit

Been using Linux since 1999. Of course I tried rm -rf / as root, on purpose just to see what happens.

But unintentionally, the worst I've fucked up my OS is by going to far uninstalling packages to make space, especially back in the days when 1GB was a lot for a HDD. "Why is this library taking so much space?? I don't need it!", lol.

After a bunch of trial and error, I had got pretty good at keeping a super lean operating system with plenty of available space. But it took quite a few fatal errors to get there. I did that with windows 98 and XP too, btw.

Nowadays, with so much space available, I don't even bother removing any of the bloatware that comes with the distribution.

[-]

sunkenrocks@reddit

Nowadays, with so much space available, I don't even bother removing any of the bloatware that comes with the distribution.

That and two weeks later you were left wondering how much was placebo and its actually running the same, causing you to tweak more...!!

[-]

YeOldePoop@reddit (OP)

I almost made a similar mistake. I was thinking I was smart by writing a script to sort packages by their last update and I thought "Great, now I can just remove all the old packages!"

I didn't realize how dumb I was being until after writing the entire script... I sort of just sat back and went, "Wait no, I am being stupid". Yes, it took me that long. I am surprised I havent destroyed my system yet, one of these days for sure haha.

[-]

hugh_jorgyn@reddit

Oh, I've made so many stupid mistakes. The one that I'll always remember is when I was ssh'd into a remote machine at work and I was documenting some network config stuff and for some god damn reason I decided to run "ifdown eth0" on the remote machine. I don't even remember why I thought I needed to bring the card down, but I realized my stupidity the millisencond AFTER I hit Enter. Too late. Connection gone. Embarassing call to the IT guy who had the key to that machine room. Got cussed at for making him get up from his desk and walk all the way across the building to reboot the machine.

[-]

sunkenrocks@reddit

Somehow deleted my GRUB partition about a month into using Linux. 20 years ago and as a tween, without a smart phone or a second computer in the house besides my dads work laptop... That wasn't an easy fix to come up with!

[-]

5f4d65@reddit

Thought I was rm -rfing a local directory, nope, was at /

[-]

olafkewl@reddit

One funny : I was working late on an issue with an application on a server, after long time, I've ended up fixing it. I was happy and ready to leave, so I entered "poweroff" in my terminal, but ma laptop did not shut down, only the ssh sessi9n did I've turned off the server without being able to restart it because I had no access to VM control

[-]

heliruna@reddit

We were shipping an ARM device running Linux, using a chip from manufacturer A on a board from manufactorer B. In case you are not aware, ARM devices have no BIOS like a PC to discover hardware, it needs to be specified in a device tree compiled into the system. Manufactorer A switched RAM vendors, we missed the announcement, new hardware arrives and doesn't boot. We get the new drivers from A, they are completely different from the previous version (they support all their devices in one massively complex file) and are incompatible with the patches from manufactorer B against the previous version. The device boots but cannot do any I/O anymore. I took the board schematics, spent a week reading kernel documentation and wrote (that part of) the device tree from scratch. Worked better than before.

I upgraded to a new version of ubuntu, after the upgrade any time I use ethernet I get a kernel panic. So I boot Linux on my old laptop, connect to the new machine via serial, capture a kernel stack trace, report it to launchpad, a developer gets back to me, I build a kernel from source with his patches on the ubuntu kernel, copy it over serial to new machine, get a crash, report it back. We iterate that about three times, it was a known bug (with a known fix) in their kernel patches that only triggered on some hardware.

When I started going to college, I discovered that you can fix a lot of things by booting into a second Linux installation and chrooting into the first. Or if that doesn't work, wipe it and re-image it with debootstrap from within a running system. I was dual-booting two linux installations from separate physical hard disks and treated bare metal OS installations like we do containers today: I don't care if I break it, I can easily rebuild it. In retrospect, it is surprising that I never accidentally wiped my home partition.

I am old enough to have used a rescue floppy disk booting a linux kernel.

[-]

indiancoder@reddit

About 10 years ago, I had a RAID-10 (mirrored and striped) Array in my home server, with 4 disks. I never got around to setting up email notifications. I checked in on it fairly regularly, and it always seemed fine. But that day, I checked the server, and one of the drives had failed.

No problem I think. I shut down the computer, run to the store, and buy a new hard drive to replace it. I install the new drive, and begin re-assembling the array. But it doesn't finish. A second drive fails, and a third is showing signs of flakiness. The MBR was on one of the failed drives.

I shut down the server, bought a pair of new hard drives for it, and started over. I put the old drives in bubble wrap for an attempt at recovery at a future date. I was too scared to do it right away, and it wasn't long before I lacked the storage space to save drive images. As it turns out, that future date was just last week. Finally having enough storage to do the job, I made images of all the readable disks. One of them was perfect, one of them had a bad sector, one of them was the spare, and the other two had completely failed. Using the images, I force-assembled the array, and copied my data over to my server. There's 4k of corruption in there somewhere but I don't know what file is corrupt. I tried to figure out what file it was, but I had already mangled the data a bit with a file system check after the imaging, so without starting all the way over again, I won't know which file. I figure it probably doesn't really matter. It's probably some old MAME CHD, given the rough location on the disk.

But my laziness almost cost me all the bulk data I had hoarded since high school. I got very lucky in that I had a complete copy of the data on the remaining two drives, rather than 2 copies of half the data. My new server has email notifications, and I'm using drives better suited to the job.

[-]

Linuxologue@reddit

I want to do cross-compilation so I do a lot of chroot-ing in a system with QEmu and installing Debian for another architecture (aarch64 and ppc64). Problem is that the relative symlinks in the chroot work fine but not outside of it (which would be necessary for cross-compilation) so I tried to write a script that fixes the relative links.

I was in the directory but not chrooting (I didn't really need to be since I was doing filesystem operations). I wrote a for loop that lists all symbolic links and I first wrote a noop that just lists the link and the target. I wrote this:

for i in \find ./usr/ -type l`; do j=`readlink -f $i`; echo $i -> $j; done`
that was just supposed to print it but that printed nothing. I was confused.

I tried to run it again but this time it told me there was an error executing the binary, because my ld.so was dead. I was like what???

It took me 10 minutes to understand my stupidity. I was so sure there was no side effect. In fact, my pretty arrow -> was actually the redirection operator. I had listed all symlinks in my chroot, for each of them I wrote the name (plus a dash) into the target file. And because I had used "readlink -f" the target files became absolute directories. I had just overwritten some files in my root system when making what I thought was a dry run operation in a subdirectory.

There's no recovery, there's no binary that can be run anymore, back to the install disk.

[-]

stormdelta@reddit

Yeah, two lessons you learn pretty quick with bash:

Always quote variables unless required not to
Never use backticks, always use $(...) syntax instead.

[-]

Buddy-Matt@reddit

I've trashed my initrams more then once. Ventoy and chroot sorts it out in minutes every time.

[-]

jdfthetech@reddit

I once messed up my fstab when trying to add a partition as part of the /opt directory. I fixed it but it took like a half hour to figure out what I messed up.

Always keep a recovery drive around just in case.

[-]

oregszun@reddit

rm -rf / home/whatever

backup fixed it!

[-]

cpt_justice@reddit

When I ran "rm -rf *", wondered why it was taking so long and realized I ran it in a root terminal where pwd was "/". Good times.

[-]

kombiwombi@reddit

Corruption of DNS caches throughout the world. Tracked it down to enthusiastic systems administrators on Solaris and Linux systems turning off chexksumming of UDP packets to squeeze a little more performance on their authoritative name servers, a packet being corrupted, and that being cached by DNS. Some Solaris even had it off by default, because NFS benchmark tests were a thing.

Recursed through the entire DND hierarchy, emailing then ringing sysadmins as needed to enable UDP checksumming. Thankfully whois worked in that era.

Raised a fault with Sun, sent a email to LKML. These days the checksum is added by the interface hardware, and so there is little performance gain from disabling it.

[-]

kombiwombi@reddit

Should have added. Linux was great for networking. Hiding things people should not use deep in nerd knobs. Making it hard to disable ethernet auto negotiation meant that only people who knew what a poor idea this was were able to do it.

[-]

-illusoryMechanist@reddit

There's this tool called Junest - https://github.com/fsquillace/junest - that lets your run Arch commands on a tiny Arch distribution loaded on top of your current distribution. (It's really cool actually- you can use it to run software you normally couldn't via proot, as well as more generally the AUR on non-arch systems.) Anyways, I was trying to delete all the files in it, so I stupidly used sudo rm -rf --no-preserve-root to clean it out quickly. The problem was, in my efforts to make switching back and forth smoother, I had given it full file system access somehow (I don't remember exactly how I managed it) and so it jumped outside of the Junest instance and wiped my machine clean. I fortunately had a backup of my files, unfortunately it was only partial so I did lose a few things but nothing I could wholly live without.

Don't run sudo rm -rf --no-preserve-root unless you're trying to destroy everything, even if you think you're sandboxed.

[-]

YeOldePoop@reddit (OP)

Interesting. From this thread alone I decided to try to write an alias script on "rm" that should detect if I do rm -rf in the wrong way and stop me and ask if I wish to proceed if so, but ignore and continue the command if not in the wrong way.

I think it works? I don't really want to try though, but maybe I could use this to troubleshoot?

[-]

-illusoryMechanist@reddit

Another problem (though it was more of a learning thing) was when I initially got started and installed *stock debian* on my pc. Which meant it came with none of the proprietary drivers, and all I had was a terminal screen. I had to painstakingly debug what exactly went wrong, and find and install the correct drivers. It took me a few hours, but eventually I got it working.

Tldr; make sure for your main machine you install a distro that makes it easy to get driver support for your hardware (if not have it inbuilt.) If you have Nvidia I'd reccomend Pop!_OS for that reason, as they have an iso that includes the Nvidia drivers baked in (with some tweaks because of some fashion of partnership they have with Nvidia iirc)

[-]

Cyberimperative2024@reddit

I used "rm -rf $DIR/" in a bash script running as root. Unfortunately $DIR was empty, so I really did a "rm -rf /". The stupid thing is that I knew the trailing / wasn't necessary, but I somehow felt it looked better that way.

Nowadays, the GNU version of rm will warn you when trying to self-destruct, but back in the days it just did what you told it to without complaining.

[-]

dkarlovi@reddit

Two come to mind:

I had just started using Linux, doing dual boot, Windows partition mounted RW. Messed something up, said "Well, best reinstall" and did rm -rf /, it didn't have preserve root yet
In college I had an email account on the campus machine, it allowed to set some . forward file where I'd set it to an email my mobile provider gave me, it would send the first 160 chars of an email to my mobile as SMS, very useful before smartphones. The issue is, the inbox was 5MB, so started rejecting the mail back to my school inbox which forwarded it back where it got rejected... This loop crashed the campus machine, I got my account suspended. The sysop was MAAAAAD.

[-]

HelloWorld_502@reddit

I was trying to get sound to work on a laptop for a very vanilla installation of Debian without a desktop environment. I had installed alsa-utils and was using alsamixer to try to get things working. I started unmuting different sources and cranking up volume...really I was just throwing darts at the problem without much of a plan beyond, "For sure one of these settings will start making sound work." At one point I did hear a quick *POP* when I unmuted one of the things...but still no sound.

After a little while I noticed the palm rest was warm...then I started to smell that iconic burnt electronics smell...then the palm rest got very very hot. The plastic around little holes in the bottom of clamshell near where the speakers are located started to melt. I immediately powered down the device and took it apart to find the speakers had FRIED.

I guess I figured out a way to send straight DC current to those speakers with one of the alsamixer settings I tweaked and they couldn't dissipate the heat and fried. Luckily I was able to install a new set of speakers and the laptop still worked. I was surprised the mobo wasn't damaged in the process of frying the speakers. I acted fairly quickly, but usually that magic black smoke is too fast to combat...perhaps I just got lucky.

The solution to getting sound working was to also install pipewire and I didn't have to tweak anything in alsamixer.

It did get me wondering if a bad actor could leverage alsamixer to fry a system...I didn't even need root or sudo access to make this happen. Just needed to be a member of the audio group.

[-]

lotanis@reddit

Accidentally created a directory called '~' through a script I was writing. Absent mindedly did "rm -rf ~" to delete, before realising what I'd done a second later. Too late though, had blown away most of the config.

Had to wipe it and reinstall to get it back in a good state.

[-]

BarisBlack@reddit

Misidentified the drive I had to wipe, using dd and overwriting with urandom.

My system failed and didn't come back with a power cycle. Working from a Live USB, I learned my error.

My other was cloning a drive and incorrectly identifying the source and target drives. I quickly learned of Clonezilla afterwards and praised that it also identified drives by serial number.

[-]

LivingDecent9828@reddit

I don’t quite remember what exactly happened. All I do remember tho is that I tried a Linux script from ChatGPT which resulted in an complete reinstall of Ubuntu (thanks ChatGPT….)

[-]

YeOldePoop@reddit (OP)

Oh, yeah this one I can actually relate to. I have learned the hard way, not through system failure, but other stuff to never run any AI scripts without going through it with a fine comb first.

[-]

AvidThinkpadEnjoyer@reddit

I'm not a veteran, but it happened to me today. Isn't really a kill story, but basically, I wanted to try KDE plasma on my Linux Mint install. KdE was installed smoothly, and everything worked, KDE was amazing, but I wanted to uninstall it because I decided I'd stick with cinnamon until I migrated to Cinnamon completely. So with that in mind, I uninstall, and it goes smoothly. I rebooted the system, and viola, I'm greeted with a CLI login. Somehow, that uninstall managed to uninstall lighDM as well. So I figure not a problem. I try to install it, but it won't work. Just flat-out refuses to install. After hours of trouble shooting, I resort to SDDM, and it works. I head into the terminal and try to install it while logged in, and it shows it's installed ? I'm like the hell, and I uninstall it, and it managed to uninstall cinnamon itself. The whole DE nuked itself. Cinnamon Goss into fall back mode and then crashes, I log in via CLI and install KDE because cinnamon refuses to install again, load in the system and there's an update for the kernel and x org update. They break my manually installed Nvidia driver because the system doesn't load. At that point, I just got fed up, took my installation media, and reinstalled the Dang thing.

Might be because I can't troubleshoot much, but definitely not installing KDE again haha

[-]

YeOldePoop@reddit (OP)

Wow thanks for all the responses, I really enjoy reading your stories. I am a bit shocked how many of them is rm -f basically though, I thought that was more of a meme but it seems to be such a big reason behind a lot of system failures haha. I am afraid to even check, but does it at least warn you now if you accidently do it these days?

[-]

james2432@reddit

Arch
Didn't update in over 8months
/usr/bin /bin fusion

Tried to update, circular dependency as well as arch-keyring being outdated.

Booted liveusb, chrooted. Updated

[-]

Either_Speed_9828@reddit

A mounted drive was owned by root. Sudo chown -R user:user / instead of ./ …Took longer than I expected also saw errors(maybe coz I added aliased chown, chmod cmds with —preserve-root. I stopped it but etc files were already affected including the sudoers file(no more sudo cmds).

I was gonna copy affected files but decided to reinstall.

[-]

ingred46@reddit

I once accidentally deleted an important file, when I turned off my computer I HAD DELETED THE INFO FOR THE MONITOR and nothing would function.

I had to hard reset my computer with a bootable usb, I love linux but sometimes I dont think I should be able to access everything in the computer so easily. With great power comes great responsibility and Im not a very responsible man.

[-]

orangeowlelf@reddit

When you’re in the trenches long enough, it’s just a blurry waterfall of broken drivers, kernel panics and missing shared objects. I think I’ve broken my system in at least 10 unique ways, this is not counting all of the boring regular ways I’ve borked my machines. 9/10 times, I reinstall the OS and mount my home volume. From there, I have a few scripts that get me back to where I was.

[-]

__konrad@reddit

BIOS in my 386 computer had option to format drive. This how you learn what formatting is. I fixed it using MS-DOS fdisk. Also years later I destroyed my partitions using Linux fdisk.

[-]

DrPiwi@reddit

On our Xen server, because the storage was running out of space we went looking for old stuff that could be removed. The sysadmin before-before me, they went through a few - aparently had made a snapshot of the file server VM. And left it.
My colleague removed the snapshot as it was probaly done before an upgrade. to be safe. Because the storage was tight the merge didn't go well. -> the VM crashed and never resurfaced

Lucky for us we could set up a new VM and attach the old storage volumes that served the files.

Same colleague also shut down the airco in the serverroom while he was doing something there and forgot to start it again when finished, on a Friday evening. I can tell that that was not a fun weekend for me.

[-]

suid@reddit

This story has been making the rounds since the 1980's: https://www.ee.ryerson.ca/%7Eelf/hack/recovery.html

[-]

th3t4nen@reddit

Reloading iptables on remote, physical systems, flushing all sessions including ssh..

I still to this day hate iptables.

dd to wrong device, never system though.

[-]

ArrayBolt3@reddit

I had the "brilliant" (uh oh) idea to do full installations of Kubuntu to multiple USB drives in quick succession by booting from a Kubuntu live USB, inserting the USB I wanted to install to, running the installation program, then removing the freshly installed USB and plugging in the next one to do the whole process again without rebooting. I was too inexperienced to realize that there's a faster way (install once, clone to many), but whatever. In went the live USB. In went the target USB drive. The installer ran. It succeeded. I was happy... and then I removed the freshly installed USB from the system.

Turns out that Kubuntu 20.04 actually enables and uses the swap partition that it puts on the target installation drive, and does not swapoff it after the installation is complete. I did not know this, so I just ripped the drive right out of the computer, meaning that the still-in-use swap partition just vanished from the computer's standpoint, taking with it all of the memory contents that had been swapped out.

Things went south fast. Pretty much everything started crashing, including the desktop and my X server.

The other really fun one was when I was experimenting with BTRFS subvolumes, and unintentionally deleted the subvolume that was mounted as my root filesystem. The result was about the same as if rm -rf / worked instantaneousl all at once. Desktop crashed, commands ceased to exist, etc. Thankfully I had just snapshotted the root subvolume so I was able to boot from an external drive and fix things.

[-]

LostVikingSpiderWire@reddit

3 decades in, I would have to say when I was dual booting back in the days, then WinBlows would frequently destroy the boot loader. I knew the first 512 bites with all the commands by heart 😜

[-]

CyclopsRock@reddit

Not mine, but whilst making Toy Story 2 someone at Pixar mistakenly recursively deleted their entire file server. IT noticed before everything had gone and they pulled the plug on the server.

They had old backups, and had to piece the rest together from random local files from workstations and one staff member who was working from home during her maternity leave who had an ISDN connection syncing stuff down. They still didn't get it all back.

They made a lot of changes in the wake of it, but they decided not to actually find out who did it, wanting to avoid an ultimately pointless witch-hunt.

[-]

kolorcuk@reddit

Rm -rf /

Chmod -R -x /

The . key is the worst one to be stuck on keyboard. In the first case i reinstalled the system, it was oldish anyway. In the second, i did package sanity check and reinstalled packages conflicts with pacman-static.

I fornatted wrong partition number once. Ended up restoring data and reformating over multiple days with the help of external drive.

I also once typed mkfs instead of fsck by muscle memory once. Mkfs should really throw a big fat warning before doing anything. Data were not important.

Several filesystem errors. Booting from live cd, fsck -fy, repartitioning. Usually boot partition errors, so reinstalling grub, grub-mkinstall, mkinitcpio.

I would say many times the cause is a typo, which also prompted this stackoverflow question https://unix.stackexchange.com/questions/739478/how-can-i-protect-dev-sdx-against-accidental-formatting

The important part is lessons. Backups, rule 123. Local snapshots with btrfs or zfs. Remote automatic backup to external location.

[-]

FL09_@reddit

Didn't destroy my system, rather nuked \~
I was browing a telegram channel and I saw a post of a random "aplay" command so I decided to see what happens if I run cuz "its just a aplay command, what could go wrong", then when I go to open firefox woke up to no account and then I knew the damage.

[-]

Beautiful_Crab6670@reddit

Eh, just the usual "thought this was a good thing, ended up as a bad thing" situation.

how did you eventually ... fix it?

By reinstalling the distro (again) and a single git pull.

[-]

whaleboobs@reddit

I was missing some core binary or it was broken, cant recall details but it was a source only distribution and the package manager was unable to build packages. I sourced the binary from some guy on IRC to get going.

[-]

eriomys@reddit

despite messing with installations and libraries, system at least managed to boot so I could revert the process. However it broke the moment everything was perfect via a normal update and reboot on Ubuntu. It would not boot even on safe mode. Probably some corrupted data.

[-]

MrScotchyScotch@reddit

I remember early on when, in order to get an app installed, I looked for any package that had the app. I basically just started force-installing random packages. It was a slow process of increasingly breaking the OS until it was a toxic waste dump and I finally had to format the thing and reinstall.

The same sort of thing happens with config file changes, and just changing what packages are installed. The more changes you make, the more it breaks.

Over the course of my career I have come to hate any kind of software update, and I don't install anything that isn't packaged with my distro unless I can 'rm' it and start over.

[-]

denverpilot@reddit

Blowing up home systems was common in my early days, and was easier back then... then transitioned to doing it for work and realized the ways pros handle it... backups, backups, backups, and a written plan for changes, along with a risk anaylysis of what's likely to blow up in your face. Have even worked places that required commands be pasted from the planning documents that were peer-reviewed.

If you don't plan and assess, the backups might save you. By the way, commerical backups are tested, or they aren't considered valid backups. Along with thinking about ways those backups won't work in certain hardware failure scenarios... another risk for the planning...

[-]

dicksonleroy@reddit

I had a WD MyCloud that I somehow figured out was running a version of Debian. I decided to sudo apt update sudo apt upgrade it. It seemed to work well, but then it dropped network connection and essentially bricked it.

I ended up pulling the drive out of it and putting it into an Odroid HC1, which I ran for a few years.

[-]

sws54925@reddit

Had a disk going bad in a server setup with RAID 5 and a hardware RAID card. I was assured (multiple times) that the drive was hot swap, should just be a matter of pulling the dying drive and plugging in the new one.

As it turns out, the server did not have hot swap capabilities.

[-]

MemoryNotSignificant@reddit

I caused an inifinite reboot because of a bug in /etc/rc.local script. Bascially, I wanted to add a autoreboot logic in the startup script, but my condition check was wrong and it always rebooted during the startup. There was no way to recover remotely unfortunately.

[-]

J3S5null@reddit

Sudo rm -rf /

Buh-bye

[-]

LinuxRich@reddit

Tidying up old kernels after a few updates. Accidentally tidied up all the kernels. Realised before shutting down, so installed latest kernel/same as was running in memory. Rebooted...

[-]

LadyPerditija@reddit

disabled root account and botched the sudoers file

[-]

niomosy@reddit

Not Linux but Solaris. Security guy ran rm -rf .

He was in / at the time as root.

We booted off the network, restored a bootable OS from tape, and restored the rest from standard backups. Roughly. It's been a while.

[-]

SithLordRising@reddit

Timeshift restore never works. Boot config and encrypted partitions, it's very difficult to restore an image especially if the kernel changed

[-]

pppjurac@reddit

Late 1990s I think

Used some tools to process scanned negatives (dia film actually) with linux.

Partition was full so I tried to expand it.

Of course I failed. Lost whole lot of digitized photos and had to start anew.

Lecture: Always separate system and data and care to have working backups.

[-]

Necessary_Chard_7981@reddit

Altering config files and then forgetting I have done so can lead to unexpected results. I am not sure if I am a Linux veteran.

[-]

Flashy-Dragonfly6785@reddit

I compiled my own glibc and upgraded the system in-place. It broke horribly

[-]

paris_kalavros@reddit

I have three horror stories:

One was an update on gentoo, something like 15 years ago, when they switched from udev to devfs if I remember correctly. I didn’t check the release notes and got an unbootable system.

Another case was when I tried to reset a third party bootloader on a Slackware laptop, something like 20 years ago, and when I edited the first block on my hard drive I accidentally nuked my partition table. Good stuff it was a laptop, so I managed to get an external hard drive and backup everything before reinstalling.

Last one was when I tried to resize a ReiserFS drive and somehow corrupted it. Again, my backup saved the days.

[-]

Vorthas@reddit

I had updated my video card but the kernel version I was on didn't support it, so I had to upgrade my kernel to get it to boot with graphics again. Was a bit painful but not too hard once I figured out the solution from looking online on my phone.

[-]

adrian_vg@reddit

Sound, network and nfs completely borked after an Kubuntu upgrade {this was maybe four or five LTS"s ago. Anyway, hours later, frustration running high, I did the common Windows jig - I reinstalled/rolled back from a backup.

To this day I have no idea what went pear shaped. Bare metal backups rock big time. I guess that was the main takeaway I learned at the time.

[-]

GreatBigBagOfNope@reddit

Not a veteran, but when I built my current PC I started with a Broadcom WiFi card. I was planning to dual boot, so had Windows set up already and had used it for a few days, before installing Fedora. On first boot, couldn't access WiFi, not exactly unheard of so I opened the logs and found the card failed. No worries, I'll download the drivers for an offline install (didn't have access to cables at the time). Tried to boot Windows. Boot loop after grub. Bear in mind, this is a machine and install that had worked fine for several days already. The WiFi card was the only suspect log item, and it had failed so hard in Linux that it corrupted the Windows install on a separate disk.

Fixed by replacing with a proper Intel card and reinstalling

[-]

MrMikeJJ@reddit

I thought i was in a subdirectory of /etc when I did a "rm -r". Turns out I wasn't in a subdirectory.

There was no recovering the system. Needed a re-install.

Also, i love that it did it (after the "are you sure"). It did what I told it to do. Taught me to pay more attention, especially when logged in as root.

[-]

NeverMindToday@reddit

Apart from conscious experiments on disposable systems, for me the only things that destroyed systems were disk failures.

[-]

GregorDeLaMuerte@reddit

A while ago I wanted to uninstall something, I can't remember what it was. I used sudo apt remove something-* (with an asterisk), didn't read all the packages that were due to be removed and ended up without my desktop environment.

[-]

mrvictorywin@reddit

Ran a deduplicator on OS drive. Full sustem freeze at %8.

[-]

bruisedandbroke@reddit

opensuse and weird libcurl version specific requirements for me to compile something I needed (vulkan, or something similar?) long story short, computer did not like the system version of curl being removed. whole thing totally borked, no WiFi, no ethernet. crawled back to Debian forever after that.

[-]

Previous_Monk_2602@reddit

Very early on in my Linux journey I was looking for a solution to an issue and blindly copied and pasted something like “rm -rf /“ (that some total fucking dickhead had posted somewhere as a solution) into my terminal. Needless to say upon rebooting I realised my error. Understand what you’re copying and pasting people!

[-]

kwyxz@reddit

Back in my Slackware days, around early 1997 or so, Linux moved from binary format a.out to ELF and from libc4 to libc5. I tried to carefully follow the instructions to upgrade, and don't remember exactly what I did wrong, but it broke so hard, no matter what I tried to do, I could not recover the system. Every command was failing, and every attempt to boot on a rescue floppy to restore the previous files led to a complete disaster.

That's when I switched to Debian so better-knowing package maintainers would handle those transitions for me. Has been mostly a smooth ride since.

[-]

EquivalentArachnid19@reddit

So unfortunately I’ve done this twice.

There’s an app for dealing with a Flipper Zero call qflipper and the debian/ubuntu version has a malicious dependency list. The most recent time I needed to upload a better phishing page to the fake wifi network phishing thing, and just like last time it completely uninstalled my windowing system and window manager.

It’s annoying but not too hard to fix. Some distros have a desktop bundle, all distros have instructions on how to (re)install your window system if you installed a cli only machine. This latest time I took the opportunity to readd xorg and switch to XFCE, as a direct result I now have my blessed menus back. File / Edit / etc. I’ve already stuck the Flipper Zero into a bucket of halloween give aways, with strict instructions not to return it to me.

[-]

DFS_0019287@reddit

I've been using Linux since 1996. Only once have I accidentally borked my system. This was on a VPS that for some reason uses syslinux/extlinux to boot rather than GRUB. I did an upgrade and the system wouldn't boot because the new Linux kernel image was too large for the KERNEL directive. I had to change it to LINUX instead.

Luckily, the VPS provider provided KVM-over-IP facilities and I was able to boot into a rescue ISO image and edit the extlinux.conf file.

This is very much an edge-case that probably nobody else will ever run into.

[-]

brimston3-@reddit

Similar VPS story. Upgraded a system running on a 2.6.x (.18 or .32, I can't remember) openvz/virtuozzo kernel that masqueraded as a much newer kernel to run modern linux userspaces. The upgraded system had a new version of glibc which was not patched to run on 2.6.x and tried to access newer kernel interfaces. It failed very early at boot, being unable to run any programs.

Nobody with any sense runs these kinds of kernels anymore since lxc and other container technologies have matured using mainline kernel features.

[-]

nooone2021@reddit

I have dual boot computer where I shrunk Windows partition to make space for Ubuntu. When Windows decided to upgrade from 8 to 10 it shrunk one of its partitions and inserted another partition which changed partition numbering of all following partitions. During Windows upgrade computer did not boot any more because grub did not find partitions where they were supposed to be. All I got was "grub rescue>" prompt. Somehow I managed to change grub settings and make it boot again. Then I had to change Ubuntu setup because partitions were numbered differently. Partitions (including swap) were encrypted, but I managed to make it work without reinstallation.

I am still using that computer. In fact I am writing that post on it, and it still has the same Windows (10) and Ubuntu (now 24.04 LTS) installation. Both operating systems are pretty regularly updated. It is 11 years old laptop and still works like a charm. It is not any more my primary workstation, but for browsing and less demanding tasks it is just fine.

Windows won't upgrade to 11, because computer is too old, but who cares. I rarely boot Windows. I keep it because it was installed OEM when I got new computer 11 years ago.

[-]

seventhbrokage@reddit

Not my own story, but a friend of mine somehow wiped out everything in /boot specifically without touching anything else, so when I tried to help fix it I spent a very long time spinning my wheels bwfore eventually finding the issue because grub was fine, root filesystem was there, but it just refused to boot in. We ended up just reinstalling because he wanted to try a different distro anyway, but I'm still baffled at how he managed to pull that one off on an atomic distro.

My own personal best isn't quite as good, but I once made it impossible to use a terminal emulator on an old arch install. I tried to switch to the fish shell and clearly did something wrong in the process, because it flat refused to read any commands I typed in. Then desktop elements started failing. This was still fairly early in my linux experience, so I also ended up wiping that install and starting fresh. That's usually not the answer, though. You can always drop to tty and work on stuff from there if needed, but I didn't know that at the time.

[-]

SerpienteLunar7@reddit

Once, back in time at my first time on an arch based distro (Endeavour OS) I was using plasma and wanted a widget which was only available through the -git packages of plasma. When I installed it I accepted all the dependencies changes (yeah, rookie mistake) and ended up even with problems in grub (those glory Arch days when a dependency chain could remove all of your system).

Wasn't funny NGL, maybe my love for distros like Guix and NixOS comes from there

[-]

starlevel01@reddit

tried excising infinality a few years after it became unsupported. cue losing all of my fonts everywhere.