What’s the hardest Linux interview question y’all ever got hit with?
Posted by yqsx@reddit | linuxadmin | View on Reddit | 462 comments
Not always the complex ones—sometimes it’s something basic but your brain just freezes.
Drop the ones that had you in void kind of —even if they ended up teaching you something cool.
eodchop@reddit
List 4 reasons while a writable file system errors out when attempting to write to it. You check du and it shows there is plenty of space.
Fazaman@reddit
I've been a sysadmin for 25 years and never had this issue. The comments below make it seem like this happens often, and maybe I've been lucky, but inode exhaustion would not be the first thing I would think of.
segagamer@reddit
Yeah after reading this I immediately went to
df -i
our servers, yet the highest consumption we have is 5%, even though we work with a lot of tiny files (development). This must be a more common issue in much larger enviornments than mine.Fazaman@reddit
Judging by what I've read, it was more of an issue with older filesystems, though I didn't have issues back then, either. My highest seems to be 6%, and that's on my (file based) backup server, which has copies of most everything from every server. Everything else seems to be 2% max. Though all of these systems have a bunch of filesystems on them, so things are broken up a lot. I'm sure that helps to keep the numbers down.
My workstation has one filesystem with 16% usage, but that filesystem is also 94% full, so...
segagamer@reddit
Our company works with fonts. Each projects source code is a .ufo folder which houses a human readable format of each character for each weight for each font family (think of them like SVG files). Often with multiple scripts (languages), and up to 16 weights (bold, light). Doubled if italics. Plus versioning.
As you can imagine, it's thousands of tiny files per font. Decades old company... Yet only 5%. Honestly expected it to be a lot of more lol.
Fazaman@reddit
Apparently, they work the same (allocated at filesystem creation, and so, if the default ratio is used, the number is the same, but ext4 is more efficient. I asked an AI, cause I didn't know offhand:
This could be wildly incorrect, or there might be subtleties that I don't know... it is an AI after all, but all I know is that I never ran into inode limits.
mgedmin@reddit
That's why it's so surprising when it happens.
(I had it happen to me once, because I made a mistake and ran
mkfs.ext2
with the option to optimize for large files, which reduces the number of available inodes.)GreenHairyMartian@reddit
Depends on the environment.
Ive had it happen a few times in the 10years I've been involved in production Linux systems.
Usually been on NFS mounts, or on stupid web servers that write a bunch of tiny files, but never clean themselves up.
Not really an issue on modern architectures, and sane designs.
segagamer@reddit
Heh, I learnt something today. Thanks for this.
WiseassWolfOfYoitsu@reddit
Off hand my answers would be inode, basic permissions, ACL permissions, and then selinux permissions.
RomanRiesen@reddit
btrfs metadata exhausted and all btrfs blocks allocated :/
thatsbutters@reddit
Good list. I'll add mounted read-only
H3rbert_K0rnfeld@reddit
I'll add nfs mount gone stale
Derp_turnipton@reddit
Read the question more carefully
thatsbutters@reddit
without specifying the filesystem type the answer is valid. For instance, du will write over the ext3 journal or NTFS last access dates when mounted ro.
cullor@reddit
I'll add reserve space. 50 gigs may seem like plenty of space, but on a 1TB LV, that's still 5% that nobody but root can use.
souptable@reddit
Reserve doesn't show in df, it would show as having no space free until you release it.
Superb_Raccoon@reddit
Drive is dead. Your listing is coming from cache.
spaetzelspiff@reddit
The drive is fine, but it just got yanked by remote hands who was asked to pull not-the-good-array-member-on-the-nfs-server-mounted-o-soft
OikuraZ95@reddit
That's possible? I clearly don't understand I does enough
August_XXVIII@reddit
Inodes were the first thing I thought of after reading the first sentence.
H3rbert_K0rnfeld@reddit
An open file handle against a log file that filled up the filesystem was my guess.
manapause@reddit
Me too, because it has bit me in the butt on production.
praminata@reddit
How? I just commented that in 25 years I've only encountered that in interviews
thespud_332@reddit
You've never worked with an overcommitted shared web hosts cPanel/WHM server, have you?
praminata@reddit
Haha, no. Different user base entirely.
0bAtomHeart@reddit
Hi it just happened to me (yesterday)
Non-networked embedded machine. Middleware loves to create 0 byte log files. Heavy use for ~3 years and you get inode exhaustion.
FS was set up with large file performance in mind so only had ~120k inodes
manapause@reddit
Running 300+ LEMP asset/erp portal applications for customers on a single instance, storing user uploads locally. Customers in big organizations had uploaded 30K+ files, as well as some issues with application-specific log rotations. A great lesson in accommodating power users. Also AWS Linux inodes (ulimit) settings were pretty.conservative back then.
“df -i” alerting is easy to set up, don’t let it happen to you! (Also use S3 for uploads if you can.)
Derp_turnipton@reddit
I found 4 without including this.
minektur@reddit
SEL has subtly wasted hours of my life - it defies my muscle memory of stuff to quickly check and it usually bites me on some service that is running but can't update logs or some other random thing that makes me think "config issue" for the application.
choss-board@reddit
FWIW I also thought of inode exhaustion but coming at this with fifteen years experience—who cares? If it’s not already there, that’s the kind of problem quickly becoming obsolete by search and AI. (Yes, yes—I get all the caveats there. Point stands that they’re good at that sort of “point me in the right direction” question.)
praminata@reddit
That's a classic one from back in the day. Btrfs threw in similar scenarios like that, but they were actually useful to know about in production back in the early days of docker. I have never actually encountered the inode issue in 25 years.
poolpog@reddit
i've encountered this question
I've asked this question
juniors don't often know what inodes are but they should
hikertechie@reddit
That was my first thought as the problem but only because ive had to deal with it irl on prod servers.
bdashrad@reddit
Trivia doesn't belong in interview questions unless you can use man pages, docs, or search the web.
When I interview people for a role where someone might ask something like this, I'm looking for problem solving ability, research skills, etc., not just someone who already knows this because they've seen it before.
My goal is to find someone who can fix problems you haven't seen or haven't thought of, so seeing them pick apart an error message on an unfamiliar tool, then searching docs and other sources to find an answer, tells me a lot more of what I need to know, even if they don't get the correct answer.
Invisible_Stalker@reddit
It’s always a question when I interview people. What I like to segue into is hard and soft links and see if they can deduce that inodes are 2 with soft link 1 with hard links from first principles, or know this already. I tick a big checkmark.
autogyrophilia@reddit
I would simply use a monitoring solution to catch that...
03263@reddit
>monitoring software crashes because it can't write a cache file
autogyrophilia@reddit
Yes. That's what the monitoring software is for.
eodchop@reddit
It’s in a a dev environment and due to Datadog costs we do not monitor disk activity in non production environments
Intergalactic_Ass@reddit
Seems like a Datadog problem then. People have been monitoring inode usage for decades without Datadog. If the costs are so bad that you skip monitoring things it's time for a new solution.
trapped_outta_town2@reddit
Not so easy in a large-ish multi national org where the newly hired IT Director is kind of useless. https://old.reddit.com/r/linuxadmin/comments/1kye5bh/whats_the_hardest_linux_interview_question_yall/mux613u/
Intergalactic_Ass@reddit
Stand up something open source. Costs you nothing and your IT Director can take credit for it. CheckMK Raw is an option.
autogyrophilia@reddit
I mean it isn't a headache I ever dealt with ever since I made XFS and ZFS the standard filesystems to use at my org. With a bit of BTRFS if the usecase justifies it.
I find it hard to justify not using XFS as a default other than EXT4 being generally good enough. But I digress.
Datadog is good at metrics and traces. But it doesn't do what tools like Zabbix or Prometheus do.
The basic template includes preemptive alerts before you run out of inodes : https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/os/linux?at=release/7.2
Nicolay77@reddit
That one got me. Been using Ubuntu for 15 years, never experienced it.
03263@reddit
That's the first one I thought of.
Also permissions, bad disk, corrupt filesystem? Cosmic rays...
gzk@reddit
In addition to what everyone else has said, extended attributes (chattr / lsattr)
vainstar23@reddit
The file is write locked
The file is protected
There are not enough numa cores (not really no writes but slow writes)
There are not enough inodes
There is not enough space
There is a I/o failure
Just off the top of my head. Damn I could write a whole chapter on this question.
trowawayatwork@reddit
that's like a standard production engineer question at meta. they get so deep into linux
spaetzelspiff@reddit
Asking about filesystem metadata on an interview at Meta is kind of...
theloneabalone@reddit
Ooh, sometimes it’s deleted files that haven’t been so deleted and are hanging onto those inodes for funsies. lsof to the rescue!
piratnisse@reddit
Not an uncommon issue to pop up on cloud instances with small root drives, so I kinda see where they’re coming from.
tindalos@reddit
If you’ve worked with Linux in production this should be one of the easiest questions to answer lol
courage_the_dog@reddit
This seems pretty straightforward and easy for me to answer. I've ran into inode issues a lot of time in a role i had, can't remember what the actual cause was but the fix was to go in and delete a large number of files that get created and weren't needed/removed. Disk can fill up with file size or amount of files, which is where inodes come in. Though i dont know what other reasons there are.
BathroomEyes@reddit
That’s a pretty good one
z-null@reddit
I've ran into inode exhausion several times. it was always because devs wrote session files in /tmp and never cleaned them up.
wcpreston@reddit
Got a text file called foo First line is “#!/bin/echo” Second line is “bar”
It’s executable and you can run it. What will show up on the screen when you run it?
I know and can explain the answer, but I do think it’s actually a pretty good question for an interview. I’m curious to see what others think.
Emergency-Scene3044@reddit
What's the difference between a hard link and a soft link?"—brain went poof in the moment 😂 What was yours?
punkwalrus@reddit
I got one where it was obvious that they got the interview questions from a book and didn't know the context because they asked in weird ways.
"Suppose you have a web server, and it has no directory. What would you do?"
"What do you mean has no directory? Like the web server doesn't list files and you want it to on a web page?"
"No, in the cloud."
"And by the cloud you mean what? What is your end goal?"
"Don't over think it."
"I'd set the apache or nginx to list files in that directory."
"Passive or active?"
"Those are ftp terms, not web terms."
"But they are part of the kernel. The kernel's ftp."
"... I am not sure the context of this question. What are you trying to do?"
"Go to ftp.kernel.org. See?"
And so on. It was like we were speaking two different languages.
yqsx@reddit (OP)
I’d be like: “Set up a webserver… with no directory… in the cloud? Yeah ok, you’re definitely from HR.
jake_morrison@reddit
How would you connect to the database using CSS?
NoMansSkyWasAlright@reddit
I once saw a job add that listed "server side CSS" as a preferred skill. That one was a favorite of mine.
bapfelbaum@reddit
While it's obviously not the abbreviation they were looking for, if you just construct one from the word itself it's not that bad of an attempt.
FuckImGettingOld@reddit
I once came across a program that ran a for loop to generate html header css, just +='ing the output string through this huge if/else block. I needed somebody to hold me that night.
ttl_yohan@reddit
Hey, that's called PHP.
JollyJoker3@reddit
I assume it's a common requirement for some, but having to store a visually exact copy of a report that was made with interactive, clickable and filterable elements was a bit of a wtf for me.
InadequateUsername@reddit
Ihttps://images.app.goo.gl/dtEbqJH2QBpAQ2nK9
fezmid@reddit
I'd use mauve. It has the most RAM.
shaliozero@reddit
Easy
database { host: localhost; port: 3306; database: css; user: web; password: 12345; }
Wonderful_Device312@reddit
That's not the worst question actually. It's kind of a unique combination of creative problem solving and demonstrating an actual understanding of how various things work. It's more relevant to the job than any leetcode question at least.
I mostly do backend stuff but some ideas I came up with:
IE supported CSS expressions - aka JavaScript in CSS. Do a fetch to an endpoint that connects to the database or some other creative thing with JavaScript.
CSS's url function could be used to send get requests to various API's which effectively connects you to a database.
You could do some crazy CSS shenanigans and build a really shitty database in CSS and connect to that. I'm not a css wizard but I know it supports basic toggle like logic, which means you can implement logic gates, which means you can simulate any program... Kinda. Sorta. I think theres counters and other stuff too so... It might be even easier.
CSS custom properties maybe? But that's similar to css expressions.
On the server side, there are css processor libraries that have had vulnerabilities which could allow arbitrary code execution from specially prepared css.
jackauxley@reddit
Hah, I see, a trick question, it's not CSS, it's CS2. I don't play counter strike source, that's deprecated.
Sheratan@reddit
"The defence is wrong!"
xplosm@reddit
Overruled
TradePaladin99@reddit
Hahaha. This is the best possible response to that question.
lemon_tea@reddit
With a melodious vapor connection string
meadowkat@reddit
Im in HR and used to be a Linux sys admin. I assure you the asshatery goes both ways lol. The crazy things people respond to basic questions when they think you are just HR and wont notice, I am writing a book.
MyClevrUsername@reddit
“Don’t overthink it.”
robotfromfuture@reddit
Certainly the interviewer didn’t overthink it.
ogn3rd@reddit
Had the opposite happen to me once. A lady said she was from HR then started asking all these storage questions. Called her out jokingly before answering and all I got was a smirk.
Dabger1@reddit
This sounds like something from an A+ exam
CodeFarmer@reddit
I've ended an interview early when that happened (and explained why).
I felt bad for the HR intern on the other end being put in that position, but they're wasting everybody's time.
GolemancerVekk@reddit
"See what, browsers stopped supporting FTP years ago."
"No no, it works fine in IE6."
k-phi@reddit
ftp.kernel.org. was shut down in 2017
Firefox stopped supporting ftp in 2021
Inquisitive_idiot@reddit
It’s in their IE6 offline files 😭
GolemancerVekk@reddit
Ah, a double-entendre.
H3rbert_K0rnfeld@reddit
Who's on first?
Fratm@reddit
This is bizarre, If all the questions were like this, I would probably shake there hands and leave.
I have not interviewed for a job in 20+ years, but I do interview folks when we hire new sysadmins, and I am in charge of creating the Linux questions, and I try to ask real world questions and mostly I look for their ability to solve them, so if I asked the above question, I would not tell them they are over thinking it, and I would appreciate the questions, you really get to know their skill set with the back and fourth.
RootedRebel@reddit
Are u the youtber guy?
punkwalrus@reddit
I felt like that guy in the Monty Python Mountaineering sketch:
"Well I'm afraid I shan't be coming on your expedition sir, as I've absolutely no confidence in anyone involved in it."
thebigt42@reddit
That's a great sketch
UbieOne@reddit
Reminds me of that scene about "we can't accept your surrender" in A Bridge Too Far. And that British officer was even carrying an umbrella! That was funny as hell. 🤭
InfaSyn@reddit
How did you even handle this from a situation standpoint? Id probably fumble so hard lol
I take it you didnt get that position?
punkwalrus@reddit
I remember leaving mad that this interview had been a waste of time. They didn't call me back, and part of me thinks that it could have been that they didn't know how to operate a phone, or got stuck behind a pillar and starved to death.
RovBotGuy@reddit
"Stuck behind a pillar and starved to death"
Slayed me, I thank you sir.
ryanknapper@reddit
Sounds like someone needs to grep the logs to netcat until there is recombinant capitulation. If VBScript is available, I suppose you could make a GUI and start measuring IGMP responses.
Fun-Dragonfly-4166@reddit
It seems obvious to me.
Since it is stupid, I won't deduct any points from you.
My web server without any directory computes specified digits of pi and exposes it as a REST endpoint.
GET /pi/{from}/{to}
returns the specified digits of pi or returns appropriate errors if to is not a number, from is not a number, from < 0, to < 0, from > SOME ARBITRARY CONSTANT, to > SOME ARBITRARY CONSTANT, to < from, or the range to-from is too big
lazyant@reddit
I’d love to watch a sketch based on something like this, it’s hilarious
03263@reddit
Kind of reminds me of this
https://www.youtube.com/watch?v=BKorP55Aqvg
shrizza@reddit
Two YouTube channels you might enjoy: * Programmers are also human * KRAZAM
vainstar23@reddit
Lol I would have trolled
I drove to this interview, my car is blue, SOLVE THE QUESTION!
catonic@reddit
Should have asked him to hold up both hands an equal distance above the desk to see if the person is having a stroke.
meagainpansy@reddit
Yet they won't admit they're hiring for this position because they don't have anyone who has any idea how to do it.
MairusuPawa@reddit
I lived through that with an interview for Amazon of all things. HR asks me if I was proficient in English. We switched to English for the interview. Turned out HR didn't understand a single word of English - the conversation was atrocious and they couldn't even provide me with a full sentence, refused to acknowledge the situation, eventually threw my resume in the trash.
housepanther2000@reddit
Geesh! Clearly the interviewer has no concept of the way the internet works, let alone Linux.
Cherveny2@reddit
its like a dilbert pointy haired boss interview
punkwalrus@reddit
Mauve has the most RAM...
brentownsu@reddit
Ive been in that situation before and politely finished the interview - and then told the recruiter that as much as they were interviewing me, I was also interviewing the company to determine if I wanted to work for them and this probably wouldn’t work out. Don’t try to get anyone fired or anything, but they need that feedback to put their best foot forward for future candidates.
capt_bmiller_12pct@reddit
lol!
hamnstar@reddit
This is making me angry just reading this lol
ruyrybeyro@reddit
Your answer reminds me of an interview I once had with a very rude IT manager in Gibraltar. There was also an Indian guy asking the technical questions, but he didn’t seem to know much more than the manager.
There was a bit of a language barrier with both of them, too
inbetween-genders@reddit
Some of the interviewers remind me of that scene where it’s a British 3 vs a German 3. The questions are all wacky then you realize they aren’t from the area.
cdn-sysadmin@reddit
An enterprising young junior sysadmin has run the the following command on a production system:
Without rebooting into a LiveCD how would you fix this?
-rwsr-xr-x@reddit
I've used, and had this question on so many interviews, and so many people have Google'd solutions, I tend to exclude all the obvious ones that they haven't directly tried themselves.
I have one I used on an interview years ago, and the interviewer said "Your answer won't work.", because his own Google'd search result, didn't include my solution, so he blindly excluded it.
Until I said: It works. Try it, or I can show you right now.
He did. He realized it works. I told him not to just trust Google, but to always "test your assumptions".
I got the job.
The answer?
cp /usr/bin/ls /var/tmp/chmod
cat /bin/chmod > /var/tmp/chmod
/var/tmp/chmod --version
chmod (GNU coreutils) 8.32
Copyright (C) 2020 Free Software Foundation, Inc.
486321581@reddit
Beautiful
thesaddestpanda@reddit
Can you please explain how this works?
z-null@reddit
When you overwrite a file, it keeps it's permissions. So chmod without +x goes into something that does have +x will result in chmod with +x because that file already has it. It's metadata preservation, or if you want, when you copy file a into file b, permissions of b aren't changed to that of a.
shrizza@reddit
Copy a file with the desired executable bits, then copy the broken chmod's binary contents into that file. You should be able to rescue
/bin/chmod
with/var/tmp/chmod
now.m15f1t@reddit
Second action is not a copy but overwrite.. This is crucial because that's why the rights of the file stay the same.
shrizza@reddit
I would think the wording of copying the contents into the file would suggest as such.
marsd@reddit
Looks like it's copying
ls
which I assume is still working and has executable permission into tmp chmod then overwriting the contents with the actual chmod.HalfPastMoon@reddit
Holy cow, that's interesting!
NorgesTaff@reddit
Nice.
vainstar23@reddit
Lol I did not think of this but it makes sense
InvincibearREAL@reddit
oh thats good
Embarrassed_Fan7405@reddit
You guys are monsters
rfc3849@reddit
Several come to mind.
Reinstall the package containing chmod
perl -e 'chmod(0755, "/bin/chmod");'
python -c 'import os;os.chmod("/bin/chmod",0755)'
cp /bin/chown /bin/chmod.tmp cp /bin/chmod /bin/chmod.tmp
cp /bin/chmod /bin/chmod.tmp install -m 755 /bin/chmod.tmp /bin/chmod
BlackPignouf@reddit
Couldn't Perl or Python delegate chmod to /bin/chmod?
I don't get the third one. Shouldn't the second cp be a cat?
Dolapevich@reddit
I thought the reinstall package option, but I am not sure if
chmod
is a dependency for that. Most likely it will useinstall
so it should work.mgedmin@reddit
I'm pretty sure apt/dpkg/rpm call the libc fchmod() APIs directly instead of shelling out to an external /usr/bin/chmod or /usr/bin/install for each file.
Postinst scripts might break, if they invoke chmod. There are a number of these on my system:
but coreutils itself doesn't have any of those.
meditonsin@reddit
Another option would be to run the binary via the dynamic linker. So e.g.
/lib64/ld-linux-x86-64.so.2 /bin/chmod +x /bin/chmod
mgedmin@reddit
Wasn't the dynamic loader fixed at some point to check for executable permissions before running the thing you asked it to run? Because it was a way of sidestepping system policy like
-o noexec
mount options and such.checks
Ah, no, it still works, for chmod -x at least. Didn't try mount -o noexec.
cdn-sysadmin@reddit
Nice, yeah, I didn't even think about using perl/python.
The three ways I know:
1) Sacrifice (or make a copy of) an executable and copy chmod over it 2) install -m 755 (as you mentioned) 3) /lib/ld-linux-x86-64.so.2 /usr/bin/chmod +x /usr/bin/chmod
poolpog@reddit
i've encountered this question and i've asked this question
i like it because there are quite a few valid solutions
vainstar23@reddit
sudo chmod +x /bin/chmod
Nicolay77@reddit
Would mc call chmod or change the bit by itself.
Makes me want to try it.
Catenane@reddit
Rpms make it easy. rpm --restore coreutils
I like all the other options here too though. Not sure if dpkg has a similar restore facility or not.
cdn-sysadmin@reddit
It's funny how sometimes your brain looks for the hard answer instead of the simplest and most obvious - just reinstall the stupid package. As for dpkg:
Catenane@reddit
Also, obligatory https://xkcd.com/356/
Thanks for nerd sniping me, lol.
Catenane@reddit
So yeah that was my obvious first thought, but I actually tried that in a container—and with a fresh container (and thus empty package cache), you can't update the cache after chmodding chmod, making it impossible to --reinstall. Apt-key calls chown in a few places and pulling repos fails if it can't do so. There may be an option to ignore/override. Now the interesting part is after cleaning the cache, it'll still work if you've initialized. That's because of an (lz4 in my case) compressed archive description file for the repo in /var/lib/apt/lists. That doesn't get deleted with an apt clean, but if you delete it again, you can't --reinstall anymore.
It's such a contrived example, but something you can definitely walk away from with some extra knowledge after playing around a bit, haha. In this case it's almost certainly due to deletion of as much as possible to shrink the base container, but I feel like this contrived problem could make sense in the context of containers anyways.
lordgurke@reddit
I do that one, too.
The most straightforward solution: >!Use busybox's builtin chmod to fix it.!<
Some other solutions I was presented:
cp -p /bin/bash /bin/chmod2 && cp -a /bin/chmod /bin/chmod2
dd if=/dev/zero bs=1M count=1 of=/tmp/fs.bin && mkfs.vfat /tmp/fs.bin && mount -m -o umask=000 /tmp/fs.bin /tmp/vfatfs && cp /bin/chmod /tmp/vfatfs && /tmp/vfatfs/chmod +x /bin/chmod
gdb /bin/chmod --args +x /bin/chmod
and typerun
mcd1992@reddit
Only way I can think of is invoking ld-linux.so with chmod as an arg. Any other trick for it?
ChowSaidWhat@reddit
chmod -x /usr/bin/chmod.
fix it. No internet, no CDs, no python, nothing, use just OS commands.
bqw74@reddit
ouch!
bqw74@reddit
Give me a shell 1-liner that find and prints out the 10 most common words in the bash manpage with a count.
Answer:
punklinux@reddit
The hardest I ever got were weird trivia questions about Linux and UNIX history. Like:
I did not get that job. I got the sense the interviewer just wanted to appear clever and stump everyone.
bigkahuna1986@reddit
Maybe you accidentally time warped into 1993?
punklinux@reddit
It was in 2002-2003, so that was less than 10 years.
erikosterholm@reddit
If I were asked these questions, I'd ask what what the relevance is to the job for which I'm interviewing.
Superb_Raccoon@reddit
"We are still running Slackware on 386s."
nonviolent_blackbelt@reddit
386s with the original UART?
erikosterholm@reddit
See, it's important to know when the job just isn't for you!
Fazaman@reddit
No idea
No idea
No idea
I could name several, as I've used them.
And if I recall correctly, this was originally a paper jam error on a particular type of printer where the paper was prone to jam in a hot part of the printer, so this error could literally mean that this particular model of printer was on fire ... but paper jam.
maryjayjay@reddit
WiseassWolfOfYoitsu@reddit
They started on PDP-7 but PDP-11 is where it really got going. The big technical limitation was disk space. The PDP-7 supported one disk pack with an astounding 1.5MB of storage, which wasn't quite enough. The PDP-11 supported... gasp... TWO 1.5MB disk packs!
This is also why /bin and /usr/bin are separate plus the origin of /home. Originally /bin was binaries and /usr was user storage, but they ran out of space for binaries and so /usr/bin had user executables not needed to boot. Then they made /home for user personal files because /usr became full of binaries.
maryjayjay@reddit
Nice! TIL
catonic@reddit
I've read about that, and he owes me two beers now. One because I had to remember the 3B series, and another to erase the pain of recalling the memory.
Go read about it if you want to have some sort of down-the-rabbit-hole chase of the white rabbit like The Matrix into a world of layered emulation and virtualization. There is a reason why the NSA has people who are damn good at telco stuff and it's in there. Any sane person wouldn't let that stuff within the effective blast radius of nuclear weapons to themselves.
Clock speed options, buffer size, interrupts (8-bit originally, so Interrupts 2-7, and serial data speed (IIRC 38400, 57600, and 115200 were sketchy).
Don't remember other than F0 0F C7 C8. Pentium.
RH, Slackware, Debian, SUSE, I think Yellow Dog as well. I'm sure I'm missing a few.
IIRC you have something going very wrong with interrupts associated with the printer port.
-rwsr-xr-x@reddit
I still have ALL of those versions, running as VMs on my infra (in a suspended state, of course). I also have versions of Windows 1.0, 2.0, 2.2, Microsoft OS/2 (before they sold it to IBM), every version of DOS (including FreeDOS), Windows, BSD, Linux (every distro), and many others.
Lots of versions pre-date RHEL, including the former Red Hat Linux versions.
All of these and more, predate RHEL which began in 2002.
I still have several of the 6-cd boxed sets from the "Linux Developer Connection", which includes full versions of all of these, full ftp archives of funet.fi and sunsite.
Those were fun time!
WiseassWolfOfYoitsu@reddit
I know people who could answer all of those, but they'd be so close to retirement I don't think they'd be job shopping!
I could do 2.5 of them? I know some history but Linux 1.x stuff is dang old...
HeligKo@reddit
It wasn't the hardest, but it cracked me up. "Do the alphabet in linux commands like you were writing a childrens book"
A is for at b is for bzip c is for cat d is for dd e is for export
and so on
doubled112@reddit
f is for fsck this.
StatementOwn4896@reddit
G is for growpart
courage_the_dog@reddit
H is history.
UltraChip@reddit
I is for ip
GolemancerVekk@reddit
"j is.for.jmacs"
"Get out."
mpvanwinkle@reddit
K is for kill … with a 9 🤘
nicky9door@reddit
L is for ls
privacy_by_default@reddit
M for man
bluetac92@reddit
N is for netcat
dect0r@reddit
O is for open
dclaw@reddit
P is for ps
Slight_Student_6913@reddit
Q is for :q!
How to get out of vim without turning your pc off.
Aminacakan@reddit
R is for "rm -fr ./*"
Removes the french language pack.
HaykSD@reddit
S for "sudo rm -fr /"
Quiet-Protection-176@reddit
T is for tree
JollyGreenLittleGuy@reddit
U is for umount
jackauxley@reddit
V is for vim
AreBeingWatched@reddit
W is for wget
doubled112@reddit
Pro tip: the killall command on AIX box is not like the killall command on a Linux box, especially when you are root
vainstar23@reddit
kill 1 👁️👄👁️
minektur@reddit
seriously? not grep? I use grep about 5 orders of magnitude more than growpart
Hour_Ad5398@reddit
I'd start with the GNU copypasta.
dig-it-fool@reddit
I got them all except for i, and took some liberty with a couple like jq, since it's not built in..
I went and looked at my commands that start with i and don't think I've ever ran any of them.
rockandrollalice@reddit
ifconfig is the only one that comes to my mind starting with i
dig-it-fool@reddit
Bah, of course it's one I've typed a thousand times, as well as ip.
I originally looked on my Mac and just overlooked it, now looking on an actual Linux machine I see a lot more I use /used frequently
420GB@reddit
See I was debating whether shell built-ins like echo or if would count and decided against it. But I also definitely had to resort to some tools that definitely aren't preinstalled on most server systems too lol
cullor@reddit
I haven't used ifconfig for ages. "ip -br a" is my go-to unless I'm looking for the MAC.
erikosterholm@reddit
init!
Catenane@reddit
iperf(3) would be a good one here. Or just ip lol.
ip neigh
or a/r/etc.Or like, iotop...I'm sure there are others that aren't too obscure.
catonic@reddit
Jeez. Not bad, most can generally handle that.
Esoteric is: "Which six letters are not valid options to the ls(1) command?"
420GB@reddit
Easy:
420GB@reddit
Easy:
HeligKo@reddit
Now there is a question that eliminates 90% of guys with a ton of experience with no good applicable skill attached to it.
doubled112@reddit
If somebody asks something like that I always hope they’re looking for effort and thought process.
It isn’t a for all, or l for list, or n for numeric, or Z for selinux, or … wait, are we talking GNU ls or maybe something like busybox?
lebanese-beaver@reddit
I got this one too! (from a verizon interview I think) it's one that I'll never forgot, it really eats up the interview time lol
vincentdesmet@reddit
A is for ash B is for bash C is for csh …
vainstar23@reddit
c is for alias cd="rm -rf"
Wise-Reputation-7135@reddit
lmao
wishnana@reddit
r, for “rm -rf”. Once you execute it, it becomes your core memory.
Catenane@reddit
R is for rg because I'm a turbo speedy boi and I'm hitting the | s so fast my brain can't keep up with my fingeys
catonic@reddit
or that one time you accidentally put a space between * and something else.
Fazaman@reddit
Oh god... I suck and pulling shit out of my memory at random like this. I would suck at this task.
Took me a few seconds to come up with 'a is for awk'
HeligKo@reddit
Normally it can be struggle for me, but for some reason it just started rolling for me during that interview. Too bad that doesn't happen on the interviews for jobs you really want.
More_Bid_2789@reddit
Did we have the same interview? So out of left field. -_-
HeligKo@reddit
I can't remember who it was with, but yeah it was out there for a platform engineer/DevOps role.
maryjayjay@reddit
I had someone do that for python. It was tedious. To this day I don't understand what good it is as an interview question.
HeligKo@reddit
He said he usually stopped people at about H, because it showed they were familiar with the CLI, but was entertained by how fast I was moving through them, so I did the whole alphabet. They then told me I would be bored at that job as we got into the interview and we moved on.
punklinux@reddit
This reminds me of something Richard Feynman said about a science textbook, where you had to add the temperature of stars the father observes and then subtract the different of his son observing different stars as a way to "add mathematics to the physics curriculum." Only, why would you add the temperature of stars, and subtract the ones of others? Nobody would do that, and that doesn't tell you anything about how stars are observed.
cknipe@reddit
How many entry points into the kernel can you name?
zack6849@reddit
Out of curiosity, what kind of role was this for that involved this level of knowledge about the kernel?
cknipe@reddit
SRE at Google, probably around 2007 or so. There was also a lot of brain teaser algorithm stuff, and a bunch of code exercises. I did not get an offer.
catonic@reddit
Are we talking about known and documented or vulns and exploits?
cknipe@reddit
So what they mean there - The kernel is a big collection of code with many functions and subsystems. Entry points are places where something invokes some part of that code.
For example there's a portion of it that begins executing as part of the system boot process. The bootloader hands off control to the kernel and the kernel sets itself up and starts managing the system. But the boot process and the subsystems that get initialized there aren't the only place kernel code runs.
Sometimes a userland process needs to do something that it can't do directly, like read and write to files, or traffic on a network. There's a mechanism called "system calls" that allow code to ask the kernel to do something of that nature. That's another entry point - another way that kernel code can be invoked.
There also interrupt handlers. Generally when various pieces of hardware need attention from the kernel they can "raise an interrupt" to ask the system to stop what it's doing and deal with their situation. These handlers are another entry point into the kernel.
There are more, but that's the general idea.
dig-it-fool@reddit
I've been using Linux for 20 years and I can name 0.
After typing that, I did the math and it's been nearly 30 years, damn I am getting old.
cknipe@reddit
I don't know that I did particularly well on that question. Off the top of my head I knew syscalls and interrupt handlers. Past there I was encouraged to speculate, and speculate I did... Didn't get the job in the end.
excludingpauli@reddit
Implement a basic version of ps using only bash natives.
rockpunk@reddit
Yuck. How do you list files in a bash native way? If you assume basic coreutils commands and procfs, it could be as simple as ls /proc/*/cmdline and xarging into cat
What was the interviewer looking for?
excludingpauli@reddit
Yeah I think so, but I didn't get that job partly on account of that question. I just asked ChatGPT for how to do it and this was its solution:
greenFox99@reddit
Had an interview for system engineer role at Amazon. I had to take a scripting language of my choice to make algorithms.
I had two interesting questions: - Print all numbers from 1 to 10. - write a function that minify the path in argument (
/var/../usr/bin//bash
should return/usr/bin/bash
)The first was probably a warm-up to make me write a loop. But I went with
seq 1 10
which does exactly that.For the second part, I don't know how to make it programmatically clear, maybe regex, or cd or pushd based solution, there are many ways. The easiest I found so far is
realpath "$1"
.And I guess both commands come from coreutils. It is worth having a look to those commands, because it can save you some pain every day, and I guess if I ever do recruitment, I'll ask for coreutils commands.
-rwsr-xr-x@reddit
How small can it be? :)
printf '%d ' {1..10}
or
echo {1..10}
or:
seq 10
# as you didHour_Ad5398@reddit
I would write
print '1 2 3 4 5 6 7 8 9 10'
and start observing his expression
pithivier@reddit
I had to stop asking "what's the difference between Linux and UNIX?" because not many people can answer it any more.
Looking for a response involving POSIX compliance and the syscall interface, but it's open ended to give people a chance to show me what they know.
mia_rosecore@reddit
Could you elaborate on the goals with the question, if it's not too much to ask? I'm a very junior sysadmin and my default answer would revolve around kernel design and historical considerations in regards to the development of Linux as an opensource clone fit for generalised usage vs Unix as a research environment that grew from that. Curious what the sort of most ideal answer is
pithivier@reddit
My goal was to assess the candidate's fundamental knowledge by asking them to describe how Linux fits into the Unix ecosystem. Specifically with regard to the system call interface. Decisions by Linus to not implement full POSIX compatibility is why Linux isn't Unix. (And that's fine). But now that the other Unixen are so uncommon, it felt like unfair historical trivia. I'd expect someone at the Staff or Principal level to be able to answer this, but I adapted to the modern context by using a different open ended question: I just ask "what is Linux?"
Bilb-@reddit
One of the questions I used to ask is just to show sticky bits usage. A simple Linux thing but not something people which just passed through may know. Totally different nowadays though.
RomanRiesen@reddit
What do you mean "totally different nowdays"?
flickerfly@reddit
What happens when you delete /Ben. Turned out to be an accent issue, but I was so confused. Fortunately, someone wrote it on the board and it finally made sense. They were saying /bin.
CostaSecretJuice@reddit
Most of this stuff ChatGPT can figure out now. A good reason to abandon the traditional sys admin position and get into more customer-facing, revenue driving positions.
wlavallee@reddit
Interviewer:
So tell me, how would you defrag a Linux server?
Candidate:
Linux filesystems generally don't need defragmentation. They’re designed to minimize fragmentation inherently.
Interviewer:
Okay, but what’s the Linux version of Disk Cleanup?
Candidate:
There’s no direct equivalent, but you could clear cache with
sync; echo 3 > /proc/sys/vm/drop_caches
or clean package manager caches with something likeapt clean
.Interviewer:
Got it. So, what’s the Linux Registry called?
Candidate:
Uh... Linux doesn’t have a registry. Configuration is typically stored in plain text files under
/etc/
.Interviewer:
Interesting. But where do you install .exe files then?
Candidate:
You don’t.
.exe
files are Windows executables. Linux uses ELF binaries — or you compile from source, or install via a package manager.Interviewer:
Sure, but like, what’s the Linux version of
msconfig
? How do you set startup apps?Candidate:
Startup processes are typically managed through
systemd
units or init scripts.Interviewer:
(systematically ignoring)
Okay, next: when you open Linux, where’s the Start Menu?
Candidate:
Well, there’s no Start Menu. Different Linux desktop environments have different panels or docks, but the shell doesn’t have one.
Interviewer:
But if you can’t click Start, how do you reboot it?
Candidate:
...By typing
reboot
in the terminal?Interviewer:
Wait, so no Task Manager? How do you Ctrl+Alt+Del?
Candidate:
You can use
top
,htop
, or kill processes manually withkill
commands.Interviewer:
Alright. Last question:
How would you remove a Linux virus if you can’t open Windows Defender?
Candidate:
Generally, viruses aren’t a major threat on Linux if you follow best practices. But you could scan with ClamAV or, you know, not run random scripts from the internet.
Interviewer:
So, like... no Control Panel either?
Candidate:
(staring blankly)
No Control Panel.
Interviewer:
(leaning back, satisfied)
Wow. Linux sounds kinda limited compared to Windows.
Candidate:
(mentally updating resume mid-interview)
apathyzeal@reddit
98% of the ones I've received are really cookie cutter questions somebody googled.
"Where are crontab files stored"
"Have you ever compiled a kernel"
"Why would you use a dash after 'su'?"
The interview I had for my current position gave me some that I had to think about. I wouldn't necessarily call them hard, but they weren't bad questions and caught me off guard a little with their specificity and actual practicality. Here are two:
"Why would you choose a specific mpm in apache, and describe the common ones."
And
"Explain why you would choose tcp over udp, or vice versa, when configuring rsyslog."
The second also led to further questions about choosing rsyslog over cloud native logging.
shemanese@reddit
We had someone who self-rated themselves as a 9 on a 10-point scale on networking. (10 being the highest). We asked him the difference between tcp and Udp, and he didn't know. We asked him what the term 3-way handshake meant in networking terms, and we finally just called the interview when he started drawing out the third firewall.
homelaberator@reddit
One flaw in self rating is people often don't know what they don't know, so can't form an objective view of how much they know.
shemanese@reddit
We're actually looking for Dunnubg-Kruger
super_trooper@reddit
Lol na y'all should have let him cook
cocacola999@reddit
I have an internal skills matrix to help identify holes in the teams knowledge Vs what is coming up on the roadmap (training needs etc). One guy marked himself 8/10 for linux. Sure cool nothing weird there, until I was on a call with them and asked them to quickly check the logs. He didn't know where to look. I suggested he looks in the default place... Still no idea.. urgh fine, just look at var logs please.... Still no idea and starts fumbling around...... Sigh. Same guy had a history of data work on CV. Dropped a database because they didn't know sql and just copy pasted from the internet. Sigh
Vitringar@reddit
I have been using Linux for 30 years and I still have not worked out the fucking logic for where files are stored in Linux. Sort of hand around /etc but the rest is a confusing mess ( :
GolemancerVekk@reddit
In 30 years you've never come across the FHS and never wondered if there might be some logic to it?
Catenane@reddit
locate -r some.*fucking.*thing | rg -i probably.in.this.dir
(With plocate, of course. Although at this point in time I think plocate is mostly default in sane distros) and then pipes from package manager rpm -ql/dpkg -L are also nice. And of course find/fd. We're gonna find what the fuck we're looking for one way or another is my motto. :P
mgedmin@reddit
And
lsof
to see where some random daemon is storing its logs.Catenane@reddit
Haha. I have a whole lsof-proc script I use for that too to grep by command name, pull out PIDs, etc.
mgedmin@reddit
I have one that greps lsof output by filename, finds the file offset, stats to get file size, and then loops showing % of file size with an ETA, assuming that the program in question reads the file from start to end.
I use it as a substitute of a progress bar.
catonic@reddit
ls -laR /var/log
otherwise, go through the config files for the system logger in use and follow them out to the logical conclusions and locations.
/var/www/html - old web root.
TabTwo0711@reddit
What other handshakes besides the 3way do you know in networking
sedwards65@reddit
"the difference between tcp and Udp"
You could explain it to him, but he might not get it.
vainstar23@reddit
Do you understand? No
Do you understand? No
That's TCP!
...
I don't get it..
That's UDP!
Team503@reddit
I giggled.
vainstar23@reddit
Nooooo
Why would he say that? I thought it was common knowledge never to rate yourself 10/10 to prevent exactly this..
apathyzeal@reddit
A lot of people way overrate themselves hoping to avoid scrutiny. I remember an interview I gave to a help desk person who taught himself entirely on chatgpt. He had a functioning home lab (or claimed to) and I found this fascinating, if not alarming because he didn't say much about how he verified what he learned from the llm was accurate. I called the interview though when he expressed not knowing active directory (a requirement for the position) despite it being listed as a top skill on his resume.
-rwsr-xr-x@reddit
I am a pedant when it comes to this one. I constantly see peers and customers on shared screen sessions do
sudo -i
, and think they're getting a root shell.They're not.
Using:
sudo su -
vs.
sudo -i
Are not the same thing. Quite far from it, and the latter will eventually bite you if you're not aware of the difference.
sudo su -
is almost always the correct choice.Hint:
printenv > /tmp/$$
under each context, and thendiff
ormeld
the two files to compare.Will-E-Coyote@reddit
I hate when people use
sudo su -
. And hate when people say this is the correct way to get a root shell.-rwsr-xr-x@reddit
It's unfortunate you don't understand the difference. But at least there are manpages to help you educate yourself!
IHaveTeaForDinner@reddit
What do you consider the correct way?
Will-E-Coyote@reddit
I consider
sudo -i
as the correct method. This way it doesn't need to start a slightly misusedsu
to get a login shell.bmayer0122@reddit
I am only getting a difference of:
The above being in the
sudo -i
instance. How is that going to bite me?mgedmin@reddit
Several years back various distros (Ubuntu specifically, probably also Debian) shipped a default /etc/sudoers that had
PreserveRoot HOME
by default. So when you did asudo -s
to get a root shell and then launchedvim
or something, your /home/notroot would have a bunch of files (.bash_history, .viminfo etc) owned by root:root, and you would notice things like bash and vim no longer preserving your non-root history.(BTW I can't for the life of me remember the difference between
sudo -s
andsudo -i
. I always used the -s version because it's mnemonic for "shell". I will now go look it up in the man page, again, and then forget in 30 seconds.)jgo3@reddit
One of my favorites was, "How do you use cron?"
"With the crontab open in one terminal window and the man page open in the other."
"Whelp, that's how I do it!" makes a check on sheet
Catenane@reddit
By using a systemd timer because it's not 2004 lol
420GB@reddit
Yea I was gonna say this question is not that easy lol.
Last times I configured a cronjob was always through ansible but mostly I've been adjusting to systemd timers
jgo3@reddit
"Fuck you, kid, I got nothing to prove." --Jill Sobule
eightdigit@reddit
That man crons.
mcmx2@reddit
During an interview I was asked which command used to see the syscalls of a process (something like that).
I only knew how to use "strace command". I didn't know it could connect to a running process 😞
I still got the job... Years after I knew there was only another applicant to the position, I felt bad for a while.
WTFpe0ple@reddit
My one and only question at Sprint was write down all the 3 digit chmod numbers and what they mean. I also included the 4th digit they were impressed.
Reddit_User_Original@reddit
Thread is a goldmine
Amidatelion@reddit
A docker node's filesystem is full.
docker system prune
has already been run. What could be the issue?mgedmin@reddit
IIRC
docker system prune
doesn't prune volumes, so I always randocker volume prune
as a second command afterwards.praminata@reddit
Depends on the filesystem too. Might have to do a btrfs balance.
Amidatelion@reddit
Yeah, pretty much + dangling images. I think it ended up being
docker system prune -a -f --volumes
. It was designed as a sort of XY problem - tl;dr reporters might make bad assumptions, what do.After saying I'd investigate with
df
anddu
, that lead me back around to/var/lib/docker
having gigs of dangling images and volumes. Don't think I was able to pull the switches out from memory but I explained the problem and passed.ugadawg239@reddit
Do you like snaps?
mysterytoy2@reddit
How do you fix a server with a lost root password
Foxxthegreat@reddit
If it's a Rhel/centos 7 box with selinux enabled , reboot the box,
At the beginning of the boot process, at the GRUB 2 menu, type the e key to edit.
Then, go to the kernel line (the line starting with linux16) and add the following statements at the end: rd.break enforcing=0
then ctrl x to resume booting
The amount of people who failed that part in the RHCSA always surprises me
Fratm@reddit
You don't, the server should be set to not allow root login, and if it does allow it, it should be ssh key logins only :)
Holiday-Medicine4168@reddit
You should never enable root ssh logins, thats the double death.
Fratm@reddit
This is true, but some people do.
Holiday-Medicine4168@reddit
What is the use case? I have been a professional sysadmin for 25 years and I cannot think of a single valid reason somebody would do this.
Holiday-Medicine4168@reddit
I know I was trying to be nice. :) if somebody on my team did this it would send alerts like mad and they would be looking for a new job
Fratm@reddit
Some people just don't understand the risk involved, usually jr admins. There is no use case.
mgedmin@reddit
It's very convenient when ansible can ssh root@yourserver so you don't have to keep typing in passwords all the time.
Holiday-Medicine4168@reddit
Make a group that has the permissions you need and add yourself to the group, or add yourself to sudoers with the no password option. You can just type “sudo su -“ and you are the root user. This (also bad) btw should send an alert to your monitoring system as well as being logged. If you are seeing people becoming root, you need to fix the workflow, it means permissions are not set correctly and they are going around the carefully crafted permissions boundaries of the system.
mgedmin@reddit
What is the security benefit of allowing ssh ansible@host with passwordless sudo, as opposed to ssh root@host? Assuming that password logins are not allowed, only SSH keys work.
(The link you gave is not very specific, only offering vague platitudes.)
Holiday-Medicine4168@reddit
There are a couple of reasons. You want to ensure that the system really never uses root because it makes activity a lot harder to follow and it grants access to everything. This will fail any audit. When you add users to a sudo group you really don’t want the becoming root, just prepending the command with sudo and keeping their users. This is important because if you start doing things as root you can make system resources unavailable to other users. If root goes in and moves files or changes permissions they can become unusable to other users on the system. Assuming you are forwarding your logs to a central logging resource, if you go in as root it may be the last thing that people see before the box is toast or it’s PWND. Additionally it stops sanity checks on actions performed on systems. There is a reason it asks you to confirm you are executing a command. If you do a recursive delete operation in the wrong directory as root, the system is toast. When you enter the system and non root, you are in a proper user directory. That alone should give users pause from doing this. Lastly I would say there is no reason to be logging in at all to a system, if you have to go log into a box there are things you can do to make your life much easier. The way to do this without risk is to create a user group with narrowly scoped permissions that does what you want it to do and then add the ansible user to that group.
Holiday-Medicine4168@reddit
I would also suggest setting up AWS IAM roles anywhere to use them in your on premise systems, you pay 400 bucks a month for the use of their CA service and it provides logging via cloud watch out of the box. It’s the cheapest thing to do out there and you can centrally manage patching from the AWS console, generate reports, lots of stuff, and do it all in terraform. I did it to 1500 windows instances 2 years ago. The first week sucked, after that been never thought about it again
beheadedstraw@reddit
You have local access. All the keys have been lost and you no longer have any of the passwords to login, then what?
mgedmin@reddit
Is there a BIOS password? Is the server using full disk encryption? If the answers are "no and no", you can boot a recovery live system from some removable device, mount the disks, and change the password hashes in /etc/shadow to known ones.
catonic@reddit
yank the power, image the hard drive, divide and conquer
Fratm@reddit
That's it.. keeps raising the goal post :P
Reboot into single user mode, and then reset the password.
Horace-Harkness@reddit
How do you SSH in if the network is borked?
bigkahuna1986@reddit
Unbork the network then login normally?
catonic@reddit
What if the AMI MegaRAID has kicked the array off and the server is running out of the cache only and the hard drives are unavailable?
Horace-Harkness@reddit
I guess log in with a regular user, Su up to root with the user password, then fix the network?
Fratm@reddit
That' wasn't the question. So depends on what the 'borked' part is.. is the network down? then you probably need physical access, reboot into single user mode.
cjbarone@reddit
And if you're local on that server? I think that's the other part of the question
-rwsr-xr-x@reddit
I can say, after interviewing no less than 900 separate candidates for Linux support roles over the last several years, that 95%+ of the candidates, even senior sysadmins who are asked this question, get this wrong.
The wrong answers include:
passwd root
passwd
as rootinit=/bin/bash
, usepasswd
orpasswd root
passwd
commandALL of these are wrong.
The biggest issue here is that you're changing that server's authentication/authorization, which has the potential to restore login access, but break any other services/processes that may have knowledge of that password, or use it interactively (for example, Ansible playbooks).
The first and most important rule when fixing any system, restoring access, troubleshooting, is to change as little as possible to restore service to the host.
You can change the password using
passwd
, but without backing upshadow
, you can't ever restore the system to its previous state.The correct answers include:
/etc/shadow
(or all of/etc/
), before making any changes withpasswd
root
account (checkauth.log
orlast -aix
), then fix only if needed/etc/sudoers
, so a known-good user now has access and authorization tosudo su -
to the root user without knowledge of theroot
password.Lots of ways to slice this that do not require altering the existing, unknown-to-you,
root
password hash.thesaddestpanda@reddit
It’s bad practice to use root for things like ansible because you should be using service accounts. Most shops see changing the password as fine because they aren’t using root for everything.
catonic@reddit
it's running? restore from backup.
wulf_rtpo6338@reddit
Live CD
tenuki_@reddit
As a hiring manager i have tiers of questions and keep asking until I get wrong answers or don’t knows. I warn people first so they don’t feel they have failed, which they inevitably do. Only way to know someone’s skill level. From that experience over the years I’ve discovered that the holes in people’s knowledge tend to cluster around different things depending on their background. Really interesting and less important to me than exhibiting curiosity. I also ask them to rate their knowledge on a scale of 1-10 with 9 being Linus ( hinting how I scale and nobody’s knowledge is complete). Then I rate them after the questions and record both ratings. Bonus points for knowing what they don’t know and more bonus points for asking what the answer is. The people with the most accurate self assessment tend to be hired I’ve noticed. I can’t count the number of jr people who have claimed a 8-10 even after I state the scale and anchor.
I also google the question if it’s a phone interview. The number of people reading the top search hit word for word is frankly astonishing. They don’t make it past that interview.
Anyway, thought I would share from the other side of the question. Your wrong answers are not what I’m judging. ;)
Nebarik@reddit
Linus is 9?!
Who the hell would be a 10 then? This is why I hate the technical questions round of interviews. The interviewer never knows what the fuck they're talking about.
tenuki_@reddit
It may be you.
cocacola999@reddit
One of my go to questions no matter the level of skill I'm interviewing for is "what happens after you type google.com into your browser and press enter"
dmills_00@reddit
Intel used to ask what happens when you push the power button on a PC, then just kept asking for more details.
Great interview question because the candidate can take it any direction they like... If power supplies are your thing, you can go all the way down the switching supply design rabbit hole, if the system management and early boot is more your bag, you can go there, DDR initialization and PCI link training by all means, have at it, BIOS is fair game, wanna talk about getting the thing out of real mode, there is some depth there....
Great fun questions if you actually know any of this stuff in depth.
cocacola999@reddit
Huh I had forgotten the real mode stuff until you mentioned it. I wrote a toy OS in assembly years ago and remember some of the init stuff. Even if it was on a simple microcontroller architecture
dmills_00@reddit
Yea X86 startup is GNARLY, mostly because of really ridiculous backwards comparability stuff, I mean the A11 gate? Come on, you just HAVE to be joking.
I think some of that dies with the move to AMD64 but have not dealt with it in years, so don't know the current state of play.
mgedmin@reddit
A20, but yes. Controlled by the keyboard controller, that makes sense. /s
dmills_00@reddit
That's the bugger, and yea, quite
Also, the PC keyboard protocol was weird, and PS2 did NOT make it better...
cocacola999@reddit
Tbh my mini os was actually ARM based, but did a bit of chip architecture as well at university
Virtual_BlackBelt@reddit
We used a variant of that a long time ago that we called the one question interview. If you start before "hitting enter" you can get into graphics interactions and (back then) Xwindow mouse positioning, go through name resolution, DNS, TCP stacks and other networking, potential hit on database connectivity and APIs.
cocacola999@reddit
You skipped the hardware driver and debouncing the key presses ;)
jgo3@reddit
Which IRQ would the Enter Key be using to notify the system of input if you have an AT keyboard?
anomaly256@reddit
The Enter Key wouldn't be raising IRQs. It would just be shorting the keyboard microcontroller's scan line to the relevant column input. But the answer you're looking for is IRQ1
GolemancerVekk@reddit
"That's between the keyboard and The Almighty, my good sir."
Amidatelion@reddit
We semi-abandoned that until I implemented a variant - "You will be judge by how much your answer sounds like the first result in google."
One guy's face journey as he realized the gig was up was priceless. He hung up the call.
FOOLS_GOLD@reddit
I’ve surprisingly gotten this question at multiple companies like Cisco, Juniper, Spirent, and others. All companies I ended up getting hired at.
One time I got stopped caused I explained the entire process from flipping the power button on the system to getting the web page to load in the window.
lungbong@reddit
Well you break the Internet of course.
lopahcreon@reddit
Which browser?
tenuki_@reddit
lol, very first question I ask, and be ready for follow up questions like history of tls versions if you get past the details of the protocol… ;)
cocacola999@reddit
If let's encrypt and Aws ACM provide free certs, why the hell do orgs spend $500 a year on a cert from some vendor? Allowed answers, 1) they don't do non DV and apparently people care, 2) lol enterprise gonna enterprise, 3) $500 is less than the ducking around time
lopahcreon@reddit
Who’s Linus?
tecepeipe@reddit
an famous youtuber
tenuki_@reddit
lol, good one. I actually had a candidate ask that.
jgo3@reddit
"I thought it was Leeenus."
Catenane@reddit
Leenoos
Fazaman@reddit
Funny thing is: Depending on how he pronounced this, I might ask the same question.
Pronounced like the Peanuts character? It might not even register in my head who you meant because that's now how he pronounces it.
I have the old sound test file in my head "My name is Linus Torvalds, and I pronounce Linux as Linux"
evild4ve@reddit
Lucy's younger brother.
bmayer0122@reddit
These questions tweak me, but I am seeing that they are common.
You bring up a good point that Linus isn't the know everything of Linux. Which brings up the question, What about Linux am I rating myself on?
* How some distro does something?
* How loading modules actually works?
* How to build a kernel, know what the compiler chain is actually doing?
* How it runs an ELF file?
* How many different types of permissions can we deal with on a network connected system and have it still fall under "My Linux machine isn't working"?
There is so much going on, at so many different levels the question is non-sense from the start. But that is why we ask questions.
uberduck@reddit
This guy interviews. And it's the good kind. I'd trust you getting a solid employee in!
Superb_Raccoon@reddit
Linus? The kid with the blanket?
Brru@reddit
Why can't I get reasonable interviewers like you. I hate the idea of claiming I'm an expert in anything, but that seems to be all anyone cares about; Perfection.
zapman449@reddit
"at a shell prompt, you type 'wget https://kernel.org'. What happens when you press?"
I talked for a solid hour on DNS, IP connectivity to the local router, connectivity to the end host over the internet, TCP handshakes, TLS handshakes, HTTP protocol...
And I got the question wrong.
Because they wanted me to talk instead about shell fork-exec the wget binary, signal handling, process management, IPC mechanisms, etc.
(not that they ever TOLD me they were interested that rather than what I was talking about... )
roughly 2010 for a FAANG.
microlit@reddit
Similar to yours: “Tell me how /bin/ls works. Go as in depth as you can.”
I made it to dirent structs in the kernel before waving the white flag.
It was a really collaborative thought exercise. I liked it so much that I still use it in the rare event that I conduct an interview. Gives you a chance to work together and it quickly exposes a bullshitter. I’ve had people make up stuff with extreme confidence, and others get as far as the readdir() libc call and admit they couldn’t go any further.
The depth never mattered, it was the relationship made along the way.
zapman449@reddit
That’s the trick.
I love “tell me about something you’ve done in the last year that you’re proud of.”
I’ve got enough breath/depth that I can tell whether and how much they were involved in $thing.
Fazaman@reddit
I hate those questions because I forget all the cool shit I do a few seconds after I've done it.
Ask me later and all I'd be able to remember is the last time I did some boring update.
ganzzahl@reddit
Same problem for me. The instant anyone asks me about what kinds of cool projects I've worked on, or what I'm proud of, I instantly forget everything.
It's probably a matter of practicing interviewing, making myself a little less nervous – but also my memory is just shit
grievre@reddit
I like to ask "describe the boot-up process of any Linux system you are familiar with".
People invariably answer one of two ways:
I'm always looking for the candidate that gives an answer that crosses that boundary.
bdashrad@reddit
This is a good question, but the interviewer should be steering the candidate towards the area of focus for the role. Your answer isn't wrong, and if they wanted to know less about the network part of the answer and more about the system, a good interviewer would have redirected with something like "how about before the request even leaves the system" or similar.
NeverMindToday@reddit
You said you got the question wrong. That is more accurate than you may have intended.
ie right answer for the wrong question - but they made you guess what the question was. FFIW, I would've assumed it was network question too - if they were after OS level stuff they should've chosen something that just works locally.
zapman449@reddit
Either that or stopped me after 10 minutes to redirect…
ColdToast@reddit
Not sure if it's same for these, but FAANG SWE interviews can ask questions in vague ways expecting you to start by clarifying the context.
It's useful because it shows what areas you have an understanding of and, presumably, could drill down deeper in. Also an indicator that you won't just take one approach to problems, but consider possibilities.
That being said, since you were clearly well-versed in the route you were exploring they should have nudged you given they had a specific goal in mind
Thick_You2502@reddit
how to use lsof to identify files owned by zombie processes? what's is an i-node?
grievre@reddit
Here are questions I ask software engineering candidates and many of them get it wrong despite having years of experience with "Linux":
michaelpaoli@reddit
Hmmm, can't recall many specifically that were all that tough. Perhaps a somewhat esoteric networking question that wasn't even at all specifically Linux - in fact wasn't reaally a Linux question at all ... unless perhaps one was using Linux as a router or the like, then perhaps might be considered a Linux question.
Anyway, some I've asked, and challenges I've run across (and done):
(more to follow)
michaelpaoli@reddit
(continued from my comment above)
mgedmin@reddit
I would probably suggest reading Vim's
:help
on the 'backupcopy' option. If pressed: one is creating a new file + renaming on top of old file; the other is truncating the old file and then overwriting it with data (or overwriting and then truncating). The difference is (1) what happens if the program crashes in the middle of the write, and (2) what happens if some other program still has that file open. E.g. one method works for replacing executables that are currently being executed while the other fails with EBUSY. Another e.g. is crontab -e that wants the same file back and not a new one with the same filename.And this is where I would get stuck, because I don't know (and don't much care).
The rest of this I think I know, except for practical effect of dr--r--r-- directory permissions. You can ls but not stat/open the files inside?
All I know is that they exist and can be used to grant access without adding each key into authorized_keys.
I'm pretty sure I do, because rsync has optimizations. The man page says the option is --checksum/-c.
Last access (with digression about mount -o noatime/relatime), last modification (of file contents), last inode change (eg. chmod/chown). I remember doing experiments checking if opening a file for write/append access and writing zero bytes to it changes the mtime. (IIRC it doesn't.)
Inode creation ("birth") time. When I last investigated it seemed a very non-standard thing with almost no POSIX APIs exposing it, requiring debugfs and such to see on ext2. I now see that even tools like
ls
can show birth times./bin/touch, or the utimes() syscall.
umm, chmod?
create a new file, move it on top of the old one?
Ooh, is that possible? Without temporarily changing the system clock? Or fiddling with debugfs/banging bits on an unmounted filesystem?
The above (changing system clock + debugfs).
Evaluate its parameters as a shell command in the current shell.
eval "$(ssh-agent)"
Replace the current shell process with a new process running the specified command. All of my wrapper scripts that, idk, set extra environment variables (export MOZ_USE_WAYLAND=1), end with an
exec /usr/bin/original-binary "$@"
.Whee I would fail this. I almost never use $() without wrapping it in "", except when I know it will produce one word of output (like
$(pidof process)
when I know one and only one copy of it is running).The output is preserved exactly, I think.
You can nest $()!
I would try
and then
rm -i ./file<tab>
before it has a chance to mess things up.probably. I have used
> file.txt
to truncate files, but I've needed a replacement fortouch
. (Although> file.txt
would also create, but I would fear accidentally overwriting an existing file if I mistype the filename.)Good old
ls -l /proc/$pid/fd
.sfdisk -s /dev/thingy
. (Only I see the manual now says it's deprecated and I should be usingblockdev --getsz
orblockdev --getsize64
.)I have also occasionally poked in /sys/class/block/* for this information.
Yes.
Ehh. What is a 'file'? There are directory entries and there are inodes. Is a file an inode?
(Now I'm curious if one is allowed to hardlink device nodes. I don't see why not, TBH.)
They could be two names to the same inode, or they could be two separate inodes, or one could be a symlink to another.
Hm.
find /dev -ls
gives me what looks like major, minor device numbers in the size column. I could do something with grep and eyeballing. I don't see any options on matching on device numbers in find's man page.I could write a Python script that uses os.walk() and os.stat() if I needed something automated and reliable.
michaelpaoli@reddit
True edit-in-place vs. not - another difference is >!if the original file has multiple hard links.!<
Yes, can get the names, but not stat/open. With d--x--x--x >!the reverse is the case - can stat/open ... if you know the name, but can't get name by reading the directory.!<
You got it, those would be the two possible ways.
" quoted or not, it's still the case that >!trailing newlines are stripped.!<
There's noclobber option (and syntax to override that), but if one needs check the option, already lost the brevity advantage, and yes, of course >> is safe(er), that's also why I'm commonly doing ... >> /dev/null - notably in case I ever typo the filename as root, and as for brevity, the whitespace before the filename isn't needed unless the shell might otherwise misinterpret as something other filename.
Ah, blockdev --gets* options, nice, wasn't aware of (/ didn't recall?) those. Thanks, I learn something every day! Oh, and /sys/class/block/.../size - I'd been using /sys/block/.../size, yeah, ... /sys/block/ and /sys/class/block have quite similar, but not quite identical content ... learned another thing today. :-)
Same >!inode number on same filesystem,!< same file (of any type), otherwise not.
And more generally, *nix allows superuser to >!hardlink directories!< - but that way madness lies, and Linux stubbornly refuses to do so (even though the documentation may still suggest otherwise).
Yep, you're almost there. Add >!-follow and grep!<, and that can do it. Or POSIXly, instead of >!-ls, -exec ls -lLd \{\} \;!< and either way, also include >!-type b before that!< to avoid >!other file types (and symlinks to such).!<
Yes, and similarly, Perl has a built-in find function.
mgedmin@reddit
Wait, what? They do?
checks
Yeah, one is full of symlinks to /sys/devices/..., excluding partitions; the other is full of symlinks to the same /sys/devices/..., but also includes partitions.
Me too!
michaelpaoli@reddit
Yup, for /sys/block/ partitions are down one more level, whereas with /sys/class/block/ they're directly there (and also down one level).
mgedmin@reddit
Oh yes, hardlinks, forgot about those. My biggest fear from the new Python package manager uv using hardlinks to speed up installation of the same packages into multiple Python virtual environments is that I like to edit .py files of installed 3rd-party packages and add debug prints to them when I'm debugging on my dev machine -- what if I forget to remove the debug print and it's reflected in uv's cache and all the venvs, not just the one I used for debugging?
thesaddestpanda@reddit
Wow I’m stumped on a lot of these. Do you have to answers as well?
michaelpaoli@reddit
Have answers, I know the answers ... though for some of the more complex ones I might have to sometimes lookup a bit of syntax or the like (e.g. I've certainly not memorized all the table details to construct a device with the dmsetup(8) command). So, ... pick one or two that you think are the toughest and/or that have you stumped, and I'll give answers.
tenuki_@reddit
Great list. Some of these will probably make it onto mine. :)
Twattybatty@reddit
This, is treasure! Many thanks.
mgedmin@reddit
This is a nerd-snipe, sir! I apologize for adding to the inevitable pileup of answers, but I could not resist!
rm -i ./-rf<tab>
or hitting F8 in Midnight Commander or pressing Del in Nautilus.Ooh! Ooh!
sudo -u THATUSER kill -9 -1
, right?Although this is a trick question because the system is not responsive enough to allow you to enter any commands because no Linux distro ever sets resource limits in a way that would allow it to survive a fork bomb out of the box.
Dunno, but I'd like to know. A few rsyncs, then stopping all the processes that touch the device, then one last rsync?
If you're using LVM you could use pvmove.
(1) deleted files (check with
lsof | grep -i del
) and (2) subtrees hidden by mount points (check withmount --bind
into a temporary location, followed bydu
, because a non-recursive bind mount doesn't have any nested mount points to hide parts of the tree), and also maybe (3) filesystem corruption that throws off the numbers (check withfsck
after remounting read-only).Ehh I bet there's a chance these are exposed somewhere in /sys/, but I don't know. I'd have to look for things.
find /sys -name 'rfkill*'
gives me interesting things already!Not sure I understood the question correctly. You mean like r-----r-- $owner:$group? and I'm neither the $owner nor a member of $group? I do have read permissions then. A more interesting question is what if I'm not $owner but I'm a member of $group. I'm not sure; both options make sense to me. I'd have to test it out or read the documentation. If I had to guess, I'd say I don't have permissions.
Lack of tail compression: each file is rounded up to a multiple of the filesystem block size (e.g. 4K). Plus each file takes up space for its metadata (inode + directory entry).
Finding/modifying one item in a large list takes longer than finding/modifying one item in a small list. Unless the filesystem uses a btree or something for large directories (it's an ext4 option iirc?).
Maybe? Depends on the on-disk data structure.
mkdir, move all the files into the new dir, delete the old dir, rename the new dir to the old name?
Whee please don't tell me backup + mkfs is the only solution here.
Is the answer
badblocks
here? I'm not sure I ever ran it.I could find the offset in the kernel log for the error, but that wouldn't give me the filename.
I could expect to find the filename from the program that tried to access the file that gave me the error.
I could read all the files by doing something like
tar -cf /dev/null --one-file-system /path/to/thing
and then see which ones aren't readable.I could run e2fsck with the option that checks for bad blocks -- iirc there is one? (yeah,
-c
), but I probably won't bother -- I'd get a new disk and copy the files, note down which ones are missing, then try to restore those from backups.Hmm, you could overwrite just that block with
dd
using the appropriate seek/skip/count options. A modern drive ought to reallocate the sector. I would want to check if it worked by dropping the disk caches (echo 3 |sudo tee /proc/sys/vm/drop_caches
) and doing a sha256sum of the entire file, but I'm not sure I would trust that drive. A SMART self-test is in order.ls -l /proc/$pid/exe
readlink on /proc/$pid/exe returns '/path/to/file (deleted)', IIRC
cat /proc/$pid/exe > /tmp/copy-of-old-binary.
AFAIU there's no way of creating a hard link to a deleted file that would prevent it from getting garbage-collected when the last process that has it open closes it.
michaelpaoli@reddit
Impressive! Yeah, you got most of those spot on, and those that you didn't totally nail, still generally pretty damn close, so yeah, good showing!
Don't need the -i, but sure, safer with it. And yeah, the leading ./ prevents the - from looking to rm as introducing option(s), alternatively, for non-ancient rm, one can >!use a preceding -- to indicate the end of options, then any arguments after that that begin with - are taken to be non-option arguments.!!quote the space and * characters - by whatever means (preceding \ or surrounded within ' or " characters).!<
And basically nailed that kill one. And depending how (not?) badly those user's PIDs are behaving, might not need SIGKILL, but that'll certainly do it, or could, e.g., try SIGTERM first, and if that doesn't do it, then SIGKILL. But yeah, most don't know about >!the pseudo-PID target of -1!<, and that's key to beating the race condition.
The moving data one, yeah, if it's under LVM there's pvmove, but if not, as I show in the linked, one can use >!device mapper, via dmsetup - basicallly RAID-1 it onto another block device, and after synced, drop the original, and then get rid of the device mapper device - but will have to make the device available for some bits, notably where one substitutes in - and out, the device mapper device for the underlying device one wants to move that data from/to.!< What I linked to has example (in that case moving md raid10 data from a set of 4 old drives to a set of 4 new drives, while generally minimizing the time the md device is unavailable).
Well nailed the df/du discrepancy - many don't know, fair number cover the most common reason, few come up with 2 reasons, you got 3, very few get 3 (or more? - not even sure there's a possible 4th). Oh, and unlinked open files, can also locate those >!via the /proc filesystem!< - so don't even need lsof.
And yes, rfkill functionality without rfkill command - can be done via the /sys filesystem - I find that highly handy when helping users attempting to install Linux via Wi-Fi, and they need rfkill functionality to do/continue such, but they don't have the rfkill command - and of course can't yet get it via Wi-Fi.
And got the large/huge directory one - concisely explained - a more full explanation gets rather long. And after removing the files, for (most) filesystems where directories don't shrink, yeah, recreate the directory - and that's bad news if it's the root directory of the filesystem, because in that case, yes, that means recreating the filesystem (that's also why I highly prefer to never give untrusted IDs write access to the root directory of any filesystem).
And yeah, unrecoverable read on a single sector/block on drive, badblocks (with -w option) could do it. And yeah, non-ancient drives will automagically remap such upon write, so long as one writes the same location on the drive and the reserved block table isn't already full.
I think, at least in theory, there is a (deep dark magic) way, but I've not actually done so or attempted such. Oh, but there is one relatively ugly dirty way to do it - >!crash the filesystem, then fsck, and then should have it by its inode # under the filesystem's /lots+found directory.!<
oriondracowolf@reddit
Honestly simply asking what happens when you enter a URL and hit return in a web browser is a very telling question. It’s my filter question.
I only hear the term “DNS” 5% of the time.
It’s amazing how many “experienced” IT people don’t know the basics of how me typing this post and hitting “Reply” actually works.
oupsman@reddit
I had to answer the same question for an interview. I think my answer took 5 minutes. The recruter seems to have appreciated the answer.
ryanknapper@reddit
Years ago I was asked if I was a hacker who liked hacking after saying that I had flashed the firmware on a Vonage device to make it work with other VOIP providers. I thought they were kidding. They were not kidding.
treuss@reddit
Had to patch a 3½" boot diskette (loopback filesystem) for Xeon in order to install the os and afterwards Oracle Database during an interview.
bluetac92@reddit
How do you pronounce GNU and what does it stand for
beheadedstraw@reddit
As a Senior Linux Engineer of 20+ years, if I ever got hit with that question I'd just walk out. GNU hasn't been relevant for over a decade and most of their tools have either been completely rewritten or replaced by non-GNU members.
Watn3y@reddit
As a Junior Linux Idiot of 4+ years, could you elaborate on that? Aren’t glibc, bash, coreutils, etc. still very much GNU and used in most popular non-minimalist distros?
beheadedstraw@reddit
Most of the code base has been re-written for those tools and the original GNU members that maintained have either kicked the bucket or left the project. A lot of the original maintainers are also pissed with Richard Stallman for being a drama queen because the Hurd kernel never took off and is more of the "look at me bitching" type vs actually trying to move linux forward.
The only time you hear about this argument is in academic circles still pushing this asinine concept because they haven't updated their courses in a decade. GNU is not what it used to be, and is basically dead besides a few of the side projects like Guix.
I'm not knocking their contribution to Linux 40 years ago, but they're a pale shadow of what they once were and over half of the original toolset isn't even being used anymore with projects moving away from the name because of it's drama filled history.
zakabog@reddit
Ah okay.
Different question, what does the first letter in GPL stand for?
Follow up, what does that stand for?
beheadedstraw@reddit
I'd refer that to legal, as it's not my fucking job to know nor interpret that.
Next question.
zakabog@reddit
You're a Linux user with 2 decades of experience and don't know how to find out what the first letter in GPL stands for? Maybe this is a good interview question, I'd quickly skip over someone that incompetent.
beheadedstraw@reddit
I obviously know what it stands for moron, it’s not my job to interpret licenses for company use, nor should it be yours. All licenses and usage of said software under said license should be approved by legal. Otherwise good luck getting SOX and SOC2 compliance.
It has literally nothing to do with the performance of being able to my job, therefore a completely useless interview question.
Next.
mia_rosecore@reddit
You seem unpleasant to work with.
beheadedstraw@reddit
Ask stupid questions irrelevant questions, get stupid irrelevant answers 🤷♂️
zakabog@reddit
I would say it's clear you don't as you've completely avoided the question and resorted to insults because you have some problems answering this simple question. It seems to be quite good at filtering out the bad apples.
beheadedstraw@reddit
And you gave me an absolutely useless hiring question which would make me walk right out of that interview because it's clear you focus on bullshit rather than actual skills. It's a good thing I have a job and actively contributed to both mainline and RT kernels for the last 25 years also huh?
Those questions are also good at weeding out the people that don't want to work with moronic hiring managers that expect their admins to also not consult with legal OR infosec when implementing new software and tracking supply chain attacks on the security side. God help you when you file a claim with your cyber insurance.
But yea, sure, "WuT duz the G in GeePeeEll stand for?". Kkthxbai rofl.
zakabog@reddit
You took a joke about the pronunciation and recursive acronym that is GNU as a personal attack and declared GNU to be meaningless these days. You then doubled down on your rage boner when jokingly asked about the first letter of the most widely used open source license on the planet (the GNU Public License, though IANAL so you should consult your legal team to verify first....)
Sounds like the question is a perfect filter.
beheadedstraw@reddit
Yea, the perfect filter to get the fuck out of a hellscape of manager mind games and most likely people that have no idea what they're doing so they ask bullshit loaded questions. Typically the people that ask those are unqualified to hire for that position, let alone actually know what they're doing. If I was in a culture fit interview, maybe I'd entertain it, probably not, I'd still have reservations as obviously loaded questions are obvious.
If I go in for an interview I'm not here to joke, I'm here for a job. I'm not your friend, buddy, pal, or anything else. And I swear to god if they use the bullshittery of "We're like a family" I'll punch them in the face because you already know just from that single line the benefits are shit, there's no on-call schedule besides 24x7 and you'll be expected to work 12 hour days. I've been doing this for over 2 decades, I know a shit workplace when I see it.
Maybe it's the military in me and the reason why I get along with my coworkers that are also prior military.
Hotshot55@reddit
I'm prior military and I wouldn't hire you based on your responses here. You're acting overly butthurt over nothing and if anything it just shows you're not great to work with.
zakabog@reddit
lol sure ya do bud, sure ya do...
daniel-sousa-me@reddit
General :|
z-null@reddit
This is a very bizzare answer and not really correct, especially since no one gives a shit about hurd and gnu coreutils are used widely for example in ubuntu and debian.
beheadedstraw@reddit
Hurd is essentially dead, it's a side project that basically no one uses with no clear path or leadership. Coreutils are just the COBOL/Java of the linux world, slowly getting replaced with modern equivalents made with Rust.
z-null@reddit
Hurd is dead since Linux 1.0,.I have no idea why you even brought it up. Gnu coreutils are still gnu, weather written in rust or C and still used. You live in the 90s.
beheadedstraw@reddit
You’re arguing this and still have no idea Hurd/mach is still has an active codebase receiving merges up to this day 😂.
It’s only GNU if they say they’re with GNU. zsh is based on sh/bash but it’s clearly not GNU, just like the team replacing coreutils with uutils rust equivalents isn’t GNU.
z-null@reddit
I can merely feel pity for the people that work with you.
beheadedstraw@reddit
kkthxbai
Spicy_Poo@reddit
Yes. You're correct.
Superb_Raccoon@reddit
I interview in the GNUd.
Spicy_Poo@reddit
That's objectively false.
Nicolay77@reddit
You pronounce it Ñu, and it stands for: "Richard Stallman read Gödel, Escher, Bach and it changed his life".
For how many decades everything open source was named with self recursive acronyms?
much_longer_username@reddit
Never hurd of it.
dusktreader@reddit
welcome to the herd
yqsx@reddit (OP)
Hoje em dia é difícil encontrar
zakabog@reddit
What do you mean...? It's a joke...
Sad_Dust_9259@reddit
debugged a missing semicolon for hours once. Still haunts me at night :D
xstrex@reddit
A physical production server in our datacenter is down, when you connect a crash cart to it, you realize that it’s crashed during boot, with some kind of kernel panic. The server has no internet connection, and you have no external media, but we need this server up ASAP. Walk me through the steps you’d take to get it back online.
ancientweasel@reddit
You are handed a terminal and when you enter a command it says "No more processes". You may not restart the machine. What do you do?
mgedmin@reddit
Ooh, this actually happened to me. I was running fetchmail from cron every 15 minutes and for some reason the fetchmail processes were not getting cleaned up. Luckily I had an xterm open.
Identifying what processes are running could be hard without being able to spawn
ps
, but you can make do with shell builtins likeecho /proc/
andwhile read line; do echo $line; done < /proc/123/stat
andkill
.In my case I think I found a sacrificial GUI process to close (gkrellm probably), and then I could run
ps
/pstree
/killall
until the next 15 minute cron interval. This showed me a few hundred fetchmail instances in zombie state. I don't remember if I ever figured out why the zombies were not getting reaped, I remember a killall fetchmail fixing my session.There's always an option of using
exec
to replace your shell with something else -- maybe a /usr/bin/python -- but this is risky, because then you lose your one and only available shell and might not be able to spawn new ones.ancientweasel@reddit
`exec top` is the answer IMO. Python could be even better because you may be able to list the process tree and see what is thread bombing the node or get open file descriptors of each process and easily manipulate that data and catch it in the act.
Your hired.
tenuki_@reddit
A couple ones I enjoy, not necessarily the hardest:
Four ways to list the processes running on the machine from the command line. Variants of the general command don’t count ( ie top htop atop are one not three.
What do you think of the new scheduler, can you compare and contrast it to the old one and discuss pros and cons?
Describe fixing a bug in a module and updating your host with it. Lots of follow-up questions here around internals, live loading, debugging, grub, ect.
How would you restrict a particular process to certain paths in the file system?
mgedmin@reddit
are
ps
andpstree
two commands or variants of one?I would go (1) ps, (2) pstree, (3) top, (4)
ls /proc/
and variants (likehead /proc/*/stat
andhead /proc/*/cmdline
)tenuki_@reddit
I count ps and pstree in the same family but usually accept that answer. I ask for four in case there is another way I don’t know about. If I was answering it would be some python for the fourth, although I suspect Unix being what it is there are probably ways I haven’t thought of, maybe that I have used on the past but forgot. lol . ( see previous comments about Linus being a 9/10 )
ebriose@reddit
You accidentally ran
chmod -x
on every file in /bin, /usr/bin, /sbin, and /usr/sbin (including chmod itself). How do you get yourself a workingchmod
again to fix it?shemanese@reddit
We have a standard question we give more to see how people work thru the answer than actually know the answer...
Question: On a standalone server, what are the steps that the computer and OS go through from the time you push power-on until you get a login prompt?
They hit me with it, and we still give it. The idea is to see if someone can reason through the main steps needed. It's not a right or wrong kind of question and is intended to see how they think. We have hired people who missed steps because it was clear they had never studied that set of steps but could logically think about what was needed in general terms.
phorkor@reddit
We did a similar thing for junior admin positions. I'd give them a laptop that was mirrored to a screen that we could see, ask a few questions and see how they worked through it.
Candidate: "I don't have much experience with XYZ, is it okay if I use google?"
Me: "Absolutely!"
/few minutes later he had the answer
/hired
Candidate: "Uh, I, uh, hmm"
Me: twiddles thumbs
Candidate: "So uh, hmmm. I don't really know how to do that with XYZ"
Me: "That's ok, you can use any resource you want to figure it out"
Candidate: "Oh, I see"
/he started doing random shit that had nothing to do with XYZ and never got even close to what we were asking
/not hired
Candidate: "I have very little experience in XYZ, but I know the basics. Is it okay if I ask questions to get me on the right track?"
Me: "Of course, what are your questions?"
/asking questions is always acceptable
/hired
Some of the candidates were like deer in headlights and while they might have known ABC really well, if they weren't resourceful enough to even open up google or ask questions they weren't a good fit for us.
Nicolay77@reddit
Nowadays it would be:
Is it OK if I use ChatGPT?
dub_starr@reddit
I blew a python scripting section of an interview because I was trying to be too literal, and didn’t have experience with library they were asking me to use. Feedback was given that I should have just said I didn’t have experience with the library, and asked if I could use google/stack overflow/documentation
maryjayjay@reddit
I had a similar question but it was at a networking company. One of their networking engineers asked me from powering on your PC until you can see Google's website in your browser, describe every packet to and from your computer. I got the job
yqsx@reddit (OP)
First interview ever, they hit me with the “boot process” question. I froze for 10 secs, muttered “f***,” then said something like, “RAM loads OS, GRUB shows, then filesystem check…”
Somehow I panicked and started rambling about filesystem issues. However the senior guy who was asking was cool—kept it chill, added points, didn’t shut me down.
vivkkrishnan2005@reddit
Have reserved this for someone who claims to be a 10 on 10 - What is the best antivirus we can put on a linux server :O
mgedmin@reddit
I have an antivirus (clamav) running on my linux server.
It scans emails received by postfix.
catonic@reddit
a Trusted OS, stripped binaries, no source code, no setuid/setgid binaries, mounted read-only, running out of a RAM disk.
PuzzleheadedSpot1215@reddit
mount / ro
geolaw@reddit
Interviewed for Google years ago. I was warned they're very "scholastic" .... They asked me how tracert works under the hood. I had used it a million times but choked on the answer
mgedmin@reddit
It sends out UDP packets with deliberately reduced initial TTL fields, then waits for ICMP time limit exceeded messages and notes down who sent them, right?
zxjk-io@reddit
List all the 2 letter commands
SuspiciouslyMoist@reddit
My first interview; not the hardest question but the one I remember - "What's the difference between /bin and /sbin?"
It was a while ago.
hbp4c@reddit
Given a directory tree with a few thousand subdirectories and files, find the oldest file. During an interview my head wasn’t in that mode - I knew how the setup the test (they just touched a random file somewhere in the tree) but my brain locked up and I couldn’t think of a good answer.
Answer is: find . -print0 | xargs -0 ls-ltr | head -1
mgedmin@reddit
I once asked that on stackoverflow! I don't remember the accepted answer, but I think it was something like
find -printf "%A@ %p\n" | sort -n | head -n 1
The xargs solution has a limitation where you can hit the kernel's command line length limit if you have too many files. "A few thousand" might be pushing it.
Fazaman@reddit
I'd need to use a man page to figure it out exactly, but my first thought was a find /basedir -exec stat $options {} +|sort |head -1
The specific option to stat to print the appropriate date (%W or %w for time of file birth, it turns out) with the filename, is what I don't know off the top of my head, but either unix time or human readable would work, because they print posix date/time, so it sorts really well!
sedwards65@reddit
The question needs refinement. Do you mean 'in a single directory' or 'a tree with arbitrary depths?'
I'd suggest:
Bonus points if you can convert the EPOCH back to human.
(The grep was to discard some flatpak cruft that may not be relevant.)
(Yes, EPOCH's used to be 'less than 10 digits.')
lazyant@reddit
Or ls -lt | tail -1 ? Not a great question since chatgpt et al are pretty good at this trivia
Hotshot55@reddit
Nah,
ls -lt
isn't going to be recursive and even if you add -R it doesn't really sort all directories well.Dolapevich@reddit
I would also do an
ls -lart|tail
autogyrophilia@reddit
You think they would allow powershell there?
Anyway that solution is a bit inefficient, this will run a lot faster and use much fewer resources :
You will probably want to add a way to filter files with null mtime :
find . -type f -exec stat -c '%Y %N' {} + | grep -v '\^0' | sort | head -1
Twattybatty@reddit
Seeing these answers, I do feel a little under-confident now. It's humbling. But my curiosity will always serve me well. It's great to find your level, from fellow sys admins.
Daniel0210@reddit
I love this feeling. 10 years of programming and networking and still so much to learn.
TabTwo0711@reddit
What happens if you press a key on the keyboard? Go deeper Go deeper
AntranigV@reddit
usually I'm the one interviewing people, and there are some questions that I like to ask, not to judge their knowledge, but to see how they think.
Here are some of my favorites:
ping 123456789
?dig com
it says NOERROR and shows the proper SOA record, but if you dodig ch
(.ch is for Switzerland btw) it says REFUSED and doesn't show an SOA record?Again, the point of these questions is not to make or break, but rather, to see if the engineer has an engineering mindset and know some of the fundemantals.
dariusbiggs@reddit
What happens when you enable do to the traffic it is processing..
Shouldn't have said "please don't ask me about BIND DNS servers, the last four interviews already have"..
That was 20 years ago for a job as a support tech for Google.
cajoel42@reddit
let’s name as many two letter unix commands that you can think of and what they do. bonus: how do i get a list of all two letter commands installed?
marx2k@reddit
"How do you force update the database for mlocate"
For some reason, I knew the answer ( https://www.commandlinux.com/man-page/man8/updatedb.8.html ). I still didn't get the job. I'm guessing they just didn't want me.
thedsider@reddit
Not an interview question but an exam question. About 15-20 years ago I was doing a SuSE Linux Enterprise Professional exam. You work in a couple VMs, complete some practical tasks and then they check your config and pass/fail you. One of the tasks was to set up time synchronisation between servers and domain joined clients using NTP. No problem, except that the VMs were so out of sync with real world time that you couldn't get the server to sync, and we didn't have access to Xen to resolve this issue at the host level
So we failed. Took months to get the testing centre to acknowledge the issue and credit us
MentalSewage@reddit
"What are your top 10 Linux commands"
...wtf? How do you even begin to answer that?
I said the classic " unzip; strip; touch; finger; grep; mount; fsck; more; yes; umount; sleep" and they gave me the job
oldzoot@reddit
My favorite linux / unix interview question was ( I was the interviewer ) You have a computer with your favorite distribution installed. Please tell me what happens from the time you turn on the power until you get a login prompt.
This is a great question for learning what they understand about the system. Someone who understands a different OS deeply should have no problem adapting to a different system if needed.
I hired my replacement when I retired based on their answer to this question along with a few others relating to automated installation systems.
gudlyf@reddit
I used to ask -- and I'd make it clear it was mostly for fun -- if they could describe different RAID levels. Most got thrown when I asked to describe RAID 50 or RAID 0+1.
tenuki_@reddit
Here’s one that actually happed to me: here’s your workstation, last person to use it set the firmware password and changed the root password. If you can use it by tomorrow morning without reinstalling or wiping you have the job. Then find the source he was supposed to be working on and finish it. ( technically this wasn’t Linux it was a sun high end work station running Solaris. ).
beheadedstraw@reddit
Does Alpine linux use glibc? If not what does it use?
Bonus points for why it sucks.
stroke_999@reddit
Alpine Linux is the best distro ever! Send me the name of the company, I will do an interview with them
tenuki_@reddit
What is your favorite distro - alpine answers to the head of the class
beheadedstraw@reddit
Me: LFS (Linux from Scratch).
Some smartass: "That's not a distro".
Me: EXACTLY.
beheadedstraw@reddit
Compare multithreaded performance with glibc due to it's shit malloc and get back to me.
stroke_999@reddit
Oh cmon, security is always the opposite of performance. However for everyday use you will not see any difference, bot on servers and on desktops.
beheadedstraw@reddit
You obviously don't work in a performance mandated environment where we measure latency in the single MS to the nano's. You wouldn't survive in the AdTech and FinTech world.
Also the use of musl in Alpine has little to nothing to do with security and more to do with footprint and size. The touted touted security benefits of it are miniscule at best as most security flaws are user based misconfiguration in nature vs the random buffer overflow on something that's not even public facing 90% of the time.
I hold a CASP+, so I at least somewhat know what I'm talking about in regards to Cybersecurity implementation wise.
Thev00d00@reddit
Lol imagine using certs to win internet arguments. I passed an exam so everything I say is correct!
beheadedstraw@reddit
Ok cool, so where am I wrong oh great and wise one?
Thev00d00@reddit
I didn't claim you were incorrect, just that flexing certs in on random threads is weird.
beheadedstraw@reddit
So trolling just to troll, got it.
Flexing equivalent DoD level certs on a thread isn't weird, and I didn't claim I know everything, but at least gives some sort of credence to my knowledge level considering CASP+ isn't just a rote memory cert like Sec+/A+ (and according to other people extremely hard since it's a pass/fail with no scoring mechanism). It's the equivalent of a CISSP on the implementation side.
You would know if you actually had it or even studied for it.
IWorkForTheEnemyAMA@reddit
https://youtu.be/gYqF6-h9Cvg?si=E_0KZ1N5UGyPt6_c
beheadedstraw@reddit
Man I love watching her talks, she was amazing.
stroke_999@reddit
OK I'll stop it. This is not the right topic. Yes every distro is for a purpose. I think that fintechs should buy larger hardware and be more safe, however they all fear changing, and they are right. You can't say that having less footprint is not relevant to security. Less things = less vulnerabilities. Alpine is a security distro. If you compile apps with security in mind it is always a lot better. Just see how much 0 day vulnerabilities are found on alpine and compare them to Debian, there are like 90% less vulnerabilities.
beheadedstraw@reddit
"however they all fear changing,"
You're thinking of banks, not fintech, which consists of mostly proprietary trading firms. We use literally the latest tech, bleeding edge kernel source trees and networking hardware. In fact a lot of us help maintain the RT kernel.
Losing 100 nanoseconds is a huge deal in our world.
"Just see how much 0 day vulnerabilities are found on alpine and compare them to Debian"
Alpine is typically used for API's and other things that are hidden behind either firewalls, proxies or load balancers. Alpine also isn't focused on end user usage, but extremely more niche toolsets and usage, mostly in K8s/Docker Swarm, etc. With extreme niche's come performance and compatibility drawbacks. If you give less than a shit about your containers using 4x the CPU, can't hire a security team, and can't afford a decent SIEM... then yea, Alpine is your best bet.
stroke_999@reddit
Oh sorry, let's also talk about how apt and third party repository always break everything and how much time you loose fix all kind of things. Let's talk about the decisions of giant distro like Ubuntu that are packaging software that require speed like lxd in snap format. I think that those are worst things than keeping an eye on security. Try to use lxd in Ubuntu and than try to use it in alpine, you can also avoid benchmark because you really feel how much Ubuntu is slower
lazyant@reddit
Not a good question. I kind of know the answer because at our company we were bitten by an issue with muscl, otherwise it’s a don’t care situation mostly.
beheadedstraw@reddit
It is when you're relying on performance and measuring latency in the nano's. musl has terrible multi-threaded performance.
lazyant@reddit
In our case it was a bug in a library but thanks TIL
altodor@reddit
I don't even remember the question exactly, but it was based off of a real problem they'd had and solved. The answer was that there was a code bug in virtualbox corrupting or faking some or all writes to the disk image.
Klosterbruder@reddit
Not a hard question or anything, but something I could absolutely not answer at that time - "What do the Ubuntu version numbers mean?", back in 2012 or so.
G33kabit@reddit
interesting questions.
Bammer7@reddit
Years ago in a group interview I got asked to explain how to do some sort of complex search. It was hard to visualize, but back then I could still remember how to do all that that crap without google. I said I could figure it out if they let me use the whiteboard. So I got up and wrote some complicated string greping and auxing some excludes and includes all over the place. I could tell they had no idea if it was right but I'm pretty sure it was. They just said ok, next question. I worked there for like 7 years.
kidmock@reddit
What is the default signal of the kill command?
I was like it's not SIGKILL, or SIGHUP... F@ck what is it?
leadout_kv@reddit
Is it 9?
kidmock@reddit
SIGTERM aka -15
-9 is SIGKILL aka the signal of last resort. I die inside every time i watch devs use -9 with every kill they issue. Stop doing that 😁
Dolapevich@reddit
Some person edited
~/.bashrc
and left a typo, so when you try to login, it fails and you never get to an interactive shell. ¿How do you fix it?I end up
scp
ing an empty .bashrcgmuslera@reddit
tar command parameters
circuit_breaker@reddit
Explain what ls does, including syscalls and kernel functions when listing /dev
lazyant@reddit
Mine was to explain, for 45 mins , what happens when you type ls * I did (i prepared the answer) and still failed the interview. Fuck Meta.
unkilbeeg@reddit
What's
inetd
and how would you set it up?My answer: Why would you want to?
This was in 2004, so servers by that time had enough capacity not to need inetd. But I ten told them how you would set it up.
I just retired from that job after 21 years.
autogyrophilia@reddit
Ok, but reasonable question in 2004
elprophet@reddit
"Teach me something I don't know".