A day in the life of a linuxadmin
Posted by FatMili@reddit | linuxadmin | View on Reddit | 23 comments
Hey, was thinking if you want to share a day in the life of your current job.
What do you do? How long hours do you work? Do you get called in weekends and evenings? What’s your title? Small or large company? Pros/cons? How would you like it instead? Maybe this can be your guideline
It would be interesting to see different aspects of the Linuxadmins.
There are some older threads here already but times have changed and lots of new people here as well.
djsanwich@reddit
I've been a Linux admin, cloud admin, at managed hosting, robotics, now in AI.
I loooooove being a Linux engineer. It's like playing Sudoku all day and getting called a wizard for it.
While everyone drives to work, I'm getting my coffee and heading back to my place ...
God. What a blessing.
FatMili@reddit (OP)
Damn, this sounds so relaxing somehow. Thanks for sharing.
Adventurous-Put5718@reddit
Job title is pretty generic: Senior Systems Administrator. I support a team of data scientists. Our servers are a mix of RHEL, SUSE, and Ubuntu. Posit Workbench support, running a SUSE Manager server, docker/podman, and some other stuff. Very small team. We have fewer than 50 servers.
Pros: Public sector, so the hours are good and I only work weekends to apply patches. I could go on forever about the pros. I really enjoy being an admin in the public sector.
Cons: We're given a budget, and the budget is the budget. There are things we'd like to do technology or infrastructure-wise that we can't. We don't even have the budget for test-dev-prod, so we have to be super cautious making changes.
justinDavidow@reddit
Hey /u/Fatmili! Thanks for kicking this off!
Why not share your own story too? ;)
Lead a team running a hybrid of "on prem" and cloud infra.
OSX shop (about 1300 seats) and strictly Linux servers, business is a web shop but we also cover capacity for CRM, BI, file storage, VPN, Databases, caches, monitoring etc.
We run a blend of AWS ECS and AWS EKS autoscaled between 400-4000 nodes (nodes range from 4-core/8GB-ram to 64-core/512GB-RAM) along with a slowly dwindling number of "on-prem" (rented nodes in remote datacenters) nodes running the services that don't benefit from autoscaling. (~50 x 32-core, 256-512GB nodes)
My team is amazing and it's incredible working on such interesting projects with them! The broader organization is incredible and it makes me happy to go to work each day.
In my case, pretty standard 8-4 most of the year. This time of year (November) we get busy as hell, I'm often doing 3-4 10-12 hour days per week and an hour or two on Saturday or Sunday (as things need attention).
The org I'm with favors work-life balance, so "overtime" means taking 1.5 hours off to make up for it.
In my specific case, yes. I'm on-call 24/7/365 as the on-call team fallback. (If the people on-call don't answer, it escalates to me). In practice, we have 1 on call event every 2-3 months, and with the org spanning 6 timezones between Europe and North America: someone is usually working when such emergencies occur.
Apparently I'm the "director of IT infrastructure" these days.
Started as a junior sysadmin ~10 years back with this org and these days I hire ....shudders... "DevOps Engineers" (DevOps is a methodology.. not a job title!)
My title determines what I'm responsible for, but not what I do. I direct my team and drive high level change over time, but I also do everything from answering support requests to debugging physical network infrastructure to helping write software to cleaning the coffee machine. (I like it cleaner than most, what can I say?!)
1300 people spanning a mix of hybrid / full remote across 15 countries mostly in two continents, but I also work closely with a few other business leaders every 6-8 weeks in India, Australia, south Africa, and south America.
"Medium" (I guess) against the 5000+ I've worked with in the past!
Not sure what this means.
What would I like to change about the org I work at? My job is to change how we do what we do. The things I want to change are already changed or in progress.
We'll be doing a "move" of some workloads to cloud capacity, and move some workloads that no longer benefit from autoscaling back on-prem in 2025, along with some major structural improvements to the development stack used by ~120 developers. Monitoring is a major focus this year, we have some fun plans to get a SUPER streamlined super efficient logs and metrics monitoring stack in place at a fraction of the cost of the major players in the market.
Lots of fun ahead, I'm excited for 2025!
Dontemcl@reddit
If I want to become a jr Linux admin, what are some steps I can take? What certifications should I get? What projects to do in a homelab? Do I need CCNA? I do have 3 years of IT support experience already. Thank you!
justinDavidow@reddit
Great questions for /r/itcareerquestions
If I were starting over today, I'd:
After finishing university (~4 years), and 3-5 years work experience if you're motivated, you'll have more than enough experience to apply pretty well anywhere.
Certifications are rarely all that useful unless you want to specialize in a specific tech.
That said, broad certificates like * AWS solutions architect, DevOps Engineer; (or similar for any cloud providers) * Cisco Networking CCNA + any level above * Linux+ or redhat certs
Are all very useful skills that might help get your foot in the door somewhere, though you need a LOT more foundational knowledge than certs typically provide.
And have you progressed?
If yes: great! You're on your way. Look for a new org that fits well with your current skills that is looking for "a few things you don't know yet", you can help them with the things you do know, and they can help you with the things you don't.
If you're still doing entry level support, it's past time to either push for progression or find a new company. (If they won't help you grow, they are not helping you: they are taking advantage of you.)
Linux admins, in my experience, are highly self motivated people. We see things that interest is and we push to learn.
Unlike closed-source land, you can nearly always dig further and better understand how things work all the way from high level architecture design down to the physical electrical signaling level: use that to your advantage to keep learning and developing new and useful skills that you can put to work and grow: both yourself and the businesses you support.
Best of luck!
Jonuzi@reddit
Not OP but I really appreciate this answer! Very useful tips
CheerfulAnalyst@reddit
Get your RHCSA and some Ansible experience, you'll likely get a junior position with that. Fuck the degree, that's only required for companies with legacy HR policies.
FatMili@reddit (OP)
Thanks a lot for replying and I sure as hell should write about myself also. I'm however not working with linuxadmin but I'm in IT/Cyber Sec and would like to enter the linux admin world instead. I really enjoy it and have been using it for years in computers, servers etc.
You seem to like your job and cool to see you entered as a junior but now as a director! Good job :)
justinDavidow@reddit
I got started in the early-mid 90's, helping a local BBS get connected to the new fangled "internet". ;)
After high school I had no idea what the hell to do with myself, I figured the trades looked like a solid plan. I worked odd jobs here and there and did hundreds (if not thousands!) of what many today would call "contract jobs".
I was actually an electrician for several years, before I ruptured a disk in my lower back. I got lucky that ~2 years later I have nearly 100% use of my legs back. There was about 3-4 months that I had no idea if I would ever walk again.
While recovering, a friend needed help and set me up with a computer sales job, which I was apparently pretty good at. I started selling pretty complex involved solutions that were implemented by our "external" team (basically an MSP who worked as a department in a retail store) when they told me that they "could not" implement something I sold one day I challenged them and asked why. That led to me taking charge of a team and growing alongside them in many ways.
Years later, my old direct manager offered me a job at his competing MSP, I started in tier 1 again and loved helping people. We quickly grew to support really unusual "tier 4" requests for other MSP's and really hyper specialized stuff. More often than not, our specialty was that we knew Linux in a predominantly Microsoft town.
Still to this day, the networking I did in those jobs sends me interesting questions, challenges, and often exciting opportunities from the thousands of businesses of every size from central Canada (including the northern central states) and nearby.
Ended up at the org I'm with now a decade ago (as of Nov 1'st!) as a junior sysadmin, org needed a hand cleaning up after the last guy left in a rush and getting stuff sorted out. At the time we were a ~100 person org with a totally "on-prem" stack. We were ~300 people when we and another similar org merged into the ~1300+ person group we are today.
It's been a wild ride, and it's not something I would ever recommend to anyone. My life has been a series of happy accidents, not some carefully orchestrated plan of where I wanted to be 20 years on. (Hell, when I was 25, I wasn't sure what I was going to do TOMORROW, let alone years later!)
Linux though: Linux is fun.
There's always more to learn, there's always something new coming, there's always optimization to be performed, there's endless tuning, and then, at the end of the day, there's the business and the hundreds of people who's livelihood depends on the applications and infrastructure that I get to run.
I just like being able to help people, that's what helps get me through each day with a smile!
bityard@reddit
Hey, by any chance are you hiring? :) I recently took a cushy job at a growing tech company but didn't realize until I started that there is almost no technical work to do on this team. Despite my job literally having the word Linux in the title. Most of it is meetings, following up with people on other teams, and chasing down issues with the company's home grown platform that does everything in the company (mostly poorly). I'm itching to get back into something that'll let me dig into OSes, software, scripting, and automation again
PE1NUT@reddit
Sysadmin at a research facility for radio astronomy. The hours are officially 38 per week, in practice you work 40 most weeks and get extra days off.
Pros: no pager duty, most of what we do is batch processing and if it fails, we'll fix it after the weekend. Cons: we're in the middle of nowhere.
The servers are 100% Linux (maybe one openbsd). I've been able to design in sufficient redundancy (ZFS in the storage servers, MLAG for the switches) that we hardly ever have emergencies that would even need someone to go on site - this came in very handy during the lockdowns. There's a lot of automation (Debian net-install, Ansible, automated patching) which enables us to have new servers up and running usually on the day that they arrive. Most desktops are Linux as well, some laptops are Macs.
We have a medium sized installation (over 1000 cores, over 7 PB of storage, 2x 100Gb/s network to the outside world).
spacelama@reddit
I wonder if I know who you are 🤣. Although only 1000 cores, presumably in the middle of nowhere in West Australia, I'm struggling to think what cluster that may be, unless you mean "a lot over 1000 cores" or accidentally said "cores" instead of "nodes" (a problem I've been experiencing this week).
I've been in meetings all week to discuss what we're going to do about a ~9000 ~~node~~ oh hey did it again, core cluster. I've never been involved in this level of architecting before. I'm glad I left federal government; I feel valued where I am, even if it's only a 1 year contract.
PE1NUT@reddit
I've visited that remote corner of West Australia more than once, but that's not where me (or our cluster) are from. But it's a small world, and we might indeed have met in the past.
FatMili@reddit (OP)
Thanks for sharing! Damn your pros sounds so nice somehow. That you don't need to get a call middle of the night due to some server issues or whatever...
What would be your suggestions/focus areas getting into the linux admin field?
PE1NUT@reddit
What worked for me as a network/unix/linux admin: Know the basics of the protocols, sometimes down to the packet wire format if that's important. And with some of the more esoteric stuff that we do, it often is. The higher level stuff builds on top of that, but good debugging generally starts at the bottom. Keep learning.
Stay away from proprietary solutions and vendor lock in - they are very difficult to escape from. And keep things simple: the solution to a performance or reliability problem is rarely to make things more complicated. It's great to be in a situation where you are not dependent on GUIs and 3rd party applications, but can do everything from the command line: that way, you can do it from anywhere in the world (my job does involve a fair bit of travel).
The trick is just to find an employer who appreciates the same values...
G35Gargan@reddit
I'd love to work at the VLA. So peaceful out there.
PE1NUT@reddit
I've been there a few times as well - it's a great site, and there's some really great people working there.
Anihillator@reddit
Well, let's see...
A small gaming company with a complicated history. Tiny IT team of me and another guy on a different project. Maintain a group of about 15-20 servers. Updates, ingame maintenance, monitoring (ansible, elk, prometheus, grafana). DB maintenance and backups (mysql). Maintain various webservices (ci/cd, docker swarm, nginx, some simple monitoring). A small openvpn server to access all of it.
Depends. Officially it's 40, but as long as everything works and all tasks are done within the deadline no one gives a damn, so 10-15hr-ish? Occasionally I stay late because I found an interesting thing to do or implement. Been trying to learn k8s in my free time :D
Only if something breaks and/or urgent. Been a couple times I had to wake up in the dead of night because everything crashed, but those are quite rare. Technically always on call, I suppose?
Systems administrator, with about $1k/month (approximately, I'm not in US)
Small company, I'm not sure how many exactly, but between 50-100 people.
Caddy666@reddit
i just got made redundant, so lounging about on the couch looking for jobs atm....
CrackCrackPop@reddit
Europe
about 400 servers under my jurisdiction 900+ employees in my company
I'm the sole Linux admin
currently it's a slow period, not many change requests or orders from customers most of what concerns the systems is automated by me monitoring for every important service is in place
so I currently idle my days away a little programming nice tools to automate stuff further or just watch some series sometimes I take a peek for migrations and notice customers for deb upgrades since 11 just ran out those upgrades are rather boring though, I simply validate with goss after an upgrade and it's another automated task
honestly I'm a bit bored at the company, but I got a lot of privileges and the pay is good
throughout the year there are always interesting projects for me, very different requests I often pitch new projects to modernize the company or streamline certain processes Yet, most of what I can modernize is modernized at least internally We got nice customers that will often just say yes to pitches so it's a nice way to try something new while making money, usually bigger projects with above 100 billed hours are not an issue, and they are just happy to have nice things
FatMili@reddit (OP)
Thanks for sharing! Sounds nice to have an easy customer so you can come up with creative solutions or ideas and it actually being appreciated. How long have you been in the business? Do you also work remote?
CrackCrackPop@reddit
I work full-remote and have been there for a few years