Do you have a dedicated storage admin?

[-]

feldrim@reddit

If your organization does not have the scale to hire a dedicated team, it is mostly on the system administrators. And this is the case for 90+%.

Reply

[-]

FunnyMathematician77@reddit (OP)

That makes sense. I'm just feeling kind of stretched thin because I'm the only Linux Admin. I'm also expected to know Azure, Kubernetes, HPVC, and day to day Linux stuff. Storage is not my strong suit.

Reply

[-]

Jeettek@reddit

what did you expect if you are the only linuxadmin aka jack of all trades

Reply

[-]

solracarevir@reddit

Wait till they call him because the TV won't turn on...

Reply

[-]

Hebrewhammer8d8@reddit

What about fridge won't notify the business office they ran out of white claw?

Reply

[-]

DrCaffy@reddit

"I broke the screen on my iPhone - you can fix that right?" Set limits early. Set them often. If you're about to get a new responsibility it better come with compensation.

Reply

[-]

Tythus@reddit

what you mean is you will be called to support anything remotely tangentially related to IT in very odd ways

Reply

[-]

LostLakkris@reddit

I work for a managed storage vendor, our customers range all over from "don't have an IT person" to "have a team of 4 people just managing the managed storage service." I recall we once had a customer whose storage support team, who exclusively used our products, was larger than our own support+ops team. That customers team managed ~4 sites, our managed those 4 plus another 60 at the time. This has also led to support calls of all technical levels and sometimes resorting to offering some non-storage help because the customer is like 95% of the way there and that extra 5% from us will lock in the contract. That's ranged from monthly product training, to helping diagnose why their replica domain controllers aren't answering on behalf of the entire forest, to designing custom shim solutions to add that one dumb feature(non-standard DR, retention logic invented by a graphics designer, etc). So even just being the storage admin quickly leads to solving obscure issues in all stacks, as most customers will eventually blame the storage when the issue has nothing to do with it. I've been on plenty of calls where we've had to have their engineers draw the data flow, then we ask when does it hit the disks, ultimately it never did and was way out of scope.

Reply

[-]

SurfRedLin@reddit

Me. Will be doing mostly ceph in my new workplace but also some sysadmin work.

Reply

[-]

secretlyyourgrandma@reddit

> Will be doing mostly ceph in my new workplace but also some sysadmin work. lord. ceph is awesome, but good luck. if they cut corners on the hardware and network requirements, i would plan an exit strategy.

Reply

[-]

SurfRedLin@reddit

Afaik 10g network is planned. Extra hosts for the Mons with sdds and 128gb min ram for the osd hosts. That's quite in line what I could find online. Any tipps from your side? Thanks a lot!

Reply

[-]

secretlyyourgrandma@reddit

I learned what I know about Ceph working on OpenStack, so I got into troubleshooting and worked alongside the Ceph guys, but am not an expert. What you say sounds pretty much in line with what they recommend. I would thoroughly read the recommendations, spec it out, and then try and get feedback from experienced storage (preferably Ceph) people online. You need to have redundancy in the network, and you want to make sure that the math on replication time works out so that recovering from a failed disk takes a reasonable amount of time. Beyond that, I would just try and absorb as much about it as you can starting now, since it's a truly massive product with a lot of considerations. docs.ceph.com is great, but check out [Red Hat's documentation](https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5) as well. Some of it will be specific to the Red Hat build, but stuff like the Storage Strategies Guide seem like very useful reading. My strategy would be to set up a small lab if possible, and as I worked through the docs, run commands, test things out, etc. Don't focus on retention, you want to be able to remember that a command exists and that you can look it up. Also, ceph has command line switches like --yes-i-really-really-mean-it for a reason. Think about things before you press enter. Don't make changes when you're tired. If you're not sure about something but you think your boss is going to yell at you for taking a long time, let him yell at you.

Reply

[-]

FunnyMathematician77@reddit (OP)

How much data are you working with? Petabytes? How do you backup and archive storage that large? These are challenges we are currently facing. I have been looking into parallel filesystems like BeeGFS, but I'm unsure how to back up data at petabyte scale.

Reply

[-]

SurfRedLin@reddit

I'm starting this new job on the 1. Of January. We will build up to around 3 petabyte of storage which means 9pb raw. Ceph is very resilient we will not use a backup in the traditional sense as its not cost effective for us to do so. Instead we will use strategies in ceph to make it more robust. This is also a decision we have made as we only will be using copies of the original data and track them and if the worst should happen we can just reupload the data into the cluster. Our cluster will be an archive in a sense. How ever you could totally build your payment and cost structures to support two clusters. And then you have a backup. If you go the ceph route I would be happy to discuss stuff with you as I'm also a beginner in this. If you want hit me up in pm so we can get smarter together ;)

Reply

[-]

thedude42@reddit

It all comes down to business needs + budget. If the business is healthy and growing and they only have you then it means the availability of the systems you're managing aren't that critical to the business. If, in fact, the things you manage are critical and you're the only one, hold on tight for a wild ride. My experience has been that there are companies out there with unscrupulous owners who have no real solid notion about what they actually need and that without the boots-on-the-ground knowledge they can just hire talent and everything will work itself out. They tend to not see their business as a business but rather a way to make themselves money, like their job is to run a business rather that taking personal responsibility to provide a valuable component to the economy. These people have no vision beyond raking in profit without having to lift a finger. I hope this isn't the situation you're in, but just in case I can tell you that it is well known these kinds of companies exist and if you detect the stink of such a workplace there's no shame in cutting your losses and moving on ASAP. On the other hand, if the business isn't ridiculously overextended then you have an incredible opportunity to really stretch your legs and grow, assuming there's no oppressive manager trying to make all your decisions for you.

Reply

[-]

mysticalfruit@reddit

No, we don't have a dedicated person. I've cross trained everybody to the point we can all stuff.

Reply

[-]

secretlyyourgrandma@reddit

Yeah. Depends on a lot of factors. Sole proprietorships don't even have a guy who does payroll.

Reply

[-]

thefunrun@reddit

We use to have a storage team, but they have essentially gotten easier to manage. Usually have gui to click or you just schedule with the vendor for the maintenance, so it's not really a full time position anymore.

Reply

[-]

michaelpaoli@reddit

Varies greatly by employer or organization. Some have dedicated teams for storage - or even *types* of storage. Others, nothing of the sort.

Reply

[-]

uptimefordays@reddit

Depends on the size of your organization and industry.

Reply

[-]

AlgorithmicAlpaca@reddit

Dedicated roles? What is this? Never heard of that.

Reply

[-]

FunnyMathematician77@reddit (OP)

You've never heard of a specialist?

Reply

[-]

AlgorithmicAlpaca@reddit

Sarcastic, rather. But yes, of course.

Reply

[-]

faygo1979@reddit

Dedicated storage guy right now. We are a mid size company and a few thousand servers When I worked at a small company with under 200 servers we shared duties

Reply

[-]

FunnyMathematician77@reddit (OP)

makes sense

Reply

[-]

psmgx@reddit

at large companies, yes. current gig we have a Data Team, who handles the data warehouse, data lake, and all primary and secondary data sources. This includes backups from on-prem systems, all databases be they MSSQL in Azure, Oracle SQL on prem, etc, and any of the physical systems like SANs. Previous gig, also an F500, had a dedicated "backups" team that handled all backups and data recovery on the Infra side. Cloud and SAP lived in their own worlds and did whatever they felt like most of the time. Before that at smaller orgs it was mostly on the sysadmin teams.

Reply

[-]

FunnyMathematician77@reddit (OP)

We do have a data team, but I'm not sure what they do exactly. We are in charge of backups, DBs, and storage servers.

Reply

[-]

aenae@reddit

At my job i do storage, planning, budgets, vendor contact, budgets, dns, security, networking, loadbalancing (about 2B web requests/month), database, server setups (30 bare metal), kubernetes/containerization (three k8 clusters), vms, monitoring, backups, logging and incendent response (even caught multiple hackers/attackers and did the communication with authorities/legal, at least 3 went to court and 'we' won them all). I love my job, at least it has some variety instead of doing 1 thing all year. A single team for storage, or even a dedicated person sounds boring to me.

Reply

[-]

ybizeul@reddit

That doesn’t seem right (or healthy) but as long as it works for you and your company that’s fine. It’s just too much to ask one person to manage day to day activities with all this and stay on top of the technology and upgrade cycles. Some extremely dedicated individuals like you might be able to cope with all this but that’s not (and shouldn’t be) the norm.

Reply

[-]

aenae@reddit

It works for me, and i keep it healthy :) It is a fun job full of variety so im rarely bored.

Reply

[-]

iggy_koopa@reddit

Probably more common then you think. Take away budgets and vendor contact, then add in some dev work, and this is me.

Reply

[-]

ybizeul@reddit

It’s very common in SMB, even doable, but at some point it doesn’t scale anymore. There is so much automation can do, and thats something else to maintain to keep up with product upgrade and changes. Now how much is too much for one person, mileage varies, I work in multi petabytes environments spread over the globe with 10s of thousands of VM, probably not what the OP is talking about. I think it depends a lot of how many business units depend on your infrastructure and how much activity they generate in terms of provisioning VM, creating new environments, does that impact security, firewall and vlan, how often you patch everything, maintain master OS images, manage package repositories, capacity and performance planning etc. There is so much going on in complex organizations it can be a nightmare even with dedicated teams.

Reply

[-]