Roast my setup (actually not mine)

Posted by macbig273@reddit | sysadmin | View on Reddit | 11 comments

So, I inherited the responsibilities of managing an infrastructure (about 2 years ago). Wasn't actually what I was formed to, but hey, why not. Just wondering how "out of date" or "overkill" or "you should move that onto elsewhere" it is. I don't have the numbers right here, cause I'm not at work... I'm just drunk at home (obviously xD). That's what I have in a server room (of course everything has dual power supply with one on UPS and dual ethernet : \- Ceph / Virt cloud (4 ceph, 4 virts) to run virtual machines (abouts 30 of them, including our main gitlab), using opennebula as orchestrator \- One other storage server (because this days, data take too much size) about 30 To in raid 10 on a specific machine that will share that space using object storage protocol (minio) \- One ML machine with 4 GPU \- Our orchestrator on a physical machine \- One specific physical machine manage the backups (running bareos) and copying the backups to another place  Most of it run on dell's R4XX except the ceph/virts that are ... don't remember micro-something. (4 virts in 1 or 2 U) \- Everything is behind two network connections. multiple vlans, ... using one dedicated firewall (with a replicated backup) running pfsense.

Reply to Post

11 Comments

[-]

xeanaex@reddit

Sounds like a pretty good setup. What are you doing on the security side besides the firewall (s)?

[-]

macbig273@reddit (OP)

well, except the firewall rules, not much. Keeping up to date with the current potential issues, updating the servers etc ... I tend to think that my users (mostly developers) are not morons,... ( I know that I should not assume that...) We have 2fa everywhere it's meaningful. On every servers we're using : bareos (backup) icinga (monitoring) freeipa (for centralized ssh access) ... (I say "we" cause there is another guy in the mix. But his competence is just asking chatgpt for answer instead of thinking about it... so... )

[-]

xeanaex@reddit

Lol. Chatgpt is fun sometimes :)

[-]

sysadminsavage@reddit

> freeipa I take it the last guy was a huge Red Hat fan? I've rarely seen both ceph and FreeIPA (or the downstream RH equivalents) together in the business world outside of huge companies. If you guys are a small shop I'm even more impressed. However, if you don't really have a need for these specialized tools, i'd consider switching to more well known tools if budget permits. Only reason I say that (and it pains me to say that because I love to have full control without proprietary abstractions when it comes to infrastructure) is it can be tough to find a replacement admin/engineer down the road that can support them and is familiar. Security wise I'd strongly suggest a good SIEM, Endpoint Protection suite, Vulnerability scanning tool and log management. Wazuh is a great choice for most of those and offers professional support. Backup wise you may want to consider a more comprehensive tool like Commvault or Veeam that can restore AD objects and the like. Commvault is complex but it's saved my ass at least a few times.

[-]

macbig273@reddit (OP)

Well most of our "important" machines were running centos (until we recently upgraded all of them to alma, with a lot of stress regarding the virts and ceph, I must say). So, I'm not sure who started the RH trend in here, but I must say that I learned to love it :) Anyway thx a lot for you input. >I've rarely seen both ceph and FreeIPA ... together in the business world outside of huge companies Durring the last 15 years we've been fluctuating between 10 and 15 people... My users are mainly developpers / engineers / master degree in various IT fields. (from web dev to AI stuffs). I'm not sure why the company started to have it's own server rack, it was probably very advantagous like 15 years ago. But more and more I wonder if we could migrate some of our service on online clouds. Every few months I tend to run some checks from "hostedscan" on all our external IPs. Concerning the log management, it's in my planned tasks, but didn't really had the time to invest in yet. Thx a lot for your suggestions, I'll definitly keep them preciously and look into it !

[-]

tr1llkilla@reddit

Maybe add another firewall and integrate your custom firewall frameworks into the network routers or use the routers built-in firewalls?

[-]

aanerud@reddit

Get in a vm with openvas (greenbone Community edition) and start monthly internal vulnerability scans. That would be like an icing on the cake :p

[-]

macbig273@reddit (OP)

Gonna look into that one. In some previous research, I know that one "nexus vulnerability scanner" was on the table, but I'll get a look into that one

[-]

BlackV@reddit

> So, I inherited the responsibilities of managing an infrastructure (about 2 years ago) It 100% is yours, you've been there for 2 years, any garbage not fixed is now on you

[-]

macbig273@reddit (OP)

I totally agree. That's why, after on it for about two years, I have full understanding of what's as work. And I need another input. Is that good, is that bad ? I cannot compare to anything else since it the first infrastructure I work with.

[-]

Ssakaa@reddit

> Other than this being a glaring single point of failure, sounds reasonable. >One ML machine with 4 GPU Depending on how important that is to day to day operation, again, glaring single point of failure.