Favorite stack for accessing and administering linux systems

Posted by Antscircus@reddit | linuxadmin | View on Reddit | 36 comments

Looking for your favorite infra solution stack to access and manage your linux servers in a secure way. Currently we are using SSH sessions from client workstation directly to the datacenters. I’m thinking something bastionlike is necessary to require all admins to pass a centralized demarcation point for visibility & monitoring. What are others using / preferring?

[-]

NL_Gray-Fox@reddit

Previously we used ansible and rundeck. Basically rundeck was the only host that had access.

Log files went into ELK so there was almost no need to log into servers.

[-]

Antscircus@reddit (OP)

So all app installs and configs are done through playbooks? That’s interesting

[-]

NL_Gray-Fox@reddit

Yep, and the auditing is done through rundeck, you can easily see who, when and what was deployed.

I even had playbooks to destroy and create machines through PXE, I could deploy an entire cluster of 10 nodes in 10 minutes.

[-]

Thick_Shop6640@reddit

Puppet/foreman would be better choice to manage state across servers

[-]

nuttertools@reddit

Only for hobby stuff or massive scale servers. The license terms and pricing don’t work for cattle.

[-]

NL_Gray-Fox@reddit

Depends on what you are deploying but puppet is a much bigger investment (time, money and resources) than Ansible.

[-]

UniverseSphere@reddit

JumpServer: https://www.jumpserver.com/

[-]

Antscircus@reddit (OP)

Why does it want to be an alternative to Cyberark? Whats wrong with it?

[-]

UniverseSphere@reddit

I'm guessing open source? I'm not sure.

[-]

itsjustawindmill@reddit

Yeah, I’ve heard CyberArk is crazy expensive. Where I work (big company with deep pockets) the infosec team is very careful about how many assets they onboard to it. For everything else we just have public key SSH from jump hosts, and/or LDAP-managed sudo for specific user accounts.

[-]

vectorx25@reddit

I use saltstack which doesnt need ssh, uses zeromq message bus (although ssh can be used like ansible with salt-ssh)

zeromq is much faster than ssh and no handshakes

[-]

andriosr@reddit

Direct SSH felt like the wild west - no audit trail, credential sprawl everywhere.

Ended up building something with SSM + Session Manager in the past, but it was clunky. Main issues:

IAM roles became a nightmare to manage
No easy way to do JIT access
Teams kept asking for better session recordings

Check out hoop.dev. It's basically a modern bastion that doesn't suck. Key things that make it better:

SSO integration with our existing IdP
Records everything but with zero latency (somehow)
Can enforce MFA/reviews per command
Works with our existing tools/SSH configs

The nice part is it's just a lightweight agent, so no need to modify existing infrastructure. Way simpler than managing your own bastion fleet.

Whatever you pick though, definitely get some kind of central access point. Direct SSH is asking for trouble at scale.

[-]

SurfRedLin@reddit

We use a setup like this:

VPN to the bastion host. 2fa with Google auth. Ssh only uses keys and is setup as a proxy so it has to pass through the bastion. Firewall and ssh will only accept connections that come from bastion proxy. Bastion uses auditd for logging. On top of this we use ansible to admin our fleet. Was quite impressed with that setup when I first saw it.

[-]

snark42@reddit

Bastion uses auditd for logging.

Do you have auditd on all the hosts? Or does auditd have some way to track what's done if you ssh to a remote host? I didn't think it audited the remote session, but I haven't looked at it for quite some time.

[-]

SurfRedLin@reddit

All the hosts

[-]

Antscircus@reddit (OP)

What sort of bastion host are you using? I’m assuming that’s also some linux flavor?

[-]

SurfRedLin@reddit

Hardened Debian 12 with CIS

[-]

dhsjabsbsjkans@reddit

Jump box, ssh, ansible.

[-]

Bubbadogee@reddit

Ansible, and this is gonna sound weird, N8N

[-]

vortexman100@reddit

Oh interesting, because I've basically never seen N8N used anywhere, even though most know about it. Can you share more about how you are using this and what your workflows look like?

[-]

Bubbadogee@reddit

So, N8N can be used for anything where and everywhere, soo, it can be used for anything such as Provisioning a Linux server Creating users in a AD It's a really great tool for standardizing things with automation, which forms is where it's at (just make sure to secure your forms with passwords) One of my favorites is so we have a k8s cluster, we recently transitioned from Kasten > Velero for backups And Velero was missing a lot of those good features like a daily, weekly, monthly retention policy, preset policies for backups, staggered backups, and a easy way to make backups with 2 clicks So made a n8n workflow that allows us to set a preset priority, just say what name space, what priority and then it makes, staggered backups, and retention policies, in 2 clicks

N8N truly can be used for anything, anywhere, everywhere (cause its basically poor man coding at a certain point)

[-]

Simazine@reddit

Considering Teleport atm

[-]

Intrepid_Anybody_277@reddit

Lot of ppl say Anaible....do you mean ansible Tower for a GUI interface or Are you talking command line ?

[-]

protoxxhfhe@reddit

Command line sounds rough but it's not hard plus chatgpt is doing miraculously well on yaml

[-]

Intrepid_Anybody_277@reddit

Oh i agree. I do everything via playbooks and a single Ansible server.

Just i have seen demos of AWX , and it is just a GUI for anisble that looked cool. Never got around to setting it up so was lolking for some recommendations

[-]

Antscircus@reddit (OP)

Do you mean you don’t touch the server, and instead leverage Ansible tower to deploy and config through code?

[-]

Intrepid_Anybody_277@reddit

Yes. I never log in into the boxes for work.

With AWX tower you practically have a button to deploy patching script to all boxes. Or pull logs. Or check sizes.

Currently I have wrapper scripts for my playbook cmds so its a single click for me with the command line but with tower their are more options, i think you can schedules playbooks too which is great.

[-]

Vuiz@reddit

You & /u/Antscircus should look into Semaphore as well. It's like AWX but lightweight (less functionality, but easy to setup & install). But yes, you can schedule playbooks and other things like run them in dry-modes et cetera. AWX has Workflow, basically you can have Playbook X trigger Playbook Y or Z depending on the success/failure of playbook X.

At work we use Semaphore. Works great. We run hourly schedules on our MariaDB galera cluster(s), Grafana, Prometheus, Loki, Alloy agent -installations. And other stuff like that. If I want to change config on one of our Galera clusters I can commit it to our Galera repo -> Press play in Semaphore. Done. Probably going to write some ci/cd stuff [at some point] so that it automatically calls the Semaphore API and runs the template/playbook when repo updates.

[-]

symcbean@reddit

I was expecting this to be a question about orchestration tools, but you are describing your privileged access pathway/infrastructure. So far there is only a single answer addressing the latter, and multiple addressing the former.

For admin purposes, the more you can do to reduce the exposure of your hosts, the better. The fewer the number of places you need to control access (e.g. where you have public keys deployed) the better. Given that ssh is almost universally used for (Linux/Unix) admin access, has some very stable implementations and supports tunnelling over ssh and other mechanisms then using a jump box/bastion host seems like a no-brainer.

The one thing that is a bit awkward with ssh is actual monitoring - but most of the things that could be done to address this also risk the CIA of the channel.

In the past I've used a web based VNC session to a jump box (not very good or running orchestration from your local) currently use ssh tunnelled through ssh and am looking at ssh tunnelled via AWS SSM (which might be implemented without jump boxes).

[-]

bendem@reddit

I don't see guacamole mentioned yet.

https://guacamole.apache.org/

[-]

myrianthi@reddit

NinjaRMM