what are you using for an automation/orchestration platform?
Posted by baconwrappedapple@reddit | linuxadmin | View on Reddit | 30 comments
I'm looking for more detailed answers than "puppet" or "ansible"
What do you use as a source of truth for inventory that the system works against? how do you dynamically maintain the inventory system?
Do you have a GUI layer on top of it?
How many machines are you managing?
Do you use more than one tool? if so which tool manages what aspects of each system?
J4yD4n@reddit
Automation: Ansible. It does a great job at coordinating actions across multiple machines and platforms.
Central Configuration Management: Puppet. It is extremely efficient at ensuring all machines are always at the configuration you want. It also includes environments so you can roll out your changes as slowly and safely as you want.
Source of truth: Satellite/Foreman. It can mirror all repos (rpm, deb, docker, etc) so nothing needs direct internet access and includes both Ansible and Puppet so everything can be done from one place. Playbooks, roles, classes, and modules are all managed through git for history.
GUI: Only the Satellite/Foreman web UI. No GUIs on any servers and no web management installed on any servers.
I managed a couple hundred servers this way. However, "manage" may not be the right term. Due to everything being setup so robustly, not much ever needed to be touched other than quarterly patching, check in every so often just for peace of mind that nothing was failing and not emailing me the error, and the occasional deployment/decommission.
sdjason@reddit
Are you my coworker?? This is literally us, but with around 15k total VM's
Only thing I'd add is puppet, and ansible, 100 percent pull all code from git at all times, and in lieu of awx inventories or official integration with foreman in AWXwe've settled on custom roles that quickly "create" dynamic inventories at runtime in AWX using Foreman's rest API. Could never get the official one to work, likely API timeouts and just gave up
Shot_Culture3988@reddit
Using Puppet and Ansible gives a lot of flexibility, but man, dealing with dynamic inventories in AWX with Foreman's API can feel like trying to make a toddler sit still. I remember those Foreman integration woes – every timeout made me question my sanity. Scrapped the official integration too, and added APIWrapper.ai to help with API management alongside my custom roles. It’s less of a hair-pull that way, but still, good times… not. Tried Platform.i0 too, which was meh.
pnutjam@reddit
We have some minor changes that need to be done to our inventory so I can't actually pull from Satellite directly to Ansible.
I set up a "project" based inventory that pull inventory from a file in git.
Basically cron runs a script that goes into the git repo directory, pulls git, then pulls everything from satellite, makes the modifications and sorts into a couple different inventories based on location. Then the script pushed it up to git and makes an api call to ansible to force the project to resync.
The inventories are set to sync on run so they update as long as the project is synced.
Shot_Culture3988@reddit
I've had a somewhat similar experience. We also had to come up with workarounds for handling dynamic inventories. We integrated Jenkins to automate a script run that fetched inventory details from Git, and then sync that into Ansible Tower, which isn't that different from what you’re doing with the cron job. It’s cool to see how flexible these setups can be, but it definitely has its quirks.
For API management and integrations, tools like DreamFactoryAPI and others can be pretty handy, particularly when rest APIs are involved. Once set up, these can smooth out operations quite a bit.
J4yD4n@reddit
I don't think I'm your coworker. I didn't manage 15k machines.
100% agree, store it all in git.
I didn't use AWX so I didn't have to deal with that inventory issue, but good to know the official dynamic inventory integration has issues
regorsec@reddit
Foreman / Puppet / Git is the way
nikster77@reddit
We run nearly the same setup, git repos are our source of truth. Puppet for config mgmt., Foreman for image deployment, etc. (no repo mirrors though, but will check this out now). Ansible for autom. Tasks. We run a couple of thousands servers.
baconwrappedapple@reddit (OP)
I didn't mean a GUI on a server but front end to your management tool like AWX or SemaphoreUI.
You end up running your ansible playbooks using foreman?
J4yD4n@reddit
The Ansible playbooks were simple enough that running from Foreman was all that was needed. If I had needed to interact with something that wasn't attached to Foreman, something else that could run the Ansible playbooks would probably be needed, but I also had some PowerShell scripts running as cron jobs that did my daily error checking and cleanups outside of Foreman. PowerShell is a good cross platform scripting language that I know, but you can use whatever language you like. It could have possibly been done in Ansible, but it would have been extremely ugly compared to the simplicity of the PowerShell scripts.
Limp_Departure8138@reddit
I'm looking for more detailed answers than "puppet" or "ansible"
-- Ansible. If you ever used ansible or managed linux systems you'd understand it's all there. Are you trying to find an easy button that doesn't require a lot of learning?
What do you use as a source of truth for inventory that the system works against? how do you dynamically maintain the inventory system?
-- Ansible inventory. Both static and dynamic inventory using an aws plugin based off AWS tagging.
Do you have a GUI layer on top of it?
-- No. If using something like Tower/AWX, you still have to learn how to write in ansible. This requires learning and time. Especially if you don't have any programming experience.
Do you use more than one tool? if so which tool manages what aspects of each system?
-- Yes. Also use git to maintain the ansible code and to make changes. Also use ansible-vault to encrypt sensitive information. Again, the tool has everything you need for orchestration and automating linux machines.
mestia@reddit
Not many machines, but i am pretty happy with Rex - rexify.org for the automation, ocsinventory for the inventory and zabbix for the monitoring.
GuessSecure4640@reddit
PDQ
PudgyPatch@reddit
Automation and config management: WE use ansible for OUR stuff but somethings are managed by another department and they use puppet. Eg. Our packages our config(for stuff that other groups wouldn't use) where puppet is used for users FW and system wide updates (packages that fall out of our custom stuff) We use netbox for inventory for our playbooks.
LevelHQ@reddit
Does it need to be open source/free? If not, an RMM like Level.io is a strong fit because it combines remote access, automation pipelines, real-time monitoring, and centralized inventory in one platform. You get full visibility into your systems and their current state without needing to build and maintain your own stack. You can orchestrate scripts across servers, automate updates, monitor services, and get alerted when something breaks. It’s purpose-built for exactly what you're describing.
dhsjabsbsjkans@reddit
Ansible cli and ansible automation hub. We manage 1000+ machines. Inventoroes are static and dynamic. We automate tasks on both Linux and Windows. We use it for automated server builds, both VM and physical. We use it for monthly patching. We use it for software deployments, etc.
nitroman89@reddit
I use the Uyuni Project for centralized management for patching, source of truth and state configurations like setting snmpd etc.
I use Ansible with Semaphore UI as the frontend that connects to my Gitlab server where I've made repositories for each of my projects that handles my various roles/playbooks which is mostly used for my adhoc tasks like taking VMware snapshots and rebooting Docker Swarms etc.
Hotshot55@reddit
How is it? I imagine it's not much more than Spacewalk ever was?
nitroman89@reddit
I never used Spacewalk but I've used Oracle Linux Manager so yeah it's pretty much the same. It's built on top of using salt and minions so there's extra features you can utilize.
nitroman89@reddit
I never used Spacewalk but I've used Oracle Linux Manager so yeah it's pretty much the same. It's built on top of using salt and minions so there's extra features you can utilize.
ryebread157@reddit
Puppet + puppetdb is a solid inventory, can easily query it to form ansible inventories
baconwrappedapple@reddit (OP)
what do you use to query it?
ryebread157@reddit
They have a well documented API, can just query it with curl
jt-atix@reddit
orcharhino:
- it is based on Foreman but with support and proper release tests to make it stable - compared to red hat satellite it also supports the other linux distributions (Alma, Rocky, RHEL, Oracle, SLES, Ubuntu, Debian)
- it also has an integration for puppet, ansible, salt and with other plugins you can get more value out of it like OpenSCAP-reports
- you can use it also for provisioning, inventory, configuration management with windows if this is not your main-os
- depending on the infrastructure there are instances managing thousands of hosts
- it can be used via GUI, CLI, API, so there a possibilities to automate a lot while having the GUI as a good overview over host statusses (open errata/patches, errors during configuration runs, incompliant with openscap-policies, ...)
SuperQue@reddit
So if you really want to know, we basically don't manage machines anymore. It's almost entirely Kubernetes now.
We manage base infrastructure with Terraform and some other internal tools. This bootstraps the Kubernetes platform. Services are deployed using CI/CD tooling.
Everything is in git.
GitHub, with a side of CI/CD and Grafana.
We don't think in terms of machines/VMs anymore. Individual nodes come and go all the time.
It's all about resources. A typical cluster will be 10s of thousands of CPUs and TiBs of memory.
fubes2000@reddit
Terraform to provision and manage infrastructure in AWS, Ansible to configure services on provisioned infra using the EC2 dynamic inventory config. Depending on what/where you're deploying to there's probably a dynamic inventory plugin for that.
Inventory management was a pain in the ass with just Ansible, because Ansible just does not want to be in charge of provisioning anything.
If I had to roll everything from scratch again I'd probably either have Terraform invoke Ansible for me, or bake Ansible to the image and something something cloud-init yadda yadda userdata. But historically we ran Ansible centrally, and I hadn't had the time or wherewithal to evolve past that.
Your particular flavor of deploy/config management probably vary based on your requirements and what you're actually deploying.
skreak@reddit
I work in HPC - we have a few thousand bare metal systems, a few large storage frames, and multiple isolated networks. We're currently migrating to Device42 for datacenter management tools from an old opensource project called Racktables. We're also a Puppet house so configuration is all driven by that. Our company uses ServiceNow for asset management (and tickets, etc) but Device42 is our IPAM, floor plan, and detailed asset tracker for HPC.
baconwrappedapple@reddit (OP)
Do your automation tools for managing servers talk to device42 or servicenow?
There are a lot of different tools that you end up needing and I'd like to link them back to one source of truth.
invalidpath@reddit
AAP, and Cloudbolt. Inventories are straight from vCenter and AWS. Managing a few hundred hosts for now.
zapman449@reddit
Every inventory I’ve ever used sucked…
Until I started using AWS… then it didn’t matter any more. Because everything was AMI driven and autoscaled.
And the very few things which couldn’t scale, we made an ASG of size 1 with a remounted EBS volume.
Once you can get to that state (in a DC or in cloud) inventory mgmt goes away as a problem (or radically reduces)