Managing consistent network access controls across a hybrid Linux fleet is becoming unsustainable and I am wondering if ZTNA is the right direction here
Posted by Unique_Buy_3905@reddit | linuxadmin | View on Reddit | 22 comments
Running around 200 Linux servers spread across on-prem bare metal, two AWS regions, and a small GCP footprint. For years we managed access with a combination of iptables rules on each host and security groups at the cloud layer, which worked fine when the environment was simpler.
The problem now is that maintaining consistent network segmentation across all three environments means keeping rules synchronized across host-level firewalls, AWS security groups, and GCP firewall rules simultaneously. We are already using Terraform for provisioning the cloud security groups but the consistency gap between the IaC layer and host-level rules during runtime changes is where things break down. When something changes urgently, it changes in three places and there is no reliable way to verify those three places are in sync at any given moment.
Started looking at whether pushing access control up to a dedicated network security layer makes more sense than maintaining it at the host level, and zero trust network access keeps coming up in that research. Most of what I find is aimed at office environments managing user access though, not infrastructure teams managing server-to-server traffic across a hybrid fleet. Any of you folks applied ZTNA principles to this specific use case and found something that actually fits? Appreciated.
Hot_Blackberry_2251@reddit
ZTNA as a category is built for user-to-application access.
Server-to-server traffic across a hybrid fleet is a service mesh problem not a ZTNA problem.
redundant78@reddit
this is mostly right but there's a middle ground - overlay networks like Tailscale or Nebula give you identity-based access control for server-to-server traffic without the overhead of a full service mesh like Istio/Linkerd. you get the "single policy source" benefit OP wants without having to sidecar every service. for 200 servers across 3 environments that's probably the sweet spot between "just fix your iptables sync" and "deploy a whole mesh."
PhilipLGriffiths88@reddit
Fair point, but I’d still draw a distinction. Tailscale/Nebula feel more like identity-aware secure overlays than a truly identity-first, service-centric model. They improve reachability decisions with strong machine identity, which is useful, but the construct is still largely “which nodes can talk on the overlay?” rather than “which specific service is dark by default unless identity + policy explicitly create the connection?”
For OP’s use case, that difference matters. The problem isn’t just simplifying network policy across bare metal + AWS + GCP, it’s enforcing non-human/server-to-server least privilege without recreating broad ambient reachability inside a cleaner private network.
So yes, those platforms may be a pragmatic middle ground, but I’m not sure they fully solve the end-state OP is pointing at if the goal is authorize-before-connect at the workload/service layer, not just a simpler overlay with ACLs.
Unique_Buy_3905@reddit (OP)
That's a useful redirect. Which service mesh implementations have you seen work across a hybrid fleet spanning bare metal and two cloud providers, not just cloud-native environments where the mesh tooling assumes you control the underlying infrastructure?
PhilipLGriffiths88@reddit
That’s exactly why I’m skeptical of “just use service mesh” here. Most meshes shine intra-cluster, not across a mixed estate of bare metal + multiple clouds.
Your problem is really consistent policy for non-human traffic across heterogeneous environments. That points less to classic ZTNA for users, and less to pure service mesh, and more to identity-defined connectivity for workloads/services across the whole fleet.
In other words: one policy plane for “who/what can talk to what,” instead of trying to keep iptables, AWS SGs, and GCP firewall rules perfectly in sync.
Now, I am biased, but I would suggest NetFoundry (if you want paid product) or open source OpenZiti (which is built and maintained by NetFoundry. Its designed for identity-first, Zero Trust connectivity both North-South and East-West, and provides an abstraction on the underlay to remove the 'connectivity tax' (i.e., no need to synchronized host-level firewalls, AWS security groups, GCP firewall rules, ACLs, inbound FW rules, NAT, public DNS, VPNs, load balancers, etc).
PhilipLGriffiths88@reddit
Unless the ZTNA platform can handle non-human workloads as easily as human (some can, as you say, most do not).
chickibumbum_byomde@reddit
i wouldnt panic, seems normal, host firewalls plus cloud rules don’t really scale well in hybrid setups.
ZTNA can sure help for user access, but for in between servers traffic you’re better off with central policy and some identity based access, not rule synchronisation everywhere.
i would also add some reliable monitoring, otherwise drift will happen no matter what approach you take, it will keep things in check and only notify if sth is off, keeps me asleep at night, hehehe.
YOLO4JESUS420SWAG@reddit
I'm a fan of defence in depth so personally I would not pin all this to a single point of failure.
Set up Ansible and run a job on a schedule to check Amazon via aws cli, giving your instance an IAM role with perms to check that. Then remote into your network devices an finally your endpoints. Check for what you want and change where you need or alert when it's not up to your requirements.
Not sure what your SLA is but alerting may be smarter than changing in case a change impacts uptime.
Unique_Buy_3905@reddit (OP)
The drift detection approach makes sense for catching inconsistencies after the fact. The problem we keep hitting is urgent runtime changes that diverge from IaC state before any scheduled job catches it.
What's your detection-to-remediation loop look like when something is actively broken?
RetroGrid_io@reddit
Sounds to me like you're naming the problem just fine - just not recognizing it as such. What can you tell me about these "urgent runtime changes"? The kinds of things I want to know:
YOLO4JESUS420SWAG@reddit
I leverage puppet/chef for IaC separate from ansible. While ansible can be used for that purpose, ssh is something that can be misconfigured rendering ansible in its traditional form useless. Puppet/chef are great to test your changes in one environment, then role out changes to the rest via a single code change in one place. We set up puppet-agent to pull every 30 minutes.
Unique_Buy_3905@reddit (OP)
The 30 minute pull cycle handles configuration drift well but the harder problem is an urgent manual change that breaks something and stays broken until the next convergence window closes it.
YOLO4JESUS420SWAG@reddit
> Puppet/chef are great to test your changes in one environment, then role out changes to the rest
You need a test environment. Or just increase your pull rate.
Unique_Buy_3905@reddit (OP)
Fair point. Increasing the pull rate is the simpler fix we probably overlooked before going down the ZTNA research path.
NeverMindToday@reddit
Years back I worked somewhere we managed up to 1000 Linux machines (mostly KVM and EC2 VMs, a few LXC containers, and some bare metal colo hosts for the previous ones).
We used a custom config management codebase that acted a bit like Ansible (idempotent, stateless, agentless push but faster). Servers had a collection of roles (purposes, environment, location etc) assigned, and that determined what the firewall rules would be. Rules were applied to hosts, and either configured AWS security groups or network hardware via API calls.
Everything was treated as redundant livestock rather than pets, and haproxy healthchecks made sure canary changes were successful before automatically moving on with wider deployments.
Nothing was configured manually. The whole system evolved over years and got more and more sophisticated over time.
These days Terraform and Clouds etc have some advantages, but I miss operating that direct flexible targeted system.
Tricky-Cap-3564@reddit
The sync gap between IaC and runtime is a drift detection problem not an architecture problem. AWS Config rules, GCP Organization Policy, and something like Driftctl or Terrascan against your host-level iptables gives you continuous verification that what Terraform provisioned matches what's running.
That's a significantly smaller project than rearchitecting access control across 200 servers and it directly addresses the specific failure mode you described without adding a new network layer.
Special-Cause7458@reddit
Consul Connect with mTLS handles service-to-service authorization across bare metal, AWS, and GCP natively.
Identity is certificate-based per service, policy is centralized, and enforcement happens at the connection level without touching iptables or security groups.
It's exactly what you're describing and it's built for this use case specifically.
Unique_Buy_3905@reddit (OP)
Agreed, certificate-based identity per service and centralized policy without touching iptables is exactly the gap we're trying to close.
Has it held up across bare metal specifically or mostly in cloud-native deployments in your experience?
420GB@reddit
I believe this is why some people deploy cloud VMs of their firewall platform of choice.
E.g. a virtual Palo or Fort gate.
You can use the same central management for all environments and share objects, policies, profiles etc. between them all. Also makes VPNs and troubleshooting easier and all logs are in the same format.
Unique_Buy_3905@reddit (OP)
That approach hadn't come up in our research. The operational overhead of managing the virtual appliances themselves across bare metal and two cloud providers is the part worth understanding before going that direction though.
mike34113@reddit
The consistency problem across three enforcement layers is fundamentally a policy distribution problem.
Cato networks connects on-prem and cloud environments through IPSec tunnels to the same PoP infrastructure and enforces network segmentation policy from a single control plane rather than synchronized rules at each host and cloud layer separately. Changes propagate from one place.
The three-way sync problem doesn't exist because there's only one authoritative policy source regardless of where the traffic originates.
Unique_Buy_3905@reddit (OP)
Single control plane is the right direction but curious how Cato handles host-level enforcement on bare metal that isn't routing through the PoP. The cloud security group sync problem is clear, the bare metal piece is where that model gets complicated.