IP forwarding differences between Amazon Linux 2 and RHEL9
Posted by a-sad-dev@reddit | linuxadmin | View on Reddit | 17 comments
Hi, I've been migrating from AL2 -> RHEL9 in our AWS EC2 environment and one issue I'm coming across is switching the AMI from AL2 -> RHEL9 is causing IP forwarding issues on our proxy VM's.
The instance in question that's being replaced is working as a squid proxy and is the default route for the subnet it resides in (technically an ENI attached to the VM is the default route).
The process in question is VM1 is attempting to connect via SFTP to an external endpoint on the internet and traffic is routing through VM2 which is running as a proxy VM (squid for HTTP traffic).
All non HTTP traffic should transparrently flow through the machine which is the case with AL2 but switching to RHEL9 causes the connection to drop.
So far I've checked the following:
- iptables rules for port forwarding as well as NAT tables (identical on both machines)
- ran cat /proc/sys/net/ipv4/ip_forward on both machines and both return 1 (ip forwarding enabled)
- SELinux set to enabled, passive and disabled - has no affect either way
- Squid settings identical (don't think this will matter for sftp on non http port)
- All routing settings and security groups are unchanged in AWS - only thing swapped out is base AMI
- No entry in squid access log for SFTP connections
To test I run an sftp command from VM1 and with AL2 squid VM the connection succeeds, with RHEL squid VM the connection hangs. Am I missing something obvious here? Any other areas I can investigate?
Kind of running out of ideas, thanks for reading and I hope it makes sense.
Flow__9374@reddit
Hey OP, sorry to hear you are running into issues and I can’t offer much help. I am curious though what made you and your team decide to move from AL2 to RHEL? We’ve been having similar discussions so curious to hear any comparisons, research, pros/cons, migration considerations, etc that we may not have considered. Thank you and best of luck with the migration!
a-sad-dev@reddit (OP)
The issue in the end was the priority of the interfaces. The VM had the default network interface and an additional ENI attached during cloud-init. The traffic was arriving on eth1 (the additional ENI) and the OS was attempting to forward traffic on eth0 (default system interaface) which was causing the issues. I set priority of eth1 as highest and voila, started working!
As for the decision to move, we are currently supporting a legacy java 8 application that is stuck on java8 for... reasons. RHEL had the longest support timeline for java8 up until last week when AWS announced that they are extending the support of java8 from 2026 until December 2030 so that's made the RHEL migration potentially a waste of time. I initially suggested Alma as a free alternative but my team lead wanted RHEL for the enterprise support. I don't pay the bill so I just went with it.
Admittedly I'm not an expert with the Amazon Linux / RHEL / Fedora / CentOS etc distro orgy and I'm always open to hearing alternative opinions.
a-sad-dev@reddit (OP)
For anyone who cares: the issue was the priorities on the interfaces.
On the trouble machine traffic was arriving on eth1 which had a lower priority than eth0 so the machine was attempting to forward traffic on eth0 which failed. I switched priorities with
ip route addand the issue was fixed.Thank you all for your ideas / troubleshooting help!
shrizza@reddit
Track the packet at each relevant interface and compare before vs after.
a-sad-dev@reddit (OP)
Thanks for the reply. I ran TCP dump filtering for the source IP and port, I can see the SYN packets arriving at the squid server when I run the sftp command but nothing else. It's as if VM1 is initiating the TCP connection but receiving no response.
onemadriven@reddit
Do you see the traffic leave the squid VM? Check if you can comnect to that SFTP server directly from the squid server.
minimishka@reddit
It seems that nftables lives its own life
pnutjam@reddit
Did you configure the networks with Network Manager? I'd check the config files in /etc/sysconfig/network-scripts/
spydum@reddit
Running firewalld? You mention iptables NAT rules, but rhel runs firewalld.. I could imagine if it's being dropped before NAT, you would see those SYNs but not response.
a-sad-dev@reddit (OP)
Firewalld isn't installed which is odd. Perhaps because it's AWS AMI of RHEL9? I've not made any networking changes to the machine besides the squid changes to forward HTTP ports to the squid proxy.
frymaster@reddit
it might be worth checking both
iptables-save(dump iptables rules) andnft list ruleset(dump nftables rules) output for both serversGamerLymx@reddit
this, check firewall config on both machines, make sure the configs are correct and are persistent.
if you configure iptabkes, but have an engine like firewall running, this eill kot work properly.
jaymef@reddit
Check if you have source/destination check enabled for the new EC2 instances
a-sad-dev@reddit (OP)
Settings are the same for both, everything is deployed using terraform so I know the only thing being changed out is the base AMI.
No_Rhubarb_7222@reddit
You can also open a support case.
You might also look at the systemwide cryptography policy. RHEL 8 and 9 implemented this and it can affect connections in the way you describe. Essentially, it’s a system wide control on what cryptography methods are allowed on the machine and the default settings disable older protocols and ciphers. You might try setting your policy to Legacy. RHEL9, specifically removed some of these older methods as well, but IIRC it was older SSL based stuff, not things used by ssh.
Here’s a hands-on practical introduction to system-wide crypto policy:
https://www.redhat.com/en/interactive-labs/configure-system-wide-cryptographic-policy
Though it’s straightforward to change and the rh docs work as well.
a-sad-dev@reddit (OP)
Legacy ciphers made no difference I'm afraid.
a-sad-dev@reddit (OP)
Thanks for the reply, I'm not looking for a lab at the moment but will bookmark for the future as it looks interesting.
I'll attempt to enable old protocols and ciphers and see if that works, cheers!