TCP Flooder Bots
Posted by Smooth_Security4607@reddit | linuxadmin | View on Reddit | 29 comments
I don't know if everyone else is experiencing this phenomenon or what. My server is being flooded by TCP connection bots. At first, it seems like they are just the normal annoying scanners that are going to check for open ports and then go away. However, once they find an open port. more and more of them show up until it's thousands of them. Some of them connect, and hold the TCP port open as long as possible. Others just connect and disconnect quickly (but thousands of them). This prevents all of the services on that port from being available.
For example, I am building a simple LAMP application with website and database, all on one server. Since I would connect to the database from my home IP, I let it accept connections that were not local.
One day, my application is not working. I check and it can't connect to the database. I check the database and all the connections are taken up by these bots. I firewall off everything but my home IP from that port.
Then, the website stops working. Apache is configured for 512 connections and they are all taken up by these bots. I moved everything to a different port temporarily.
This application isn't even public yet and has nothing visible without logging in. There is no reason they'd be targeting me in particular.
I guess I will have to put the final website behind a proxy service like cloudflare. But amazing to think you can't leave any ports open anywhere these days without being flooded. A lot of the bots are from Russia and China so maybe it's a state actor thing.
Sad_Dust_9259@reddit
Happens all the time now. Any open port gets hammered fast. Lock it down and proxy everything, no other way for me.
Smooth_Security4607@reddit (OP)
This seems to be the only way
Sad_Dust_9259@reddit
Yeah, maybe there's some way. Let's wait for more comments.
nanoatzin@reddit
Sounds like a brute force password guessing attack. This kind of thing corrupted an old server several years ago. Install Fail2Ban. It automates the firewalling process. Fail2Ban will monitor logs for SSH and SFTP. It needs to be configured for other things, like database logins.
Smooth_Security4607@reddit (OP)
They don't get as far as guessing a password because they don't even negotiate TLS with the HTTPS server. They just connect and hold the port open forever.
nanoatzin@reddit
This fellow configured Fail2Ban to block addresses originating too many dropped packets. Thanks for sharing this issue because I need to adopt it.
Jail.local
[iptables-dropped]
FILTER: iptables-dropped.conf
Make sure you log the dropped IPs like this in the * iptables rules so the above filter works:
*’# log iptables denied calls (access via 'dmesg' command) to /var/log/messages file * iptables -N LOGGING * iptables -A INPUT -j LOGGING * iptables -A LOGGING -m limit --limit 5/min -j LOG -- log-prefix "IPTables Dropped: " --log-level 4 * iptables -A LOGGING -j DROP
chock-a-block@reddit
Never leave a database listening on the internet. If the database is on the same host as your app, use a socket and disable TCP listening.
If this is what you say it is, you should be blocking whole IP ranges at the firewall.
fail2ban will eventually be your best friend. But, you are clearly in over your head, so, not sure I would start there.
GreatNull@reddit
Supplemental to point 3., it not workable protection for even small real ddos attacks. If it works, you are dealing with amateur or very small scale operation.
Attacking control server will react in near real time to ip range or geoblocking, we saw response time in sub 2 minutes to that.
chock-a-block@reddit
I have so much respect for people that have to respond to these attacks. Good job!
Smooth_Security4607@reddit (OP)
I firewalled off the database port to anything but local connections and my home IP.
chock-a-block@reddit
Use a socket for that.
GreatNull@reddit
I am just devops monkey runing backend cluster that gosted app under attack and my this was my first real ddos in my career.
Whoever paid for 24hours of attack, they got their money worth out of it. 95% chance it was russian ordered state op, since we are " official hostile country to russian federation".
We have contracted isp level filtering service since that incident, it wasnt practically defensible at our scale.
Dynamic blocking per most frequent source ip ? Pointless.
Geoblocking ? Also pointless, attack source shifted within 5 minutes of block being placed.
Sources were residential (i.e indistinguishable from real world users, beyond geographical location) and commercial VPS providers.
Smooth_Security4607@reddit (OP)
Yes I am going to have to use cloudflare or something for the public site once it's running. I am seeing the same kind of IPs you were. They seem to just connect and hold the port open, they don't try to even negotiate TLS much less try to log in. I don't think they were targeting me in particular, there is nothing to indicate what this website is for, and it's just for a small nonprofit anyway.
nathacof@reddit
Who's gonna tell em?
nathacof@reddit
Open websites, especially those written by amateurs are often prone to root kits. It's very possible someone has used a bug in your code to install a back door. Backup your website files, stop the we servers, and start with a fresh image, while following STANDARD security protocols. Just Google it.
Smooth_Security4607@reddit (OP)
The site is password protected since it's under development. They would not be able to access it far enough to install any rootkit. If they could root this server, they could root anything running Apache. And it doesn't seem like that is happening.
thoriumbr@reddit
I have a small server running on the public internet for almost 15 years, and faced some attacks over the years. Today the server is mostly quiet because I put several protections on it.
First, install and configure portsentry. It blocks portscanners, keeping most of the random internet background traffic away. Just take care to not block yourself.
After that, fail2ban. It keeps a look on syslog, Apache logs, and anything that produces an error log. 4-5 errors in a row? Block. And again, whitelist your IP or you will end up locking yourself out. Changing the lockout duration to something like 15-30 minutes is enough, and you can increase that later when everything is working and you can work on the system without locking yourself out again.
Next, change SSH default port to something else, and keep port 22 on portsentry rules. Anyone connecting 3 times on the wrong port gets blocked. Some bots will open several connections at once to speed up bruteforce, and that will kill their attempts. Do not use something like 1022, 2222 or 22222. Use a real random port 40000+.
And geoblock. It's a private server, so I can control who can access it, and if China and Russia aren't the country where my clients are, I am sorry but my firewall won't let you in. Spurious traffic dropped dramatically after that. Block China, Russia, take a look on your logs and block whatever country keeps attacking you.
Everything that don't need to be accessed by the public listens only on localhost. So MySQL databases, transmission-daemon, Elasticsearch... only Apache and SSH are on the open.
Change Apache default settings, specially KeepAliveTimeout. Change it to 2 is enough, because if a client takes 2 entire seconds after receiving data before asking for more data, kill the connection.
And disable SSH password-based authentication, use only key-based authentication. Some bots will find your random SSH port, and some will give up when they get the SSH message saying password authentication isn't enabled.
If the load is too high, make your firewall send back a Connection Reset to everybody except your whitelisted IPs for 5 minutes. Your server isn't accessible anyway, killing all connections will make the bots go away.
I would say "call Cloudflare" like the others, but part of the learning process is to face problems and overcome them. So fight the bots for as long as you can, as hard as you can, and only call Cloudflare when you are losing money, or you are ready to give up and concede the bots won against you.
Smooth_Security4607@reddit (OP)
All great advice, thank you.
Whoa_throwaway@reddit
welcome to the internet, you want to feel like it's "you" and they're targeting you in particular but they're not. You're not that special, most of us aren't. bots and scanners find it and just poke and prod.
Smooth_Security4607@reddit (OP)
Yes there is no reason for me to be targeted. It seems like this will eventually happen to anybody who leaves "interesting" ports open.
michaelpaoli@reddit
Dig more closely into what's actually going on. Could you be getting attacked/flooded like that? Possibly. But unless you're a relatively high(er) value target (or mistaken for such), they're generally not gonna bother, and you mostly get the random doorknob jiggle and other more common mundane annoyances.
Most probably you don't have things tuned properly. E.g. many years (decades) ago, had a host that was crashing ... because it was getting overwhelmed by some bad bots, ... wee bit of tuning on the Apache side (the default was allowing excess resource consumption relative to what the host actually physically had), and ... bye bye problem. And, over the years, have likewise made issues from bad bots go away with similar counter-measures, e.g. adding CAPTCHA on a self-service registration page ... yeah, I really didn't need thousands of bots registering their own accounts - put an end to that. Etc. Anyway, had web server (and mail server, and list server, and wiki, and wordpress, and yes, even public ssh server) open to The Internet for decades ... and ... occasional bit of annoyance to be dealt with once in a great while ... and that's mostly it.
And yes, public ssh server, have a peek at:
https://www.wiki.balug.org/wiki/doku.php?id=system:what_is_my_ip_address
It's among the servers listed offering public Internet accessible ssh. Oh, yeah, and fail2ban ... that made the logging of failed ssh attempts way more quieter ... used to be dang annoyingly loud when the bots would hit that with ye olde spinning rust drive ... yeah, solved that issue decade(s) ago - much quieter ever since - literally and figuratively.
Smooth_Security4607@reddit (OP)
I am not a high value target at all, it's just for a small nonprofit project and there isn't even anything to indicate what the site is for yet (no domain pointing there even).
I can definitely see a ton of IPs that were first connecting to the database and then after I locked that down, connecting to the website. They don't even try to negotiate TLS, they just sit there and hold the port open.
I have written web apps a while ago and never had these problems, however once they were public it did use cloudflare to access the website, and a VPN to access anything within the private network.
bishakhghosh_@reddit
It is very common particularly for popular ports such as for your database. Always use IP whitelisting in your firewall or use a ssh tunnel or something similar to connect different components of your application. Only keep the web server (reverse proxy like nginx) open.
deleriux0@reddit
So of course you should not expose any port on the internet that is not meant for public consumption, so firewalling off things you shouldn't be exposing really is a must.
As for the Apache service filling up, whilst basic TCP / slowloris attacks are a thing I would be quite surprised if that is what is going on.
I'd be inclined to check the speed of your web application (is it taking 10s of seconds to do something) and you are being caught up in that.
Perhaps check the number of hits in your access logs and log the time taken to process whatever the request is.
Basically I would be checking your own plumbing for leaks first before complaining about floodwaters from outside.
Smooth_Security4607@reddit (OP)
Thanks for the advice. Even when I'm not using the application (it's only in testing and password protected, so I would be the only one using it), the ports are still completely flooded. Both the DB ports and then the HTTP ports.
nuttertools@reddit
As you aren’t going for production right now consider using an SSH tunnel to access your development stack and don’t allow any other ports.
sleuthfoot@reddit
I'm surprised that you're surprised. Anyone should know that if you put up an internet-facing server, it's going to be inundated with all kinds of malicious shit.
mysterytoy2@reddit
Try dropping ping responses. They might think your server is down.
Expensive-Soft5164@reddit
Do not open up that MySQL port to the world. Instead use an ssh proxy, passwordless.