PSA: if your web application is getting much higher traffic than you think it should be be aware about AI trainers
Posted by IRIX_Raion@reddit | linuxadmin | View on Reddit | 36 comments
These didn't really bother me up until recently where they basically started hammering on the server for over 780 CPU seconds on average for a small size forum.
I don't understand how they can get away with doing this on small scale sites. The only reason that this sort of thing wouldn't have killed it is because I heavily cache my forum. I don't understand how they can get away with doing this on sites that don't have people who have been doing this for years and know how to adjust things properly. I went from that and burning out one of my chorus constantly to 60 CPU seconds once I blocked their IP ranges and did some other adjustments to reduce CPU on the memcached service.
binaryflow@reddit
I moved our sites behind Cloudflare and turned on their AI blocking. It made a huge difference. The AI bots will consume every cpu cycle they can find and think nothing of it.
IRIX_Raion@reddit (OP)
Cloudflare is a pragmatic option for most people; but I also don't want them to have a monopoly on the web so I have chosen to do things my own way.
binaryflow@reddit
I would love to hear what you come up with. I’m always looking for alternatives.
IRIX_Raion@reddit (OP)
haproxy or Anubis is what I'm looking at.
Reasonable_Ticket_84@reddit
AI bots have workarounds for Anubis now.
Codeberg beset by AI bots that now bypass Anubis defense • The Register
sudonem@reddit
Same. Definitely recommend.
V2UgYXJlIG5vdCBJ@reddit
Look into AI poison and other weapons. https://blog.cloudflare.com/ai-labyrinth/
IRIX_Raion@reddit (OP)
Anubis and haproxy are two things I've looked at potentially using. My case is very specific though in that I must figure out how to configure them so that old browsers don't get caught (like Mozilla 1.0, Netscape 6, Net positive Etc )
V2UgYXJlIG5vdCBJ@reddit
User agent whitelisting. Something like that.
ITaggie@reddit
Most of these bots use randomized user agents. Once they find one that works, they'll just use that.
elprophet@reddit
They do not care about you, your forums, your users, or literally anything.
Academic-Gate-5535@reddit
And it's the internet, DDOS is only illegal because it uses compromised systems and has intent.
Scraping the whole internet, there's nothing for that
ITaggie@reddit
Well depending on how it is done it's debatable. When they start using exploits and/or obvious botnets to bypass controls (which happens often) then it does violate the CFAA (in the US), but obviously if it's from a country that is not friendly to your country's justice system then you just can't do anything but block them.
IRIX_Raion@reddit (OP)
I'm more so saying that I don't know how there's not dozens of websites falling all over themselves keeping up with this level of traffic. Not everyone is using such aggressive measures to keep CPU and memory load down especially if they're using docker images or these other one size fits all solutions that have become popular with this current generation. Not that there's anything wrong with that; but it does beg the question. On one hand I am flattered that they find my form useful for training AI; on the other I wish that they had some way to rate limit it or a way for us to opt out
bityard@reddit
I guarantee that no humans behind the bot's actions looked at or even heard of your site. They are just following links from one site to another and vacuuming up literally everything they find.
IRIX_Raion@reddit (OP)
Only reason I say that is because it's one of the authoritative sources for a very niche topic.
kuzared@reddit
From what I gather, lots of small websites are having problems.
Don’t be flattered, it’s all automated, they’re ingesting everything they can get their hands on.
Poesximah@reddit
This has become a norm but i usually implement f2b, rate limiting, caching to reduce CPU per request and clouflares WAF( if its in clouflare).
serverhorror@reddit
The food news, this sets a nice precedent that web scraping is now OK, regardless of terms of service.
anomalous_cowherd@reddit
Only if you're huge or judgement-proof.
archontwo@reddit
Shouldn't fail2ban and nginx rate limiting work to
dissuadebot farms?IRIX_Raion@reddit (OP)
I don't use f2b. Different software stack
ITaggie@reddit
You're going to need some sort of WAF and an Analytics Dashboard to keep up with this, since you don't want to use CloudFlare. I respect your reasoning, but that decision is going to make this a far more active effort than it otherwise would be-- just wanted to be clear with you.
The most reliable method I've had to mitigating these types of attacks is by blocking the "problem ASNs" that pop up on the dashboard. 90% of the time these will be owned by public cloud providers and shouldn't affect normal residential/mobile internet traffic.
Here is my current list of "completely block" ASNs, though this is very far from comprehensive and excludes all our WAF middle-layers to filter out other malicious or bot traffic:
136907
4134
21859
55990
62654
396982
141039
207990
20473
210906
132203
36352
IRIX_Raion@reddit (OP)
I managed to block two IP ranges, one is MS Azure (OpenAI) and Deepseek (Softlayer, Singapore) and that fixed the problem for us. I also set up in naemon some checks that allow me to identify such traffic. Anubis and haproxy are being investigated for more long term solutions
ITaggie@reddit
Use that website I linked in that list to look up offending IPs and see how many of them share the same ASN. You can then convert these ASNs into "Block Lists" of various formats with this tool: https://www.ip2location.com/free/visitor-blocker-asn
They will be back at some point, at first on the same cloud provider in the same region, and then later on the same cloud provider in a new region, and eventually a new provider until they rinse and repeat. It's only a question of when.
Cladex@reddit
Do they use a certain agent string ?
IRIX_Raion@reddit (OP)
See I tried that first and that only got rid of about 30% of the traffic.
Cladex@reddit
The fact they are manipulating the agent string goes to show they are being purposely underhanded.
ITaggie@reddit
As someone who is a sysadmin at a fairly large public library system, this is extremely common. Usually comes from Tencent and AliCloud ASNs, but of course there's a wide range. A lot has been coming from Brazilian public cloud providers recently too, and we just had a huge hit from HostPapa's surprisingly massive ASN (AS36352).
We've also had multiple occasions where they were apparently using botnets. The origin ASNs were all over the place but the traffic pattern made it obvious it was a coordinated scrape. That one was a doozy.
IRIX_Raion@reddit (OP)
You said it not me. I've already started building safeguards against this sort of thing happening again because this was ridiculous.
TomaCzar@reddit
I remember this exact same discussion when search engine web crawlers were a concern. Admins would actually tell indexers not to crawl their domain to save on bandwidth/resources.
There'll be a period of adjustment during which compromises will be made (already showing up in the comments here). Eventually, it will all be normalized and accounted for. Bad actors will fall in line (more likely the low hanging fruit of just any ol' website on the 'net will have been consumed, regurgitated, and consumed again so many times that there's just no value in the content) while hardware and software optimization work to make the issue moot.
IRIX_Raion@reddit (OP)
My experience has been that resources actually have gone up in consumption over time. Nobody's optimizing anything anymore and especially with this rust crap going around, it's turning out to be absolute trash.
TomaCzar@reddit
Raw resource usage has definitely gone up, on average, but the only reason that's possible is because hardware has gotten so much more efficient >!(~~fighting the urge to reference overused and abused Moore's law~~)!<.
As far as software optimization is concerned, some has been, and some has not. I am continually amazed at what can be run on, resources constrained devices like an RPi. There was a period where we thought we had reached a practical limit to CPU capability and optimization was on the tip of everyone's tongue. Then, chip manufacturers had a breakthrough and it was back to "get it out the door". These things happen in cycles.
bobj33@reddit
Who is going to stop them? These are huge companies writing bots to get material to train their AI algorithms. They don't care about whether your website remains usable, they only care about getting your content to train on.
AI web crawlers are destroying websites in their never-ending hunger for any and all content
https://www.theregister.com/2025/08/29/ai_web_crawlers_are_destroying/
Anthropic downloaded the LibGen Library Genesis torrents to train on. All of this data is copyrighted. They just lost a lawsuit and had to pay $1.5 billion.
They could have bought these books legally and trained on them but they don't care until they got caught and the publishers have enough money for lawsuits.
https://www.cbsnews.com/news/anthropic-copyright-lawsuit-class-action-settlement-authors-1-5-billion/
If you don't have enough money to determine who is abusing your site and money for lawyers then you've got to do what you are doing and block ranges etc. I've seen a lot of people just block entires countries because they aren't doing business with anyone from those countries.
IRIX_Raion@reddit (OP)
My forum is cosmopolitan and has people from as diverse of places as Kyrgyzstan and Laos. I wouldn't want to necessarily make it difficult for those people to access.
I completely empathize with your frustration on this and while I don't take quite as aggressive of a stance (my forum has a lot of high quality information, so I almost would have preferred if they gave me an option to just rate limit it to an acceptable level) I do genuinely think people like yourself have a point. I'm going to keep on top of it
jakecovert@reddit
Safeline open source WAF