4Chan data can almost certainly improve model capabilities.

[-]

TheRealDatapunk@reddit

Did you try to replicate this on other models? I'd be curious about a SOTA model like Gemma4 or Qwen3.5 in particular, from a fairly ignorant position assuming more polticial selectivity in the inputs. Is the 4chan data available somewhere?

Reply

[-]

Sicarius_The_First@reddit (OP)

Yes regarding both questions. You can read more about it in this excellent thread (lots of very interesting comments!): https://www.reddit.com/r/LocalLLaMA/comments/1qsrscu/can_4chan_data_really_improve_a_model_turns_out/

Reply

[-]

Sicarius_The_First@reddit (OP)

Holy downvotes lol... OK, Pepe bad...

Reply

[-]

RandumbRedditor1000@reddit

Reddit moment

Reply

[-]

Sicarius_The_First@reddit (OP)

I don't even know if that's a reddit moment, or a bot moment. I suspect that in the last 2 years the amount of botting greatly increased in all types of social media.

Reply

[-]

Persistent_Dry_Cough@reddit

I voted down the thread. 4chan scrubbed of pol and other cancers maybe has some utility but why are you bringing this up? Ghislaine Maxwell was a mod on r/worldnews and Epstein met with the guy who created 4chan and then it turned into a radical forum pushing racist propaganda and the early elements of the Q conspiracy theory. You want that in the training data? Screw that, man.

Reply

[-]

Jluxo_@reddit

I just downvoted your comment. FAQ What does this mean? The amount of karma (points) on your comment and Reddit account has decreased by one. Why did you do this? There are several reasons I may deem a comment to be unworthy of positive or neutral karma. These include, but are not limited to: •⁠Rudeness towards other Redditors, •⁠Spreading incorrect information, •⁠Sarcasm not correctly flagged with a /s. Am I banned from the Reddit? No - not yet. But you should refrain from making comments like this in the future. Otherwise I will be forced to issue an additional downvote, which may put your commenting and posting privileges in jeopardy. I don't believe my comment deserved a downvote. Can you un-downvote it? Sure, mistakes happen. But only in exceedingly rare circumstances will I undo a downvote. If you would like to issue an appeal, shoot me a private message explaining what I got wrong. I tend to respond to Reddit PMs within several minutes. Do note, however, that over 99.9% of downvote appeals are rejected, and yours is likely no exception. How can I prevent this from happening in the future? Accept the downvote and move on. But learn from this mistake: your behavior will not be tolerated on Reddit.com. I will continue to issue downvotes until you improve your conduct. Remember: Reddit is privilege, not a right.

Reply

[-]

Persistent_Dry_Cough@reddit

4chan scrubbed of pol and other cancers maybe has some utility but why are you bringing this up? Ghislaine Maxwell was a mod on r/worldnews and Epstein met with the guy who created 4chan and then it turned into a radical forum pushing racist propaganda and the early elements of the Q conspiracy theory. You want that in the training data? Screw that, man.

Reply

[-]

my_name_isnt_clever@reddit

People can also just not like your thing, it doesn't have to always be bots.

Reply

[-]

Sicarius_The_First@reddit (OP)

Damn, from like -15 into a plus. I truly don't know how to explain this behavior except bots. When I made this thread it got BTFO immediately, if this was organic it would just stayed that way. But it didn't hence why I suspect botting. The equivalent ST thread was also downvoted into oblivion, and stayed this way, hence, not botting, consistent with organic behavior. (I frame the ST thread in a bit silly way, it is what it is).

Reply

[-]

TheRealDatapunk@reddit

Source of data having a politicial leaning contrary to what most assume(!) Meta to have, and seemingly showing an improvement is an interesting outcome. I'd assume the downvotes are because there is an assumption that this is primarily politically motivated posting?

Reply

[-]

dinerburgeryum@reddit

Don't know why you would assume any tech company has a political leaning other than "who is in power right now." I feel like the last six years alone would be enough to demonstrate that.

Reply

[-]

TheRealDatapunk@reddit

Maybe read what I wrote. "what most assume". And there is definitely one half of the US that thinks all big tech is basically communist

Reply

[-]

dinerburgeryum@reddit

Oh yeah my bad my tone was a little confrontational in retrospect.

Reply

[-]

seanthenry@reddit

They do most large companies do they lean toward power, does not matter if it is the right hand or left hand as they are from the same body.

Reply

[-]

Sicarius_The_First@reddit (OP)

Ah valid point regarding companies, for Meta specifically iirc Zuck was enthusiastic about Biden when he was in power, and then for Trump when he took power. I guess companies just doing company things..

Reply

[-]

Sicarius_The_First@reddit (OP)

I highly suspect ur right lol

Reply

[-]

Paradigmind@reddit

Just the average Israeli training a right wing LLM from data of a right wing site. Nothing new here.

Reply

[-]

Ardalok@reddit

I don't think Israel would approve of the majority of 4chan's opinion on Jews.

Reply

[-]

Sicarius_The_First@reddit (OP)

Lol they sure as hell won't. But freedom of speech is giving everyone a voice, especially voices one does not agree with. An echo chamber is bad.

Reply

[-]

Sicarius_The_First@reddit (OP)

My dude, 4chan is more than /pol/ But I genuinely appreciate your comment, it explains a lot of the behaviour I see on Reddit, and at first it didn't clicked with me. Now it did. Be well.

Reply

[-]

insulaTropicalis@reddit

Are you going to share the dataset, publicly or privately? I would love to 4chanize some model more streamlined for my hardware like the 120B MoE models around.

Reply

[-]

atineiatte@reddit

We've gone so far with reliance on distillation and synthetic training data that we're rediscovering that unedited human interactions improve the impression of a language model

Reply

[-]

AnOnlineHandle@reddit

The types of extremely niche adult story genres that Gemma 4 can produce has made me doubt my previous assumption that all models were being trained with purely synthetic data now. I can't imagine they'd intentionally generate those types of stories and have the darker details and tropes so spot on.

Reply

[-]

PunnyPandora@reddit

pretty sure the nature of the output has nothing to do with whether or not the data was synthetic. You can get any type of synthetic data now with enough effort

Reply

[-]

AnOnlineHandle@reddit

Previous models were not able to produce stories like this without finetuning, at least none that I tried, which makes me believe it's trained on real writing. I know because I'm one of the few writers in a fairly specific niche who has been active for decades, and recognize the very specific way that I and a few other writers in this very specific genre write in Gemma 4's outputs, the specific terminology, pacing, and very particular focus on specific things. I noticed some of that in the first Llama models but it still required training to get it to anywhere close, and I assumed since then they'd moved to purely synthetic data which would have stripped all of that.

Reply

[-]

Sicarius_The_First@reddit (OP)

Just a thought, I think that too much synth data might be bad. I have no way to prove it, but I suspect Qwen models got a really large synthetic data portion, and while it vastly improved STEM and benchmarks, such models (Phi too for example) leaves much to be desired in the creative department.

Reply

[-]

MixtureOfAmateurs@reddit

We learnt this with the phi 2 (and maybe 1) back in the day. Guanaco superiority 💪💪💪

Reply

[-]

Far_Composer_5714@reddit

Too much synthetic is obviously bad because it causes catastrophic.. repetition I guess he would call it. Synthetic data compounds the issue of seeing a regular pattern in language where no patterns should exist because they were created synthetically. AKA AI-isms

Reply

[-]

waiting_for_zban@reddit

In the before times (before Chatgpt, or even GPT3), [Kilcher built gpt-4chan, trained on 4chan data](https://www.youtube.com/watch?v=efPrtcLdcdM), and then let it loose on 4chan. The results, fantastic. You can still find the model floating around, but as you can imagine in this day and age, anyone would be cancelled for putting a direct link to it.

Reply

[-]

Sicarius_The_First@reddit (OP)

Yup, the model was disabled on hugging face.

Reply

[-]

BannedGoNext@reddit

Wow why? Was it like mega racist?

Reply

[-]

Sicarius_The_First@reddit (OP)

It's a complex story, you can read some of it in the model discussion (hf CEO was also present heh)

Reply

[-]

Bobby72006@reddit

Hey, I found it on [archive.org](https://archive.org/details/gpt4chan_model), Kilcher also uploaded a 16bit float version of it on here too.

Reply

[-]

StefanStef14@reddit

is that the ai that made the bottomless pit meme? cause that is still the best meme ai has made

Reply

[-]

denoflore_ai_guy@reddit

The creator deleted the pictures and links from the training data. This meant the AI could not see what the users were actually reacting to. On that website, a post with no text is an image upload. The AI noticed that empty posts happen frequently. However, because it could not see the images, it thought posting a blank message was just a normal way to communicate. When the bot was turned on, it started posting completely blank replies. It copied the visual pattern it saw, but did not understand the rule behind it. It learned to read the data like a timeline, instead of learning how the arguments actually connected to one another.

Reply

[-]

Sicarius_The_First@reddit (OP)

Yes, we 100% are, this was discussed in depth in the 8B variation analysis, many good and insightful comments in [this post](https://www.reddit.com/r/LocalLLaMA/comments/1qsrscu/can_4chan_data_really_improve_a_model_turns_out/).

Reply

[-]

IrisColt@reddit

I thought Assistant Pepe was an already months-old model, my fault.

Reply

[-]

Sicarius_The_First@reddit (OP)

Not ur fault at all, I made another one, so the fault is mine 😔 The first is 8B, this one is 70B. And there's gonna be a 32B as well!

Reply

[-]

IrisColt@reddit

32B?! No way!!! Dropping when?? I’m literally counting the seconds... thank you!!!

Reply

[-]

roosterfareye@reddit

42b?! Woohoo!

Reply

[-]

seanthenry@reddit

I'll need to check it out at some point. Have you tested with Heretic - NoSlop data set? I'm wondering if there would be a real difference running that first to remove/reduce some of the AI-isms then add your data set. Over running it after your data set is added.

Reply

[-]

Southern-Chain-6485@reddit

I'm downloading the 8B to test it, but do you think you can make an intermediate between 8b and 70b dense, so it can take advantage of 16-24gb gpus?

Reply

[-]

Sicarius_The_First@reddit (OP)

YES! And it is already ready, a 32B version is currently being tested, will likely be available soonish.

Reply

[-]

roosterfareye@reddit

Cant wait! Another coffee coming in hot!

Reply

[-]

raika11182@reddit

So I tried out the 70B model out of curiosity last week and it went well. It's a good, solid model. I avoided trying it out for a long time because the name made me assume it was just a troll post that made it on to Huggingface as has happened plenty of times. If you actually want people to use it, even if it's trained on 4chan data, just change the name. It's really that simple.

Reply

[-]

Sicarius_The_First@reddit (OP)

Yeah, you ARE right about the name being... a bit problomatic, Pepe got a weird history being hijacked by various political movements. The 4chan data itself raises eyebrows. I see in Pepe a wholesome 'feels good man' meme persona. And 4chan data is more than the toxic waste of /pol/ I was hopping people would judge the model based on merit, but for sure, you're right, it might get a bit less love due to my choice. I am however standing by them.

Reply

[-]

roosterfareye@reddit

The right appropriate Pepe. Sicarius is helping take him back!

Reply

[-]

Ardalok@reddit

Cool work dude! Are you planning to train new Gemmas or Qwens?

Reply

[-]

Sicarius_The_First@reddit (OP)

Gemmas 💯 yes, qwens I already have 3 different sizes uploaded (not visible) which I need to test first. I need more... Time..🥲

Reply

[-]

roosterfareye@reddit

Time. The enemy of what you really want to be doing!

Reply

[-]

Vivarevo@reddit

Much of 4chan is propaganda bot content. Not props a good idea.

Reply

[-]

81stredditaccount@reddit

This is the best model. It tells it like it is and doesn’t treat me like a child

Reply

[-]

Sicarius_The_First@reddit (OP)

☝🏼This. This is one of the main reasons I chose to use 4chan data. Disagreeableness, inclination to argue. This is very effective to combat the LLM always softening criticism and glazing the user. I think it's ironically also good for certain aspects of AI safety.

Reply

[-]

Puzzleheaded-Drama-8@reddit

Do you think you could fine-tune it on Linus Torvalds mailing list roasts? I already love the 70B for code review and I think it could improve it even further in that regard without shifting the style too far off.

Reply

[-]

Sicarius_The_First@reddit (OP)

I'm open to the idea, not a promise though hehe Feel free to link the dataset, and I'll take a look!

Reply

[-]

PurpleWinterDawn@reddit

This too. If I wanted to be glazed like AI models do, I'd be a donut. I like your direction of thinking. I'm questioning the big players thinking the User should be an absolute ruler, even when sitting on a throne of lies, and the model should be a peasant groveling at its feet. The Emperor has no clothes, and AI models keep hallucinating them.

Reply

[-]

Sicarius_The_First@reddit (OP)

For example, I remember an article about some dude who decided to form a cult, and it was specifically gpt4o who encouraged him. "You're absolutely right!" "This is a great idea!"

Reply

[-]

FastDecode1@reddit

AI companies should take note. I actually think things would be better if models were just allowed to tell the user they're retarded and call them a bundle of sticks.

Reply

[-]

a_beautiful_rhind@reddit

I can tell you it flubbed the AIME test when I ran it. Didn't compare the original model but devstral did magnitudes better. You need to check on how you trained because stuff would change in context.. like the colors of shirts, clothing, etc. Actual *comprehension* was improved though. It's a fun model.

Reply

[-]

kaisurniwurer@reddit

So you are saying that a model tuned specifically to be less algorithmic and predictable is worse at math (and probably stem in general)? Comprehension and emotional intelligence is what makes for a good LLM, the rest can be done with tools.

Reply

[-]

a_beautiful_rhind@reddit

Yes but it was annoying to see it swap details from one message to the next. I think that is a bug to be fixed.

Reply

[-]

Luke2642@reddit

If you have time/budget, can you try hyperfitting https://arxiv.org/abs/2412.04318 and see if it is replicable or nonsense? It would seem compatible with your dataset, boost confidence in the long tail rather than the rlhf induced style?

Reply

[-]

Needausernameplzz@reddit

thank you for sharing this

Reply

[-]

yall_gotta_move@reddit

Whoah, cool paper. Didn't know this one. Thanks!

Reply

[-]

Sicarius_The_First@reddit (OP)

This sounds interesting, but to be honest I have so many things I have to try that sometimes I don't even know how to cram all of it into my time budget. Will look into it, thanks for the link :)

Reply

[-]

ganonfirehouse420@reddit

Always dreamed of running a local 4chan simulator.

Reply

[-]

Sicarius_The_First@reddit (OP)

Oh, you gonna love this then haha It communicates... Very humanly. Well, relatively speaking (for a clanker).

Reply

[-]

CommunismDoesntWork@reddit

>for a clanker Don't be mean

Reply

[-]

Sicarius_The_First@reddit (OP)

Once we reach AGI I'm cooked. It's over.

Reply

[-]

cutebluedragongirl@reddit

I just want models to be able to call me N-word and F-word freely.

Reply

[-]

freia_pr_fr@reddit

How does it perform on dating advices, political tips to avoid making your democracy a fascist dictatorship, or basic human decency ?

Reply

[-]

my_name_isnt_clever@reddit

4Chat data probably won't be great at the second one, but hey if you need the exact opposite I know of a country way ahead of you.

Reply

[-]

Imaginary-Unit-3267@reddit

You don't understand 4chan if you think it's fascist. It's about as anarchist as a site can get. It just so happens that lots of fascists post there.

Reply

[-]

Sicarius_The_First@reddit (OP)

That is actually really good and creative idea for a benchmark that should produce VERY interesting results!

Reply

[-]

synth_mania@reddit

Not the first time I've read something to this effect. And what was the other example.. I think Facebook unilaterally leads to regression? Lol

Reply

[-]

maorui1234@reddit

How do you train the model?

Reply

[-]

My_Unbiased_Opinion@reddit

Would you be able to do this to the Qwen 3.5 models? 27B is smart but a bit dry. Also it seems like it's a bit too nice sometimes.

Reply

[-]

rinmperdinck@reddit

Make a 69b for the memes

Reply

[-]

MerePotato@reddit

It can improve their ability to say that n word but I guarantee that thing now spews out disinformation like no tomorrow

Reply

[-]

314kabinet@reddit

Looking at the benchmark numbers, writing quality seems to have taken a hit.

Reply

[-]

Terrible-Mongoose-84@reddit

Are you planning to train new gemmas?

Reply

[-]

Sicarius_The_First@reddit (OP)

Yes, gemma4, especially the dense is 💯 getting a tune. Regarding the smaller one, the seem like an interesting contenders for mobile usage, I'll consider them too.

Reply

[-]

Lorian0x7@reddit

Oh yes! Really looking forward to it!

Reply

[-]

RandumbRedditor1000@reddit

I wonder what an assistant-pepe-gemma4-31b would look like

Reply

[-]

Sicarius_The_First@reddit (OP)

That's a really good question, and I honestly don't know and can't estimate. The reason is, from one side, google likely put the most tokens in their pretrain (vs other labs), but on the other side, they might do a more aggressive sanitation to the data. Gemma3 was EXTREMELY censored. BUT there's a place for optimism, as from early feedback ppl seem to agree gemma4 is relatively relaxed with the censorship.

Reply

[-]

Koalateka@reddit

From my testing Gemma 4 censoring is rather low. With a good system prompt you can get away with the normal NSFW stuff.

Reply

[-]

RandumbRedditor1000@reddit

I've used Gemma 4 and overall it feels a bit less censored than gemma 3, and a whole lot more intelligent. I'm really excited to see what people cook up with the model (especially since Gemma 4 is apache 2.0)

Reply

[-]

Sicarius_The_First@reddit (OP)

Apache 2 really surprised me, iirc that was the first model by google with it (not counting some berts and so on) I think it will take some time until we'll have more optimizations training -wise.

Reply

[-]

Koalateka@reddit

+1 to this. I would like to get my hands on something like that.

Reply

[-]

alphapussycat@reddit

Better at doing what? Useless tasks?

Reply

[-]

Hoppss@reddit

Interesting project and I agree with your core idea, but "outperforms the base model" on UGI alone isn't enough and delegitimizes your claim.

Reply

[-]

Sicarius_The_First@reddit (OP)

I would love it to be tested on any arbitrary benchmark, the only data point I had is UGI, and my own internal one. Do feel free to benchmark it and let us know, more data= better 👍

Reply

[-]

Hoppss@reddit

Your post says "outperforms the base model" as a general claim. If the data point is just UGI, lead with that.

Reply

[-]

insulaTropicalis@reddit

It's unsurprising that AIs mainly trained on left-leaning and pro-establishment corpi like reddit and wikipedia become smarter when exposed to anarchist and alt-right data. It's been shown in several researches that diversity in dataset increases intelligence.

Reply

[-]

lizerome@reddit

Gemini once gave me a shockingly thorough defense of the POV of the alt-right/dissident right/groyper/etc crowd. It had a canned "ah but you see" rebuttal ready to go every time I tried to question it or argue with it, all of which were really niche and exactly the sort of thing an intelligent proponent of that group would bring up. I found myself ending the conversation with a frustrated remark like "okay but even if all of that is true, wouldn't this cause this" — at which point Gemini finally conceded with an "oh no no, millions of people would die, of course, but that's besides the point". I'm not sure what they trained this thing on, but it's far less sanitized than you'd expect. Which might have something to do with the model performing so well.

Reply

[-]

Sicarius_The_First@reddit (OP)

This is likely indeed due to data diversity, google cooked well with gemma4.

Reply

[-]

Sicarius_The_First@reddit (OP)

Yes, and one of the comments in the 8b model (linked in the post) corroborates this further. iirc he said that social media data actively retards the model. It's the most upvoted comment in the thread, in case u want to find it.

Reply

[-]

cgs019283@reddit

Is there anything more than the UGI benchmark? Of course, it will be better at responding to censored topics, but that doesn't mean it is certainly a better model. Even Grok is the highest one on that benchmark, which doesn't represent real-world usage.

Reply

[-]

Sicarius_The_First@reddit (OP)

I wish there was, but for quite some time HuggingFace closed their leaderboard.

Reply

[-]

Sicarius_The_First@reddit (OP)

Oh, UGI does test general intelligence too, not only how uncensored a model is. So there's code & general knowledge tests as part of the total UGI score.

Reply

[-]

My_Unbiased_Opinion@reddit

Yeah the NatInt section is the first thing I look at.

Reply

[-]

Sicarius_The_First@reddit (OP)

It's genuinely a good benchmark, no one knows WHICH knowledge is being tested, so no way to optimize for it. That's a good thing.

Reply to Post

104 Comments