man, tellin people it cant be done is just gatekeepin. i had some issues gettin decent performance on older gear for my own local projects, but once i tried fibo it actually smoothed out the workflow significantly. u just need to manage ur expectations with local hardware. https://bria.ai
The important signal in Pack it up guys, open weight AI models running offline locally on PCs aren't real. đ is the operating constraint, not the headline. Those numbers -- 4, 3x, 1.5GB -- matter because they change who can actually run it.
For creators or agencies, I would test it with one repeat task: same prompt, 10 runs, track output quality, failure rate, and cost per usable result. That is where hype becomes a buying decision.
they seem to think that all the datacenters AI companies talk about are for like 1 person and everytime you message chatgpt youre using the whole thing yourself or something so the prospect that AI can run on a single PC in impossible to them because theyre too stupid to comprehend what scale can do
And the whole 5 million gallons of water. People are acting like these datacenters are using this much water everyday... They are not. They *are* using a lot of energy but they are not using that much water. All current thermodynamic cooling systems that use water are either fully closed loop or hybrid with minimal maintenance and the maintenance isn't 5 million gallons...
Now electric requirements are definitely something to be upset about.
Remind me not to ask them about their data center account stuff, such as email, Spotify, TikTok, YouTube, Reddit...or Amazon deliveries, their shopping at Walmart, etc. đ
When you point that out, these dipshits start in with "data centers aren't the Internet lol dumbass", and that gets upvoted to the moon while reasonable people are downvoted to oblivion.
Herd mentality will be the death of us all.
Running local models is kind of insane though. I want to run an 8xH200 model sometimes, I do not want or need a $500k computer that sucks down 10KW in my bedroom, I would rather use a cloud service where I can timeshare for the minute or two worth of tokens that I need generated.
> I want to run an 8xH200 model sometimes
Then you just wait 6 months until a model of that caliber is available with open weights that you can run locally.
To an extent sure. Obviously local models aren't as good as sota models from 6 months ago when you compare even something like an RTX6000 to a full server rack. But the gap is not as wide as many think, especially when looking only at inference rather than training. And really that's the MAIN reason these companies are paying half a million for 8 GPUs, it's for training. Actually running the model takes vastly fewer resources and even less if you quantize the final model.
But to say that local models are NEVER going to be as good is just flat out wrong. There are local models that you can run on modest hardware today that blow chatgpt 3.5 out of the water, and that was state of the art at one point.
I mean, I could be wrong. My feeling is that human brains have considerably more RAM (or something like it.) You'll certainly be able to run dedicated models that are fine, but I don't think the hardware anyone has in their home is powerful enough to match a human brain in its ability to consider all the angles.
You claimed that small models are going to get as good as large models, my argument is that they cannot because they don't have the hardware to do everything large models can do. I was using the human brain / AGI as evidence to support my view that there is a ceiling on the performance of small models.
I mean, there already are small models that are better than the big models from just a few years ago. There's tons of small 80B parameter models that will blow away chatgpt 2 and 3. Like it's not even close.
So yes, in time small models will absolutely be better than chatgpt 5.2 and all the current sota
Yeah yeah, sure thing buddy. You and every other anti have been saying that for years. Just like how Moore's law is going to hit a wall any decade now right?
I didn't say Moore's law is going to hit a wall. I said LLMs that can fit on current consumer hardware are going to hit a wall. In the future I hope we will have consumer hardware capable of running frontier models.
I know what you said. I'm categorically disagreeing with you based on the fact that models running on consumer hardware are already better than models from just a year ago and Moore's law has been making consumer tech better and better year after year with no signs of slowing. A cell phone is more powerful than every supercomputer combined from the 80s. If you that because we have LLMs now that that trend isn't going to continue you're just oblivious to reality.
I think it will be at least 5 years, possibly 10, 20, or even longer before you can get a consumer GPU with the 1TB+ of RAM you need to run a frontier model. 10 years ago the best you could get in a consumer graphics card was 12GB RAM, and 10 years later it's only tripled (not really) to 32GB on the 5090. And the 5090 is $3500, even adjusting for inflation it's a bit of a stretch to even call it a consumer card at this point, dollar-for-dollar a $1300 GPU is not that much better than the Titan in terms of RAM.
In terms of FLOPS, you do get about 4x the power for the same money, but anyway, the point is, yes, Moore's law is not dead, but we are not running frontier models for less than $200k anytime soon.
That's also a blanket argument against home computers, however people benefit also from the locality and the control not just having the most computing power money can buy.
No, it's not. For workloads that can run on a typical 300W laptop, I'll happily run them on my laptop but I do not have a place to plug in a 10KW machine nor do I have a strong enough AC to deal with the waste heat.
it is completely a blanket argument
you can always rent much more computing power than you can buy
the reason you don't always rent is that there are objective advantages to the control and locality of the computing power that you can buy
exactly the same happens with generative AI, but you are not using those so you're content with only renting, but that doesn't mean there is no reason not to buy and others are taking advantage of it
> exactly the same happens with generative AI, but you are not using those so you're content with only renting
not exactly sure what you're talking about here. I'm talking about LLMs, generative models. Using them for the things where they're what you need: summarization, translation, etc. Small models are strictly worse and the frontier models provide consistently better results that are just barely workable, the small models have too much error to be workable.
Small local models are VERY useful, they are deployed extensively in enterprise environments to:
1) decide semantic boundaries in documents, essential for a good chunking strategy
2) image annotation to enrich metadata for image database retrieval
3) speech to text, text to speech synthesis
And a boatload of other stuff
I've tested such models and the large ones are simply better. It is worth being able to run them offline, but they are dramatically inefficient and also worse.
There's a lot of data management tasks where I haven't found small models that can do the job as well as the larger models. The smaller models work like 95% and the larger models are closer to 99%. And that's just for the stuff where I can obviously detect the output is wrong, half the errors are undetectable, and if I have to invest time in detecting errors it mostly defeats the point.
When you have 2TB of documents to chunk and a budget, the local models start to look very good. Plus the fact that for semantic boundaries the smaller open weight models give pretty much the exact same chunks that the frontier models produce. The costs are going to be astronomical using the APIs for chunking everything using frontier models, that is not a practical solution.
How else would you find semantic boundaries of each chunk over 2TB of docs? Do you have a better way? So you're saying don't use a small LLM but use a large frontier model instead?
No, I'm saying you don't need an LLM at all to do chunking. Although also chunking is a problem you have because you're using a tiny model with tiny context, and it's not a very good solution to the problem. If I can't load the entire context into memory I tend to prefer conventional text search.
I see talk like that a lot too, so I did the math last year to compare the energy usage of training one of our models at the time on 8xH100 verses fast charging an EV.
Using the specs and stats for the large charging station at one of my offices over a few thousand sampled sessions, and the system stats from one of our models in training. It worked out that 1 minute of average fast charging uses almost exactly as much electricity as 1 hour of AI model training on one 8xH100 node. It was weird how even the units were.
It seems that one EV fast charging uses as much power as 60 8xH100 systems. At a typical 4 nodes per rack, that's 15 racks worth. That's pretty insane.
I'm not even sure how many concurrent users that much compute could server... In my benchmarks on a 8x4090 system running vllm for oss-120b, I had it doing 20-100 concurrent at acceptable rates, so I would imagine commercial inference on Hopper or newer are getting much higher than that per node. Meanwhile the other was just a single average EV sitting there with the AC on while charging.
A friend also converted these in to tree and tea cup equivalents.
> Making a bunch of shitty assumptions on the side of error, one "average" tree seems to be around 4000kWh, which is around 2.5 weeks worth of 8xH100 time.
> A single 4090 running at 100% power limit [which it won't for inference] is something like 0.001 kWh per minute for the entire PC. For reference, an electric tea kettle consumes around 0.017 kWh per minute. So you're looking at maybe 1/100th of cup of tea per second of generation. Its possible local gen is more energy efficient than your average British person's leaf broth addition.
Itâs just the lack of understanding of how anything works⌠no point talking to these people as these people probably think AI is magic at this point with zero understanding of the technicalities.
It really is utterly striking. I've been in anti-AI groups for maybe years now and you people wouldn't believe what takes pass as technical knowledge there.
What exactly is an anti-AI group? Is it like a book club where people are meeting and shit-talk the newest model releases? In my circle I only know people who are either enthusiasts or don't really care about AI.
LOL and there I am on BlueSky, literally earning a degree in AI development... The irony so rich... So pure... And the people! Oh the people... So very very dumb.
They think it's dark magic. Why would they learn more when they know for sure it's evil? They have zero nuance. To them, ff something has been touched by AI, it's slop. An image is either AI or it's not. If it can cause harm, it should be banned. It's stealing the future work of artists.
It's so strange watching them try to puzzle out how an image should be illegal if it's AI, but should be allowed if someone is really good with PhotoShop. Their whole underlying premise is wrong and it ties them up in knots because they lead with the conclusion that AI is evil, as are its products and therefore they must be made illegal.
I once criticized AI anthropomorphization to an anti audience, thinking it'd be a safe topic. But it didn't hit well, and only then I realized: it's because *they're* some of the people whom anthropomorphize AI the most.
If you admit it's a tool, then several of their common arguments crumble, including the ones that say that doing something with the help of AI is "just like commissioning someone to do it."Â
Even for people who do like AI, it's shocking. Two years ago, I ran a demo on my iPhone in airplane mode at a conference. Someone in the audience tried to prove that I was somehow faking airplane mode to them.
So if OpenAI banckrups does it mean all their GPU powers will get evaporated? No one will aquire it? No one will flood market with a ton of unused power? It's freaking silicon in a rack not NFT.
I read somewhere that NVIDIA has agreements with datacenters to prevent them dumping "old" hardware onto eBay. It might never happen at all, or it would be like 10-20 years from now on, sorta like old Tesla GPUs from 2012 that nobody wants. But even if you think about it, it would only makes sense if they're out of rack space/power for hardware that gives them much more money per watt/space.
Might be wrong tho.
I believe atleast a few ai companies will collapse, providing gpus. hopefully we get our hands on some, rather then google or something swoop in and buy them all đ
Running Truenas CE and it runs everything at once. Plex, Cloudflare, game servers, nextcloud, Audiobooks, podcasts, automatic backups of everything. Also AI OpenWebUI, Automatic, Kokoro and Anything else I can think of.
https://preview.redd.it/dl2h3fuhwmkg1.png?width=2199&format=png&auto=webp&s=10da093c7bea89e4f6eada38fe544a2171af522a
Jezzus bruh... when the bombs drop you gonna be ready to bring the interwebs back single handedly from your basement! What a playground... sounds like you'd need a diesel site generator to keep that puppy rollin tho after the grid goes
You should see the idiots on r/accelerate. They are so high on the big AI supply that they are not seeing the ecosystem that is forming with Local LLMs with Openllama and LLMStudio.
Yes rob them of all safe havens for them to communicate and develop as people, brilliant idea there... No wait incredibly dumb. Just say you want everyone to die in the closet so you can feel comfortable and be done with it.
Truly awful. All generations prior were socially starved and unable to communicate or develop as people.
....
.....Right?....
Oh wait they just communicated in person with less social comparisons and less methods to be bullied.
Oops!
Tell me you grew up in a city or suburb without telling me you grew up in a city or suburb. Lot of ignorant privileged mfers on here happy to disagree and down vote. Your life isn't everyone else's life. Were your neighbors cows? Did you have to buss over an hour to school to see anyone your own age? Was your graduating class 60 people? Are you lgbtqi in such an environment? I'm going to guess no because k if you were you wouldn't have written what you just wrote.
There are more to socializing and developing as a person than just going on social media to listen to kids brainrotting each other for years on end. If social media is the only thing for a kid to socialize, then that just means the parents and family around them use that as well.
There's more bot comments than what people realize I think. Some comments are just so blatantly stupid that it makes a legitimate idiot look not so stupid. That's typically how you can identify if you are talking with bots or not.
Bots tend to be more coherent than human idiots these days.
We're well part the point where the dumbest people are worse than then smartert AI models, particularly in written communication.
Ofc it is, models do run 100% locally, and theyâre evolving fast. So, no, they wonât âgo awayâ, the guy that wrote that comments are the one that is really insane, Iâd block him and never read any more of his comments.
Since I at first thought you were joking but now am not so sure: if you have a runtime (llama.cpp, lm studio, ollama, whatever) and a model downloaded (gguf, mlx, whatever), you can run your models regardless. it is all running on your machine, you can turn off your wifi/unplug your ethernet and it will still run.
Sorry if you actually were joking, in which case, lol.
No no, I'm actually a bit dumb when it comes to these things. Thanks. So big companies going down won't affect the locally saved models on my PC right?
Even some of the people on this sub would qualify as comments/threads that one should avoid imo.
Just saw a thread yesterday where hundreds of people on this sub were seemingly incapable of grasping the benefit from the wave of attention/builders that are pouring into the space from popularity of openclaw.
Recently I reminded myself that a large portion of social media users is made up of teenagers and that really dampened my willingness to be present. I have nothing in common with them. My own kid is out of his teens now.Â
Lmao, I keep telling people this. Thereâs this weird misguided idea that if the bubble pops and all the AI companies go out of business, or if we, I dunno, straight-up ban them from the marketplace or something, that the GPT genie goes right back in the bottle and we can all just return to 2021 like nothing happened.
Which is absurd for multiple reasons, not least of which is that if the bubble popped tomorrow and OpenAI, Anthropic et al. went under (or more likely, get acquired), the only thing that would happen is that Google, Microsoft, xAI and Meta would consolidate and dominate the AI market, likely at a higher price point. But also, anyone can run AI on their own machine, and even small models perform surprisingly well by now.
That cat is NEVER going back in the bag, full stop, any more than computers, smartphones or the Internet will.
Even in the absolute worst case scenario where all companies go under or refuse to develop it further... current models are suitable for a lot of uses and aren't terribly expensive to run. Training is the expensive part, so we'll at least have current models to use going forwards
The best part about running models locally is that no company can rug-pull your access. Every time OpenAI changes their ToS or nerfs a model, the local community just shrugs and keeps running whatever version they already downloaded. That permanence is seriously underrated.
No they donât , even if this guy is stupid as fuck, Iâm sorry, but unless you have 100k local ai (except for medium sound and image) is not valuable.
I have a workstations with 96gb vram, it costed me 10k, and I can run good open source, better than most. Hobbyist.
Iâm sorry but in the agentic workflow or serious video. It doesnât cut it. Compared to api.
so no sota ai company donât give a flying fuck, they embrace it actually since they are ripping all the works open source doesâŚ
an agentic workflow that actually worksâŚ. Hum⌠so tasty, we at open ai love seafood.
To paraphrase a remark that goes around; "It's amazing how much anti-AI discourse is just them pretending not to understand things, thus making discourse impossible."
Ran into this with someone refusing to acknowledge that AI could have *any* benefits and deliberately downplaying stuff like helping stroke patients talk, identifying invasive species from a distance immediately, etc. stuff we are already able to do just a few years in. So many people who are just blanket anti-ai are crazy, they also refuse to acknowledge the little things like transcription, translation, so on.
I think what people miss is that the advancement in generalized generative AI is a rising tide that assists all the specialized use cases of AI that you listed. All they see is what the common person does "playing with this new toy", and think literally that it is all it is good for, and that making it better is pointless at best, counterproductive and disastrous at worst. They literally lack creativity.
I hate having this conversation with people for this reason - they don't understand the fundamentals at all, and they don't want to. They only want to hear about how it hurts the environment and ruins people's lives.
I'm happy to have a conversation with someone who is well-reseadched, but... They're usually pro-AI
I am strongly anti-ai but still have fun running things locally lol. Not sure where that puts me in your regard? I think local ai is worth a laugh but corporate AI offerings are a societal disaster waiting to happen.
>but corporate AI offerings are a societal disaster waiting to happen.
That's just _capitalism_ baby, it's a feature not a bug... Corps can destroy the commons to extract maximal short term profit so societal disaster is just a guaranteed outcome, as long as it's not *this* quarter everything is _fine_.
Thats most of pro-ai. Pro-ai side mostly just want to be able to tinker with their local models and make funny pictures and stories without getting harassed for that.
The ones who worship corporate side are either grifters who just follow hype and do not care about ai at all. Or delusional people who still think that big corpos can have their best interests in mind(once again regardless of specific technology). There are also some people who literally worship AGI as a god (despite no AGI existing yet) but as far as i see, they are not too numerous.
Reminds me a lot of the midwit meme.
My aunt who's like 70y use ChatGPT daily and often ask me for tech support. While the average person online I talk too almost always is an anti.Â
I am anti cloud AI (resource consumption, privacy, etc) but pro local AI and I canât imagine how it is to deal with a full âanti AIâ person they have to be the most retarded mother fucker youâve ever talked to.
> I am anti cloud AI (resource consumption
Eh, isn't cloud AI like a factor 5-10 times more efficient than at home due to much better networking and batched operation?
A fuckton more services that are used daily eat more resources. Streaming for example is a lot worse for the environment, Gaming, a bunch of other stuff. Hell, Reddit probably is too (tho I'm not sure)
Wait till you discover most of the internet that you surf through, including reddit, runs on data centers that burn through Electricity, water and silicon.
On the bright side, when one side is consistently well-researched people while the other shuts their ears and goes "lalala", you know which is on the right side of history.
some one don't have to be well researched but just need to have some open mind. I've found most of the anti AI people hold their belief as religious belief. No matter how much you explain to them they won't change their view or even consider it.
yeah. I think it's some odd religious thing. Old school religion is dying out, and the new preachers/effective thought leaders are newscasters and influencers
That's fair. That last part was mostly a joke - I only have experience from what I see online, and in non ai threads the same talking points come up again and again.
People I speak to IRL about it seem mystified and overall positive. Social media is not a good representation of reality.
Well, it does indeed hurt the environment even if used only for local ai. Training a model requires a huge infrastructure which is exactly what they, despite not knowing it precisely, are going on about.
What we can say is -
\> AI training takes a lot of power
What we cannot say is -
\> AI training is bad for the environment
do you see the difference there? One is a blanket statement, assuming the worst. The other, a truthful statement, suggesting a problem that \*we can find solutions for\*.
It's not about assuming the worst here. AI training does indeed require huge amounts of resources. I'm pro AI btw.
But as with American politics, people are divided into two camps. And you apparently are just as interested at looking into how much it's actually hurting our environment as they are at what opensource models are and how and on what they can be deployed.
As it stands, the power mix used and infrastructure constructed is indeed bad for the environment. We can totally say it. It's not "assuming" anything. These things are known.
Yeah. Weâre going to end up giving nearly every detail of our lives to analysis, and then machines will âpredict our next moveâworse than that, design, predict, and fire our next move.â In a consumerismâoriented culture, what else can you expect?
AI will not truly prosper. It will remain a tool for the powerful, used to more efficiently âmanufacture consent.â
I really feel the pro-AI side is missing a trick here. The moral/environmental implications of *inference* or *fine-tuning* on local hardware are night and day.
Training GPT-OSS-120B takes city-sized datacenters measured in gigawatts that pollute lakes. But then it's trained.
Inference takes a Mac Studio or DGX Spark about the size of a loaf of bread, running on "fancy laptop" wattage. GPT-OSS-20B on your MacBook Air, if you want.
We can get the practical/productivity benefits that actually exist if we stop training *today.* We could massively limit the damage by training *sanely.* Imagine a system where it was done like past major scientific things like the Space Race. Companies did some jockeying, then they made just the Saturn V rocket. They didn't make nine others so they could talk about line go up.
The models are good enough, esp. with better tooling around them.
It's the AI Arms race for training that is doing a lot of the environmental damage, not inference/use.
Doesn't solve the moral/IP implications of training data or the potential psych issues, but inference is nearly a non-issue compared
Transformers and LLMs are definitely the tech of the century.
Even if we come up with something substantially better in the next decade or three, it'll be because Transformers reinvigorated AI research and transformer based LLMs kicked off the global race for AI technology.
We are past the breaking point in terms of climate change, no amount of reduction in resource expenditure is going to prevent climate change.
The only thing we can do now is double down on science and AI is going to be a big part of that.
Not LLMs, but AI models have already made contributions to designing new solar panels and wind turbines. AI models have already helped make more efficient technologies.
We need to get agentic AI and robotic AI to the point where they can be doing truly autonomous labor, because that's the point where we can be setting them to work building renewable energy plants in remote area, reversing desertification, cleaning the oceans, and other super-massive scale work that would be unfeasible with human labor, on timescales unsustainable by most organizations.
In this field, maybe. I'm not going to put a very fancy, very capable autocomplete up against the switch to green energy, thorium reactors, widespread therapeutic use of genetic engineering or GMO organisms to create currently expensive medicine at scale, or fusion power all the other possibilities before 2100. If we do get robots and AI to the point where they support them, then *robots* weren't the invention, the green tech they built was.
Those are also going to have to be public works--companies won't do shit that doesn't turn a profit *this quarter* and huge public works are losses for a decade so the resurgence of democracy and investment the public good would be the most important *invention* of the 21st even if not a technology.
We probably can't even imagine the most transformative tech, only 1/4 of the way through.
That's why I like to think in decades. Smart devices like the iPhone? 2000s. Social media? 2010s. AI? Might well be the 2020s.
Poking fun at how business articles and so on are packed with corpo-speak and culty-ness so Sacred Words Identified with Capital Letters, paired Corporate Legal Symbolsâ˘. Just take it off line, circle around, synergize outside th box a little bit, and you're golden.
The funniest thing about "local AI isn't real" takes is that these people are posting from devices with more compute than what ran the entire Apollo program. Your laptop can run a 7B model that would have been state of the art 2 years ago. But sure, it's not real because it doesn't have a subscription fee.
I guess my PC which can Kimi K2.5 at full quality does not exist according to them. This level of denial reminds me of flat earthers, that deny facts even after an explanation.
Reality is, AI is not going away. All models that have been released already can do a lot. Just two years ago, I barely could trust a model produce part of code after detailed prompt... now with K2.5 I can let it be for hours and it orchestrates entire project according to specs that it can read and discover on its own, can use web browsing and vision.
But current AI barely scratches the surface... most obvious things, there is still no large model of K2.5 scale that supports input-output across image, video and audio modalities. There is no production models yet that reason in non-text tokens (like using images/animation/audio in thinking). There are some experimental architectures that take thinking out of text token space while processing video, so clearly that's possible, but it will be a while before something like that goes into production-ready architectures. Each time step forward gives a lot of advantages so I don't think pushing AI forward is going to stop, at most it may slow down later on, when most "low hanging fruits" are discovered.
I run K2.5 on 64-core EPYC 7763 + 8-channel 1 TB 3200MHz RAM + 96 GB VRAM (made of 4x3090) + 8 TB NVMe for AI models and 2 TB NVMe SSD for the OS + ~120 TB disk space on HDDs for storage and backups. If interested to know more, in my another comment I shared a photo and other details about my rig including what PSUs I use and what the chassis look like: https://www.reddit.com/r/LocalLLaMA/comments/1jxu0f7/comment/mmwnaxg/
There's very few people that actually have valid anti AI concerns. Most of it is knew jerk reactions based off of ignorance and just repeating what they see others say.
That's generally how most of Reddit works tbh.
Is it really true that "very few people have valid anti-AI concerns"? Because I feel like we have to admit that there _are_ very many valid anti-AI concerns. It's just that the anti-AI crowd's AI-related reasoning is fucked, and so their reasons for their anti-AI concerns tend to make no sense and be contradictory. But the concerns themselves are broadly fair, no? Corporate domination, replacement of human-to-human interactions with ai-interactions, pricing out of individual consumers, sameification of culture, ease of spreading disinformation. Of course not every concern is valid, but there are many valid ones.
Yes I worded that badly tbh. There are a lot of valid AI concerns, the point I was trying to get across is that very few people actually bring them up as reasons for their anti AI stance. It's always the same few arguments you see repeated over and over. Then when you push them you find out their level of knowledge of AI stops at ChatGPT.
It's not just the anti AI people either on the flip side the full on "AI bros" can be just as bad, only the opposite.
Yeah, for sure. It sucks cause the people whom I'd want to talk to about AI have closed their heart to it, and the people who do want to talk about AI are either AI bros, or even if not, they're typically people with a strong technical but not humanities background, which is of course useful but doesn't inspire many interesting non-technical insights.
>That's generally how most of Reddit works tbh.
You can generalize this to all social media where the viewer count is large: after a certain size, you simply get too many clueless people who feel they have to say something. Parroting something they have seen many times without understanding it.
I have yet to see this in Mastodon where (so far).
Lol. Only an idiot who doesn't know how local AI models work would create a post saying such nonsense. That said, I'm not even going to waste my time explaining why they're wrong. Reddit is also full of funny and irrelevant posts where you can have a laugh and pass the time...
It's almost like the anti-AI crowd is just parroting TikTok/blog talking points without having done any serious research into the subject they're passionately arguing about.
So any person who's been given a drive-by informing on any topic they've decided they should care about and assumed that the one piece of information on the topic that managed to stay in their head was everything anyone needs to know about the topic and must be true?
Apparently my hurried, sloppily thrown-together run-on sentence is confusing. My bad.
TL;DR: People who form opinions based on something they heard about a topic somewhere without doing research are going to have bad, uninformed opinions no matter what those opinions are about.
I really was far too ambitious with that thing. I still can't decide what I'd put where. Long sentences are my jam, but usually it's fairly obvious where the punctuation goes. I read that monstrosity and I'm like, "anything I add changes what I wanted to say for the worse" even though it's terrible for comprehension as-is. I really do apologize for getting so far ahead of my skis on that one. I can and should do better.
And still, local AI inference with 20 tokens per second will not be enough to support a business that incorporates generative AI in any meaningful way. It's good for experimentation at home but that's it.
That's not enough when you're serving any meaningful amount of customers. If I land on your page and your chatbot takes ages to respond (because there are another 1000 people talking to it), i'm skipping and leaving a bad review. And that is just the most generic case that barely uses any tokens. Why do you think all of those dataservers are buing built?
People donât understand. This isnât just tech development; this is a race and a cold war in some ways. The USA is in a race against China on AI. We cannot simply give up. The results would be catastrophic for the USA to not continue the race.
It isn't just this, but i'm just amazed at the arrogance. I mean... why do people just blather bullshit without checking first?
It's these platforms. They reward the conflict, because that makes people angry, and that's addictive.
If hardware prices continue to rise, how will anyone run their own models? People with their ear to the ground in the hardware industry are already suggesting this will be a long term trend. It may even result in most software applications becoming cloud based due to lack of affordable hardware on local machines. The incestuous relationship between hardware and software companies could very well mean that every app becomes a cloud based monthly subscription. There will be very little incentive to make hardware affordable again.
Unfortunately I suppose I can't post there due to this rule :\\
https://preview.redd.it/06vvhefxdkkg1.png?width=442&format=png&auto=webp&s=3b4e977074e2992332d52b183d039e6d8d128e73
It's frustrating being the one person in my friend group who actually works with and understands the technology (at least a small, practical subset of it, I'm no Karpathy). And to some degree I get it, because the space is poisoned by grifters, hypemen, ragebaiters, etc. on top of actually concerning misapplications of AI surrounding privacy and surveillance. I'm just tired of having to answer for all the companies and CEOs that I hate just as much as they do.
worst part is giving patient, thought-out, explanations only to get eye rolls or demands that you "ELI5" something that has taken us all years to understand... lot of bad faith concern trolling out there
ironically we are not even against each other. the reason we want to use local AI is because we don't want to give money to these big corps. He just can't understand it and not only companies build these models universities do as well and we can customize these as well our selves.
I should probably research better options but I've been running the 20b gpt-oss on my 2 year old macbook and it's obviously not groundbreaking but it's fast and reasonably smart. All data centres could disappear tomorrow and this thing would still be immensely useful and it requires no data centres or even a desktop computer.
to add to this, since you're on a Mac... MLX has mxfp4 now, check out noctrex's mxfp4 quants of
GLM 4.6V Flash
GLM 4.7 Flash
Nemotron 3 Nano 30Ba3B
Qwen3 Next Coder 80Ba3B
and Qwen3 Coder 30Ba3B
I personally find GLM 4.6V Flash really useful for packing context first before I spend my paid tokens on a project
Your post is getting popular and we just featured it on our Discord! [Come check it out!](https://discord.gg/PgFhZ8cnWW)
You've also been given a special flair for your contribution. We appreciate your post!
*I am a bot and this action was performed automatically.*
The time wasted reading that comment aswell as the wasted power and compute for his phone to connect to the internet and post that is just astronomical with these ram prices.
I'm an "AI hater", for full disclosure. (I'm here because I like to learn about things though).
I am commenting to give y'all my perspective on this.
And to me this dude (@-\_XD) seems particularly dumb, almost everyone I know who I would describe as an AI hater knows you can run these models locally, clearly they have no idea how any of this works and are just an idiot. And obviously as long as people can still run local open source models and keep improving them, then that isn't going away. If anything, I see no reason why running local models won't become even more attractive in the future. After the initial bubble pop of AI (lets face it there's way too much money in this to justify) and the cost of hardware comes back down, I suspect local AI might be the direction a lot of things go.
I'm not even a hater in the sense that I don't like the technology, I think it's impressive technology and has great uses (like statistical analysis of huge data sets, or natural language searching of huge dumps of text). I just don't like it being used for art, I hate that my social media feeds are full of generated AI slop, I hate seeing it being used to do stuff like undress kids, and I don't like folks vibe coding entire pieces of software with it.
But most of all, I have major problems with the ethics of how it is being built from public data in such a way that it creates a one sided exchange where corporations are basically saying they can take anything anyone creates, books, movies, youtube videos, novels, reddit posts, family photos, anything, to use to create a valuable product (their AI models) which they can then sell, but apparently if I download a copy of a movie from 1963 to watch, I'm a criminal? Why is it OK that OpenAI or NVIDIA can take artworks I create or novels I write, and train AI from it? [At the same time, these companies have the nerve to then complain about people 'stealing' their IP by taking prompts and responses from their models and using them to train new models.](https://futurism.com/future-society/google-copying-ai-permission)
I feel like this is creating a society where no one will have any incentive to share anything and a heavily one sided unfair situation where corporations are getting all the value out of an exchange and we're getting nothing. Why should I write and publish a book, and put 9 months worth of work into it to source genuine real information I've validated extensively with research, when it will just be sucked up into an LLM and I won't even get any credit for it? Let alone anyone asking for my permission, or paying me for the work I did.
If anything I think people using open source models like this locally is 'the best case outcome' in a world where this stuff never goes away, because at least then people aren't paying subscriptions to trillion dollar corporations to access models created from stolen copyrighted material. So, yeah, Local FTW.
What do you want us to do about it? You're catching strays because you're standing in the firing line between these types one side against those dipshits who think we're on the cusp of the singularity because elon musk tweeted about it. Just don't get involved.
I don't even get what the tow in the original screen shots were saying.
Are people so bad at articulating themselves and hope LLMs understand it nowadays? Running ai on ai? Wtf does that even mean? Who is arguing against running it locally? The first poster or the second one?
I'd be okay with all social media platforms having an insta-ban rule for deliberate misspellings of words used to demean people. That, to me, seems like a far worse infraction than using the word without obfuscation.
If I just call you a rude word, that could just be a matter of not having thought about the impact my words have. I might learn and grow as a human being. But if I go out of my way to replace "u" with "v" in order to evade detection, that clearly indicates that I thought about the impact and chose to push forward.
A lot of people has zero idea about how AI or LLMs work. All you can do is point them at some online courses/tutorials/etc. Arguing with antis is always a waste of time.
my favorite part is "ai running on peoples ai" like the concept of a computer is completely foreign to them. running a 7b model on a laptop is apparently science fiction now
yeah lol i have SWEs calling me a vibe bro because i bought a used 3090 in autumn and having a blast on my local setup.
meanwhile they are now using claude code in their jobs - and no it does not make sense. i have stopped engaging with them.
If you have an old microwave or some form of metal box or something-laying around you should build a Faraday cage for you archival harddrive for models! A refrigerator and a microwave are both pretty good at radiation hardening on their own, but you can take it even further. https://tactileimages.org/en/sciences/tesla-coil-and-faraday-cage/ 101 on the concept.
If you were to touch-up an appliance to be a decent Faraday cage and then bury it on land you own then you may be the only person in your area with AI after an EMP gets detonated over your head (an absolute certainty if WWIII ever happens, global EMPS [they effect HUGE areas]).
Ohh that is not a bad idea at all I never would have considered it because I boycotted bluray drives but nowadays a stack of 25x 50gb double-density disks sounds pretty nice!
That's an open secret all us AI folks know: pre 2015 books are priceless always buy them if they even slightly tickle your fancy and they are affordable. Even if they don't appreciate in 'value', they, guaranteed will appreciate in value as 'data' gets more and more subsumed into nationstate and global day to day life and control and commerce etc. Being able to have a physical object in ones hands that was verifiably created by Humans is going to be a commodity in the future.
I think we are really early to something big. The large players are over-leveraged and hemorrhaging money without the income to justify it. The crazy part is even after all that spending - they simply donât have enough compute. We are either going to see further supply shortages, optimizations, or likely both.
People think this is still a chatbot.
We'll see with the relase of the new deepseek(v4?). If they keep up the trend of performant model with no nvda cgipset it will be a earthquake for all these over leveraged US companies and nvda too
The only issue I have with people pointing to deepseek is that 99.9% of people commenting it as a savior cannot actually run it and need to have a sub somewhere to do so.
The only actual benefit of deepseek etc is competition and pressure.
That said... non nvidia hardware does not automatically invalidate nvdia hardware... I mean, wtf kind of logic is that? No western country will ever invest in Chinese hardware of that capability even if it's not outright banned and it also assumes Nvidia is just going to lay back and say "oh sorry, we're done making stuff opps"
Competition is great, regardless of where it comes from, but China will never have hardware domination *in this space*.
What kills me is that NVidia has more revenue and RD investment than they could have ever imagined, do you think they are just having parties and buying lambos? Or do you think it's more probable they use those resources to continue advancing and innovating?
this kind of talk has been going on for three years now, it's just like every time someone says windows in done because linux installs are ion the rise. (that's like 30 years running now)
You're right, Nvidia has a bigger moat. and also has proved to come out with really innovating r&d. Still i would say that *if* all the ai **chatbot**(allow me,i know they do more then chatbot) companies are overleveraged it's hard to believe that Nvidia won't be affected if it eventually comes crumbling. Btw I don't see anything as the savior I just comment on what I'm seeing. I don't want to get all mad online defending products that arent mine in any form. Thanks for curbing my take with your perspective
> It is difficult to get a man to understand something when his ~~salary~~ sense of self-worth depends upon his not understanding it.
-- Upton Sinclair, basically
https://preview.redd.it/h7sokiu4qikg1.png?width=750&format=png&auto=webp&s=5b67f33d2fddfab977dd6406132bd8fa9a2317a5
\>You think it is some offline game
I've been running LLMs locally for two years, and I'll never go back to doing mundane semi-repetetive stuff myself again. It would feel like going back to programming by handwriting.
Already now, a collection of tiny models that each work well in narrow specific areas gets you several steps on your way.
Totally agree, counterproductive post. Just a screenshot of an uninformed and clearly biased person saying something uniformed and clearly biased. Reminds me of those videos from ~2015 with titles like âdestroyed with facts and logicâ.
I agree, there's a ton of 12yo anti on Reddit/Twitter. I choose to ignore them and I don't see the upside of bringing that level of discussion on this sub.
Have you actually tried running something like Mistral 7B or Llama 2 locally? They run pretty smoothly on a decent GPU these days. What specs are you working with? Might just need the right model for your hardware.
Confidently wrong, rude, and ignorant. What a combination! I can't understand how one can be just fully anti or pro something without understanding the nuances of each side. The first comment is sane and very well balanced.
If you think of the whole world as like a Kalshi/Polymarket of bad beds in the hopes of getting a dopamine payout, the massive number of insanely bad takes starts to make sense.
If you use it a lot and need privacy, investing in a few Mac Studios for 20K with GLM 5 is pretty incredible. But the compute-centers will always be needed. Especially once the bots get going. Yeah, thatâs going to happen.
During COVID, all these tech companies used the COVID money to hire people to WFH, burned stupendous amounts of money and had the biggest write off you could imagine.
Why does Bill Gates do charity? Is he a good guy? Think again, mfer wouldnât be in the Epstein Files if he was a good guy. He is a money guy, since day 1. Any capitalist is. Which is normal for that ideology.
So what is happening right now?
Imaginary idea is fueling the idea of imaginary profits, placing orders for imaginary hardware, that brings the price up to what can only be explained as imaginarily high numbers, which is âpurchasedâ in advanced with imaginary money, creating an ARTIFICIAL BUBBLE, which upon popping will kill these so called billion dollar valued âstartupsâ. Like wtf is even a startup by definition anymore.
Does ChatGPT âsolveâ anything? No it absolutely does not solve any god damn thing, same way a google search doesnât solve anything. The one thing the 2 have in common is giving their users the âsenseâ of reward (aka answer to their âqueryâ) while essentially data mining the living shit out them, creating this ridiculous profile for each human being to target them like pigs on a farm.
Google is so etched into our lives itâs disgusting, ChatGPT is the new drug on the block.
So AI is the future, and aint a damn thing you can do about it.
294 Comments
bria-87@reddit
rentprompts@reddit
Familiar-Mission7494@reddit
Evening-Fan716@reddit
vijgenmataq@reddit
constanzabestest@reddit
pigeon57434@reddit
TheIncarnated@reddit
mystery_biscotti@reddit
kdegraaf@reddit
BigGreenDinosaur@reddit
kdegraaf@reddit
Ansible32@reddit
EnoughWarning666@reddit
Ansible32@reddit
EnoughWarning666@reddit
Ansible32@reddit
EnoughWarning666@reddit
Ansible32@reddit
EnoughWarning666@reddit
Ansible32@reddit
EnoughWarning666@reddit
Ansible32@reddit
EnoughWarning666@reddit
Ansible32@reddit
muyuu@reddit
Ansible32@reddit
muyuu@reddit
Ansible32@reddit
cointegration@reddit
Ansible32@reddit
cointegration@reddit
Ansible32@reddit
cointegration@reddit
Ansible32@reddit
cointegration@reddit
Ansible32@reddit
KadahCoba@reddit
IkuraNugget@reddit
Coppermoore@reddit
Mkengine@reddit
Uiropa@reddit
Mickenfox@reddit
Uiropa@reddit
BlueKobold@reddit
Rheumi@reddit
ben_dover_deer@reddit
Crypt0Nihilist@reddit
SardinhaQuantica@reddit
Scew@reddit
Glad-Way-637@reddit
ShengrenR@reddit
Late-Assignment8482@reddit
1731799517@reddit
MrYorksLeftEye@reddit
OverfitAndChill8647@reddit
Potential-Leading817@reddit
Alternative_Earth832@reddit
Smart-Mission2275@reddit
MentalBookkeeper4603@reddit
No_Enthusiasm6313@reddit
Junior-hhhhhh9@reddit
Salty_Painting6184@reddit
wolfy-j@reddit
secret_protoyipe@reddit
wolfy-j@reddit
konovalov-nk@reddit
secret_protoyipe@reddit
nanobot_1000@reddit
AutomataManifold@reddit
Sad-Championship9167@reddit
ben_dover_deer@reddit
Sad-Championship9167@reddit
ben_dover_deer@reddit
Sad-Championship9167@reddit
ninjasaid13@reddit
juggarjew@reddit
Hunigsbase@reddit
Hot-Investigator9140@reddit
Hot-Investigator9140@reddit
Ok-Fold-4827@reddit
gamesbrainiac@reddit
Mediocre_Speed_2273@reddit
Waarheid@reddit
Hunting-Succcubus@reddit
gefahr@reddit
BlueKobold@reddit
TechnicalCatch@reddit
BlueKobold@reddit
TVL257@reddit
BlueKobold@reddit
gefahr@reddit
ErCollao@reddit
cromagnone@reddit
IrisColt@reddit
Hector_Rvkp@reddit
IrisColt@reddit
UndocumentedMartian@reddit
Glad-Way-637@reddit
sonicnerd14@reddit
Bakoro@reddit
Thick-Protection-458@reddit
SportsBettingRef@reddit
Outrageous_Media8525@reddit
JustSayin_thatuknow@reddit
Waarheid@reddit
Outrageous_Media8525@reddit
frozen_tuna@reddit
Outrageous_Media8525@reddit
ben_dover_deer@reddit
JustSayin_thatuknow@reddit
Statement-Jumpy@reddit
demureboy@reddit
cobalt1137@reddit
Corvoxcx@reddit
graymalkcat@reddit
OnlyYoung5892@reddit
Imtiredash@reddit
1998marcom@reddit
Some_thing_like_vr@reddit
seamonn@reddit
an0maly33@reddit
Jonodonozym@reddit
CommunicationOne7441@reddit
1-800-methdyke@reddit
ImJustStealingMemes@reddit
nonaveris@reddit
HighlightFun8419@reddit
charles25565@reddit
Potential-Magician66@reddit
shinkamui@reddit
Deep_Traffic_7873@reddit
Arxijos@reddit
Revolutionary_Click2@reddit
Arxijos@reddit
stumblinbear@reddit
cmndr_spanky@reddit
Bright-Awareness-459@reddit
sigiel@reddit
TheRiddler79@reddit
goyafrau@reddit
MrPecunius@reddit
mac10190@reddit
OldStray79@reddit
chanbr@reddit
OldStray79@reddit
Icy_Result_2249@reddit
_raydeStar@reddit
DirectJob7575@reddit
En-tro-py@reddit
Alarming_Turnover578@reddit
DMmeURpet@reddit
Dry-Judgment4242@reddit
CVR12@reddit
1731799517@reddit
CVR12@reddit
Devatator_@reddit
rumble_ftw@reddit
CVR12@reddit
rumble_ftw@reddit
CVR12@reddit
KissYourImoutoNOW@reddit
KillerShoaib_@reddit
AI_should_do_it@reddit
RlOTGRRRL@reddit
-dysangel-@reddit
Due-Memory-6957@reddit
-dysangel-@reddit
Due-Memory-6957@reddit
-dysangel-@reddit
_raydeStar@reddit
Mickenfox@reddit
Charl1eBr0wn@reddit
_raydeStar@reddit
Charl1eBr0wn@reddit
-dysangel-@reddit
tsoneyson@reddit
tsoneyson@reddit
Due-Memory-6957@reddit
FairYesterday8490@reddit
Late-Assignment8482@reddit
Bakoro@reddit
Late-Assignment8482@reddit
MrBabelFish42@reddit
Late-Assignment8482@reddit
Mennas11@reddit
francois__defitte@reddit
Lissanro@reddit
beedunc@reddit
Lissanro@reddit
beedunc@reddit
petuman@reddit
imnotabot303@reddit
sumptuous-drizzle@reddit
imnotabot303@reddit
sumptuous-drizzle@reddit
Rusty-Swashplate@reddit
Feeling-Currency-360@reddit
useresuse@reddit
ttkciar@reddit
o5mfiHTNsH748KVq@reddit
useresuse@reddit
muskillo@reddit
jferments@reddit
powerfulparadox@reddit
frozen_tuna@reddit
powerfulparadox@reddit
Soliloquy789@reddit
powerfulparadox@reddit
KissYourImoutoNOW@reddit
JustSayin_thatuknow@reddit
DominusIniquitatis@reddit
Artistic-Falcon-8304@reddit
Complex-Maybe3123@reddit
SuperFail5187@reddit
ben_dover_deer@reddit
SuperFail5187@reddit
No_Indication_1238@reddit
ttkciar@reddit
No_Indication_1238@reddit
DrVagax@reddit
iamkaika@reddit
Ok-Adhesiveness-4141@reddit
AlwaysLateToThaParty@reddit
MoneyPowerNexis@reddit
incoherent1@reddit
KissYourImoutoNOW@reddit
Useful_Disaster_7606@reddit
CesarOverlorde@reddit (OP)
Ulterior-Motive_@reddit
Agreeable-Market-692@reddit
Much-Researcher6135@reddit
Intrepid-Self-3578@reddit
-paul-@reddit
beedunc@reddit
Agreeable-Market-692@reddit
FunDiscount2496@reddit
asssuber@reddit
axord@reddit
WithoutReason1729@reddit
Sharp-Mouse9049@reddit
Daemontatox@reddit
geenob@reddit
ithilelda@reddit
grady_vuckovic@reddit
StewedAngelSkins@reddit
JamaiKen@reddit
username-must-be-bet@reddit
a_beautiful_rhind@reddit
jaraxel_arabani@reddit
Tyler_Zoro@reddit
Ill-Bison-3941@reddit
dazreil@reddit
Medium_Ordinary_2727@reddit
angelin1978@reddit
trolololster@reddit
phovos@reddit
CheatCodesOfLife@reddit
phovos@reddit
weyouusme@reddit
MerePotato@reddit
phovos@reddit
Mendoozaaaaaaaaaaaa@reddit
ZioniteSoldier@reddit
Burroflexosecso@reddit
Smile_Clown@reddit
Burroflexosecso@reddit
pfn0@reddit
BadAtDrinking@reddit
taeper@reddit
LustrousUmbrella644@reddit
FaceDeer@reddit
brickout@reddit
DarwinOGF@reddit
gatepoet@reddit
THEKILLFUS@reddit
Geritas@reddit
RayHell666@reddit
Bartfeels24@reddit
ttkciar@reddit
adobo_cake@reddit
defiantjustice@reddit
leonbollerup@reddit
MeLlamoKilo@reddit
weyouusme@reddit
XiRw@reddit
PlainBread@reddit
klop2031@reddit
One_Whole_9927@reddit
jeffwadsworth@reddit
Lesser-than@reddit
exaknight21@reddit
xadiant@reddit
dansdansy@reddit