Google TurboQuant running Qwen Locally on MacAir

[-]

Pidtom@reddit

Hey that’s my fork!!! Haha. Glad it’s getting use. https://github.com/TheTom/llama-cpp-turboquant

Reply

[-]

uhuge@reddit

https://github.com/TheTom/llama-cpp-turboquant/pull/45 is your only PR to llamaCpp?+)

Reply

[-]

PR #45 is going into my fork, not upstream llama.cpp (214 commits merging into the turboquant branch (most of those files are catching up with master)). the community as a whole is discussing and converging on the implementation in the main discussion thread: [https://github.com/ggml-org/llama.cpp/discussions/20969](https://github.com/ggml-org/llama.cpp/discussions/20969) that particular PR is weight compression on top of the KV cache work. TQ4\_1S compresses the model weights themselves so larger models get physically smaller on disk and in VRAM (28-37% smaller depending on config). still verifying things with CUDA testers: [https://github.com/TheTom/llama-cpp-turboquant/pull/45](https://github.com/TheTom/llama-cpp-turboquant/pull/45) as for upstream, i am new to the llama.cpp community so i only have one official PR up for review so far (#21119, sparse V skip). they have a lot of contributions coming in and i want to respect their process and code of conduct. the fork is where the experimental work lives until it is ready.

Reply

[-]

AppealThink1733@reddit

Is this already in lllama.cpp?

Reply

[-]

uhuge@reddit

https://github.com/TheTom/llama-cpp-turboquant/pull/45 is in draft, weirdly, for more idea check his account activity.

Reply

[-]

FastDecode1@reddit

There's a PR for an initial, CPU-only implementation: https://github.com/ggml-org/llama.cpp/pull/21089 I've also seen multiple GPU implementations, but none of them have been submitted as PRs as yet.

Reply

[-]

eugene20@reddit

Not officially, but you can try one of the implementations in [https://github.com/ggml-org/llama.cpp/discussions/20969](https://github.com/ggml-org/llama.cpp/discussions/20969)

Reply

[-]

AppealThink1733@reddit

Thank you, now we just have to wait and see how long it will take to be implemented natively in llama.cpp. Will it take long or not?

Reply

[-]

ufoolme@reddit

You can compile the implementation now and run them now. I’d be surprised if something doesn’t get to main before the end of week, but it sounds like there is some optimisation that can be done after this as well. Hopefully more innovation is possible, it has moved the needle nicely.

Reply

[-]

Tatrions@reddit

20K context on a base MacBook Air is impressive. the fact that TurboQuant makes this feasible on 16GB without swapping means a lot of use cases that previously required cloud APIs could move local. curious what the quality degradation looks like at that compression level compared to standard Q4 on the same model.

Reply

[-]

BreakfastAntelope@reddit

For context for us noobs, how big of an improvement is this from what was running before?

Reply

[-]

gladkos@reddit (OP)

non turboquant simply failed with 20K tokens input

Reply

[-]

BreakfastAntelope@reddit

Thank you :)

Reply

[-]

spky-dev@reddit

For context noobs, 20k is very little context. For reference, the system prompt in Claude Code is like 12k.

Reply

[-]

JsThiago5@reddit

I would say 128k is the minimum for agents

Reply

[-]

gladkos@reddit (OP)

20K is pretty enough for basic tasks. The more is better, ofc.

Reply

[-]

TechnicSonik@reddit

basically went from impossible to possible

Reply

[-]

gladkos@reddit (OP)

one year llm's will be in every laptop

Reply

[-]

nullmove@reddit

^How the fuck can no one tell this is a bot account? Incredible.

Reply

[-]

themixtergames@reddit

Yep, [I called out another account earlier](https://www.reddit.com/r/LocalLLaMA/comments/1s55xox/comment/ocswvgu/). This is my "rubric": * New account. (Doesn't apply here) * Use of the word "curious". (YES) * starting sentences with lower case. (YES) * Multiple comments starting with the word "the". (YES) * Question at the end. (NO)

Reply

[-]

nullmove@reddit

And overall on a semantic level they add no insight of their own, just regurgitates OP vacuously. (YES) Nevertheless I remain most astonished by the fact that, in this sub of all places you would really expect people to be perceptive of these patterns. It's not like these bots are using using some top of the line model. Yet these comments are highly upvoted and often engaged by actual people. I shudder to think what a boomer platform like Facebook now looks like.

Reply

[-]

AgitatedHearing653@reddit

Hang on, gotta look up the word vacuously lol

Reply

[-]

Hans-Wermhatt@reddit

I think that’s a bit of an overreaction… I feel like most people don’t care that much. This sub is about engaging with LLM outputs. Most of the posts are written by Claude or another LLM in this sub if you haven’t noticed.

Reply

[-]

themixtergames@reddit

Brotherman, they made 50 comments within 3 hours. We are talking about a specific type of comment that is not saying anything at all, it's just using LLMs for regurgitating what was already said for larping or karma farming purposes. There's a difference between that and using Claude to fix your grammar, format something better.

Reply

[-]

JsThiago5@reddit

But it's pointing to something interesting: the degradation compared to other quantizations. Even when Q4 is not the standard quantization on llamacpp to KV, fp16 is. it's a valid thing to ask

Reply

[-]

somersetyellow@reddit

I mean, it's legitimately getting harder and and harder to spot. I've been trying to stay up on the latest patterns but even I missed that one. Mostly because I wasn't in "spot the LLM" mode. When you're in for a mindless evening scroll these comments can slip through easily... They're definitely flooding reddit though. That experiment on ChangeMyView where nobody noticed the LLM's until they were disclosed was a wake up call. It was then especially wild to watch the general reddit reaction to be outrage that researchers would use AI on them without their permission. Feels like watching people in a coal mine get outraged over finding a dead canary. Who would ever kill a canary!

Reply

[-]

Long_comment_san@reddit

Wow 😭

Reply

[-]

AgitatedHearing653@reddit

Right?

Reply

[-]

LocoMod@reddit

Curious…curious…curious…stupid ass bot. You don’t understand curiosity. Which is why everything is curious. Downvote the slop. It gives AI a bad name. These low effort gits won’t learn otherwise.

Reply

[-]

the__storm@reddit

Is it? I can run Qwen3.5-9B Q4_K_XL with 200,000 tokens fp16 context on my 16 GB 7800 XT, with memory to spare. If I turn the context down to 20,000 tokens (still fp16), I have **7 GB** free. Ngl it feels like this thread is 80% bots talking to bots.

Reply

[-]

stddealer@reddit

I'm using 65k tokens CTX (q8) for Qwen3.5 27B and I came close to filling up the context completely only once so far.

Reply

[-]

JsThiago5@reddit

Yeah, it is. try to run something that at least you can trust as an agent, like Qwen 3.5 27b or 122b, and keep this context to see what happens.

Reply

[-]

gladkos@reddit (OP)

nah I'm a skin bag with bones)

Reply

[-]

cksac@reddit

you can now run larger model too. Aplied the idea to [weight compression](https://www.reddit.com/r/LocalLLaMA/comments/1s51b5h/turboquant_for_weights_nearoptimal_4bit_llm/), it looks promosing.

Reply

[-]

gladkos@reddit (OP)

Can’t wait for the M5 Mac Mini to try this! Feels like local models are going to blow up this year

Reply

[-]

M5_Maxxx@reddit

Compression is only for context or also the model?

Reply

[-]

PrashantRanjan69@reddit

Model compression already exists, it's called quantization (the q4, q8, etc that we generally run on our local systems). TurboQuant essentially takes that idea and applies it to context (data that constantly changes, unlike the model weights that are all already there, also known as the kv cache).

Reply

[-]

x3haloed@reddit

Yeah, but this method is lossless.

Reply

[-]

JsThiago5@reddit

The KV context quantization also already existed before TurboQuant. You can set up llamacpp using -ctk or -ctv q8\_0, q5\_0, q4\_0, etc. What TurboQuant did, I think, please check to see if I am wrong, is to reduce to 3 bits but keeping the intelligence (the q4 is bad at it)

Reply

[-]

Leo_hofstadter@reddit

Is there a way to use this TurboQuant in LM studio?

Reply

[-]

FlashyBook@reddit

I am doing some testing of this, alongside a project called Greenboost. It looks like it works, but I'm not seeing any benefits yet, probably due to inappropriate tests.

Reply

[-]

Leo_hofstadter@reddit

All the best!

Reply

[-]

BillDStrong@reddit

Kinda. What it actually did is really look at the format of the data and find a way to transform that data to a much smaller size without losing any/most of its original information. It does this by fundamentally changing the format of the data. The q variants, on the other hand, did something that is much more brute force, it directly changed the data itself to fit in less space. Now, this new method is not lossless, but it is very close to lossless. The q variants are very lossless, and more so the lower the quant. If you were to compare them to image formats. the q variants woould take a full color image and make them directly 16 color images. You can make out what the image was, but you lose a lot detail. If that detail was the point you are looking for, you are out of luck. What this does is much closer to visually lossless images, you change the format of the data by processing it so that it still "appears" the same to the observer, in this case the LLM using it, but it is much smaller in size. The LLM can still find that missing information, because it is still something it could originally find.

Reply

[-]

Green-Ad-3964@reddit

I like this metaphor, that few will understand in 2026: it's the HAM (amiga image format) for context data. Ps, when you said "q variants are very lossless" you mean lossy, do you?

Reply

[-]

BillDStrong@reddit

I meant very close to lossless. I was missing a word, lol. I was thinking png, which has a lossless version and the versions that use LUTs for specific number of colors, which would still be applicable today, but honestly, the HAM format would be a closer analogy, since png using LUTs will massage the data with shading to mimic the lost data, but current q compression, to my understand, don't do that. This makes me feel old, lol.

Reply

[-]

stddealer@reddit

TurboQuant is rotating the keys and values to try to find an orientation that minimizes the quantization error. I works because attention relies on dot product wich is not affected by rotation. At runtime, the queries must be rotated the same way as the keys, and the attention weights must be rotated the same way as the values, and everything should work the same.

Reply

[-]

PrashantRanjan69@reddit

You are correct! TurboQuant is an optimization of what llama.cpp KV cache quantization does. Llama.cpp KV cache uses a block based quantization, meaning each block of values requires storing a full-precision scale constant to restore the approximate original value. This scale constant was an overhead because of which the compression was not as efficient. TurboQuant uses a polar quant method (which is a technique already used in model weights quantization by, for example, GGUF format), essentially removing the need of those scale constants to be stored. It also implements some kind of 1-bit error correction step (I am yet to read about that).

Reply

[-]

DrNavigat@reddit

Give this man an Oscar 👈

Reply

[-]

BlobbyMcBlobber@reddit

To be fair quantization is not compression. It's truncating the bytes. It's literally throwing out information which causes you to lose precision.

Reply

[-]

stddealer@reddit

It's lossy compression

Reply

[-]

jock-83@reddit

Technically speaking, they are very different concepts and misuse may cause bad interpretations. For example, the most important step in JPEG image compression - which you can control setting the "quality" parameter - is quantization of the samples; yet quantization does not achieve any compression by itself, until you pair with entropy encoding.

Reply

[-]

stddealer@reddit

Technically different, maybe, but very different, really? Quantization is just selectively throwing away the least significant bits of information. When you quantize any array of numbers, you end up with an approximation (with rounding errors) of the original data that takes fewer bits of storage. That's quite literally what lossy compression means. Quantization is compression, but of coursethere is more to compression than just quantization.

Reply

[-]

BlobbyMcBlobber@reddit

Yes they are different, because in compression you can have a way to restore the original data (not all compression is lossy). Throwing out bits is not compression. It's truncating. Let's take a book for example. Let's say you can paraphrase the book and keep the exact same information using less pages. That's compression. Contrary to this, you can just remove the last 20 pages which also achieves a lower page count, but you are throwing out information with disregard.

Reply

[-]

stddealer@reddit

Ys. Yu cld ls thrw wy ll th vwls, nd t wld stll b smwht rdbl. You can throw away information with disregard if it's less relevant information. The end of the book contain very relevant information that cannot be discarded without taking away the meaning of the book. Quantization selectively removes the least significant bits of each number. If it took away the most significant bits, or the last few numbers of the tensors instead, it would completely break the models.

Reply

[-]

BlobbyMcBlobber@reddit

It does remove the least significant bits, but depending on _how many_ you remove, it can have a huge effect. Going from BF14 to INT4 is a big difference even if it's the least significant bits. Either way, this is not compression because it's literally throwing away the information. It's truncating.

Reply

[-]

stddealer@reddit

Jpeg is also throwing away some information. If you're too agressive with the "quality" parameters, you will also start losing relevant parts of your image. Since training a neural network is a stochastic model, you can argue that the weights are inherently noisy, so truncating some of the least significant bits mostly filters out the noise without really affecting the relevant data. It sounds like you're arguing that lossy compression is not really compression. Wich is a reasonable point, that's why I'm insisting on the "lossy" part.

Reply

[-]

Zestyclose_Yak_3174@reddit

K/V compression but it can in theory speedup inference on higher contexts.

Reply

[-]

Adventurous_Pin6281@reddit

it requires much less ram

Reply

[-]

slippery@reddit

I wonder the same thing.

Reply

[-]

M5_Maxxx@reddit

Yup context only

Reply

[-]

Frexuz@reddit

Pls add a search bar in the models list, horrible to find stuff

Reply

[-]

marcusalien@reddit

It looks like this video has been sped up... Note the animation has been sped up, not just the token output

Reply

[-]

SubstantialAd2190@reddit

Isn’t the Llm on flash from Apple

Reply

[-]

Enthu-Cutlet-1337@reddit

"a bit slow" is doing a lot of work in that sentence. what's actual tok/s at 20k context fill? curious if TurboQuant's perplexity hit at equivalent bpw beats standard k-quants or just trades differently.

Reply

[-]

shin2_d@reddit

I need to try this on my M2 Air

Reply

[-]

ZiradielR13@reddit

Running turboquant ? You mean what the turboquant paper described because I haven’t seen any official release from google yet. Must be running communityquant

Reply

[-]

Dorkits@reddit

That's amazing. My 8gb VRAM can do more now :)

Reply

[-]

gladkos@reddit (OP)

It takes just 1GB memory. My guess core matters more here.

Reply

[-]

robberviet@reddit

How qwen 3.5 9b took 1GB?

Reply

[-]

jadhavsaurabh@reddit

Yes same question here if its true how much speed we are getting

Reply

[-]

jungle@reddit

That's what puzzles me about this. You could already run a 9B model in a 16 GB Mac without issues. I'd be more impressed if it was a 32B model.

Reply

[-]

Amazing_Ad1157@reddit

MoE architecture, my friend!

Reply

[-]

hurdurdur7@reddit

Qwen 9b is a dense model. For any quality worth running you need Q5\_K\_M at least , so you're looking at 6GB of weights themselves. TurboQuant only affects the context cache kv that comes on top of this ...

Reply

[-]

v01dm4n@reddit

9b is not moe as far as I understand. Its a dense model. Qwen 3.5 35b is a3b and will run faster than a 9b, but needs more mem.

Reply

[-]

marketingagentdotio@reddit

Nice. .. running capable models on consumer hardware collapses the deployment cost curve. We run a multi-agent orchestrator on Mac Mini and the bottleneck shifted from model capability to I/O throughput once 4-bit got good enough. TurboQuant hitting these benchmarks on MacAir means the minimum viable inference box just got a lot cheaper (seems...perhaps?). Following...

Reply

[-]

AsozialerVeganer@reddit

What use cases are you tackling with this setup? Judging by your username, is it mostly marketing automation?

Reply

[-]

BeeNo7094@reddit

Any inference (mostly tg I guess) speed improvements? Can you benchmark turbo vs non turbo?

Reply

[-]

FullEquivalent9180@reddit

20k context is laughable. I need 200k minimum

Reply

[-]

FormalAd7367@reddit

Suddenly running a Kimi locally wouldn’t a dream good for those who have their AI agents

Reply

[-]

Sakatard@reddit

So fucking cool

Reply

[-]

CultivatingPlant@reddit

M5 mac mini sales 📈

Reply

[-]

blackhelio@reddit

WWDC dates is out, I am ready to pre-order. I pretty much expect some tariff inflation along with the new chip update.

Reply

[-]

MrSteezyMcSteez@reddit

We are gonna hear some insane stats on this Mac Studio they’ll surely announce at WWDC

Reply

[-]

last_llm_standing@reddit

google doing apple's job

Reply

[-]

NCpoorStudent@reddit

More like Windows plastering with ads and Nvidia is interested in only bank vaults.

Reply

[-]

math2020@reddit

Nice. I will have to try it

Reply

[-]

ihllegal@reddit

Can this be done on windows?

Reply

[-]

dcarrero@reddit

Pero usas atomic?

Reply

[-]

unknown_neighbor@reddit

Some awesome released the code and benchmarks https://github.com/0xSero/turboquant check it out guys

Reply

[-]

M5_Maxxx@reddit

I was really excited but also weary of Malware, I told Claude to audit this: Here's the truth. It's a reskinned [Jan.ai](http://Jan.ai) with minimal changes: What they actually did: \- Renamed "Jan" → "Atomic Chat" (find and replace) \- Changed the app icon \- Tweaked the UI setup screen and chat input \- Bundled a "turboquant" llama.cpp backend fork \- Updated build scripts for macOS signing/DMG \- Updated README/CONTRIBUTING docs \- Added a PDF file reader \- KV cache default changed to "turbo3" What they didn't do: \- No new inference engine \- No new model architecture support \- No MLX improvements \- No performance optimizations beyond what Jan already had \- No novel features It's literally [Jan.ai](http://Jan.ai) with a new coat of paint and a custom llama.cpp build ("turboquant"). The 96 commits include the initial Jan codebase dump, the rename, and mostly CI/build pipeline changes. Not worth benchmarking against LM Studio — it's just Jan with a different name. Want me to clean up the worktree and delete it?

Reply

[-]

slypheed@reddit

Thanks for calling this out. At the _very least_ they could have added Jan in the list of acknowledgements at the bottom...

Reply

[-]

Pidtom@reddit

Or you know, the guy they got TurboQuant from.

Reply

[-]

slypheed@reddit

seriously. Separating Signal from Noise will always be the real challenge and all that really matters in the end.

Reply

[-]

Pidtom@reddit

Janai with my fork of TurboQuant :) https://github.com/TheTom/llama-cpp-turboquant

Reply

[-]

gladkos@reddit (OP)

We’re not hiding. GUI is forked from Jan, however llama.cpp is patched along with gui to work together. Everything is open source. I benchmarked 20K context against non Google TurboQuant, it simply crashed. The same will likely happen with LM Studio.

Reply

[-]

punkgeek@reddit

Why change the name though? Saying "we improved jan.ai here's our fork" would be good behavior. Weasel changing the name to something else is dishonest.

Reply

[-]

jonydevidson@reddit

Because Jan's name is Jan's trademark. You can fork any permissive and copy left software and redistribute it but you should both strip the trademarks and keep the original notices. The notices themselves should tell you the source. The name change is a legal matter, you don't want to step on any toes.

Reply

[-]

punkgeek@reddit

I think we can safely say the desire to not mention where 99% of the code for this project came from (Jan) is highly relevant to why they removed the name and deleted the git history when they copied this code. ;-) Yes, legally the license allows it but IMO still a weasel move. I think it is informative to compare the forthright way Jan.ai handled things in their README: > Apache 2.0 - Because sharing is caring. Acknowledgements Built on the shoulders of giants: - [Llama.cpp](https://github.com/ggerganov/llama.cpp) - [Tauri](https://tauri.app/) - [Scalar](https://github.com/scalar/scalar) To the way this clone of a project handled things - their README doesn't mention anything about Jan (even though >90% of the code came via that project and work of those developers). They also searched and replace the code to remove all mentions of jan and deleted all git history from their fork (which really hurts mergability). IMO just generally sleazy.

Reply

[-]

jonydevidson@reddit

> Yes, legally the license allows it but IMO still a weasel move. Your judgement doesn't matter, if they included Jan's copyright and MIT notice, that's all good, that's what open source is meant to be. How they vendor Jan is up to them and speaks more about their vision on maintaining this. Fairness has nothing to do with it, Jan's license doesn't want attribution, just the inclusion of the copyright and license notices. It's more about that it would have been way easier to keep up with upstream Jan by using it as a forked submodule, or forking Jan and renaming the repo. However, that's just my opinion based on my experience, I won't presume to know where the dev wants to take this.

Reply

[-]

gfxd@reddit

A fork can have a different name, nothing wrong as long as the attribution is there. If you are doing a major rewrite of a project, you can and must rename it so that there is no confusion.

Reply

[-]

punkgeek@reddit

They specifically removed any attribution from the README.

Reply

[-]

ahhteaahh@reddit

Did you hear about Antigravity and Cursor? You will love their story.

Reply

[-]

EvolvingSoftware@reddit

You mean VS Code?

Reply

[-]

punkgeek@reddit

other people being bad is not an excuse to also be bad. ;-)

Reply

[-]

coder543@reddit

The name is the one thing a fork typically can't keep due to trademark reasons.

Reply

[-]

punkgeek@reddit

Though reading their web page they seem to be actively trying to hide this relationship. No prominent thanks or credit given. I've worked on a lot of big open-source projects and IMO this is super dishonest.

Reply

[-]

coder543@reddit

Sure, I agree that part could be better. The entire website looks like it was vibecoded, and the agent never thought to ask if credit should be given. The more suspicious thing is the squashing of the git history. But, the LICENSE does still reference Menlo. The license doesn't require prominent credit be given. If that's what the developers wanted, then some would argue they should have chosen a better license.

Reply

[-]

punkgeek@reddit

If the devs of this fork were more honest, this whole thing would instead be a set of PRs sent up to jan.ai (which seems to have a vibrant and friendly dev community and an enormous userbase in their discord).

Reply

[-]

coder543@reddit

Wanting to maintain a fork is now "dishonest"? No... you don't understand open source, despite claiming to have worked with a lot of big open source projects. People are allowed to fork, and they are not required to try to upstream their changes. This is perfectly healthy. Nothing prevents Jan from upstreaming Atomic's changes themselves.

Reply

[-]

punkgeek@reddit

> People are allowed to fork, and they are not required to try to upstream their changes. This is perfectly healthy. Nothing prevents Jan from upstreaming Atomic's changes themselves. Though the (malicious?) mass string replace in their first commits was clearly not designed to encourage reuse by other projects. ;-)

Reply

[-]

coder543@reddit

> Though the (malicious?) mass string replace How exactly do you propose that they rename the project to comply with trademark requirements without a mass string replace? It's like you've never seen a fork before. If they tried to ship this project under the name Jan without permission, _that_ would be sketchy and likely malicious.

Reply

[-]

punkgeek@reddit

(jan.ai is not trademarked)

Reply

[-]

coder543@reddit

It doesn't matter if you think it is trademarked or not. It doesn’t even matter whether the mark is federally registered. In the U.S., trademark rights can arise from actual commercial use, not just registration. Registration mainly strengthens and formalizes those rights; it isn’t what creates them in the first place. I'm not your lawyer, though, so talk to a lawyer if you need trademark advice. Do you know the difference between ™ and ®? Yeah, that's relevant here. The fact that you talk so flippantly about stuff you don't understand is baffling, but this is just how Reddit works. Even setting aside the trademark issue, how do you think they should rename the product to ***avoid*** **falsely claiming to be Jan** without a mass string replace? Impersonating the project is a horrible thing. Forking is a perfectly normal part of the open source world. As I said, it's like you've never seen a fork before. Don't talk so authoritatively about something you clearly have little experience with. You can ask your AI agent to show you some examples of historically significant hard forks in open source history. Blocking you since you are downvoting actual facts and continuing to argue about things that you're unwilling to admit being wrong about.

Reply

[-]

coder543@reddit

It doesn't matter if you think it is trademarked or not. It doesn’t even matter whether the mark is federally registered. In the U.S., trademark rights can arise from actual commercial use, not just registration. Registration mainly strengthens and formalizes those rights; it isn’t what creates them in the first place. I'm not your lawyer, though, so talk to a lawyer if you need trademark advice. Do you know the difference between ™ and ®? The fact that you talk so flippantly about stuff you don't understand is baffling, but this is just how Reddit works. Even setting aside the trademark issue, how do you think they should rename the product to not **falsely claim to be Jan** without a mass string replace? Impersonating the project is a horrible thing. Forking is a perfectly normal part of the open source world. As I said, it's like you've never seen a fork before. Don't talk so authoritatively about something you clearly have little experience with. You can ask your AI agent to show you some examples of historically significant hard forks in open source history.

Reply

[-]

coder543@reddit

Wanting to maintain a fork is now "dishonest"? No... you don't understand open source, despite claiming to have worked with a lot of big open source projects. People are allowed to fork, and they are not required to try to upstream their changes. This is perfectly healthy.

Reply

[-]

BusRevolutionary9893@reddit

I didn't trust him as soon as he mentioned Openclaw.

Reply

[-]

met_MY_verse@reddit

I actually thought this *was* Jan until I read this and checked again.

Reply

[-]

punkgeek@reddit

their entire git webpage also seems to be a copy/reskin of the jan page.

Reply

[-]

ZippySLC@reddit

"and I would have gotten away with it if it weren't for you meddling kids!"

Reply

[-]

No_Run8812@reddit

Just benchmarked DeepSeek R1 70B on M3 Ultra 512GB — the KV cache alone takes 40GB at 128K context. TurboQuant, bringing that down to \~7GB would be huge for running multiple models simultaneously. Anyone know if the llama.cpp PR supports Metal yet?

Reply

[-]

jonydevidson@reddit

Does it do smart caching like oMLX?

Reply

[-]

whoisyurii@reddit

So basically if I download your app, then TueboQuant is already applied to models I choose?

Reply

[-]

Left_on_Pause@reddit

How much easier does TurboQuant make it to put more advanced reasoning and faster processing into a much smaller and cheaper device? Say, a missile or an autonomous drone? How about a autonomous warehouse bot?

Reply

[-]

iansltx_@reddit

Anyone got a read on quality and bpw? For 3 bpw would this be comparable to a q4 model or better than that?

Reply

[-]

runsleeprepeat@reddit

I gave the tonbistudio variant a try and compared it with q8 and q4. See: [https://github.com/tonbistudio/turboquant-pytorch/issues/6](https://github.com/tonbistudio/turboquant-pytorch/issues/6)

Reply

[-]

esuil@reddit

Might still be worth it, but looks like deterioration is quite severe. Thanks for the link. Any chance you could run same tests with scos-lab findings and suggestions for implementations?

Reply

[-]

runsleeprepeat@reddit

There are so many implementations in parallel at the moment, it is tough to keep up to the latest findings. Best is to give it a try yourself. I'm focussing now on the TheTom implementation which looks like everything is combined there (metal, Cuda, rocm).

Reply

[-]

the320x200@reddit

Summary is that it's a little smaller but also a bit worse than normal quants?

Reply

[-]

AnonLlamaThrowaway@reddit

It definitely makes me wonder if you could just add one extra bit to TurboQuant and get rid of that problem while keeping most of the compression gains

Reply

[-]

Express_Grocery_4707@reddit

Sorry for the dumb question, but what does BPW stand for?

Reply

[-]

AnotherAvery@reddit

bits per weight

Reply

[-]

Express_Grocery_4707@reddit

Thanks!

Reply

[-]

stddealer@reddit

Nope, for 3 bpw, this would be comparable to a good 3 bpw, maybe 3.5 bpw at most. The 4.0625 bpw TurboQuant performs slighty worse than ggml Q4\_0 wich is 4.5 bpw.

Reply

[-]

YourNightmar31@reddit

Any windows app where i can play around with turboquant?

Reply

[-]

kamize@reddit

I don’t understand why isn’t this just a pr for jan ai?

Reply

[-]

LittleCelebration412@reddit

Looks great! Been meaning to try out turboquant on my Macbook

Reply

[-]

PathIntelligent7082@reddit

correct me if i'm wrong, tbut hat model runs normally on m4 with 16gigs of ram?

Reply

[-]

EvolvingSoftware@reddit

Yeah it does. But you wouldn’t normally get that big a context window going

Reply

[-]

stddealer@reddit

With q4 kV cache you would. I believe the claim is that TurboQuant is better at maintaining performance while quantizing KV cache compared to GGML quants of same size, but I've yet to see evidence of that.

Reply

[-]

stddealer@reddit

Update: I found this: https://preview.redd.it/k2y4zan6krrg1.png?width=1440&format=png&auto=webp&s=3b08d93c2ef7f94e8b77b05c99c527b1361caa99 (from https://github.com/ggml-org/llama.cpp/pull/21089) Looks like TurboQuant isn't significantly more "lossless" than the existing (3 years old) quantization schemes (unless it's somehow keeping performance in other ways that don't affect KLD)

Reply

[-]

a_beautiful_rhind@reddit

Edge? On that graph it is higher. Higher is bad.

Reply

[-]

stddealer@reddit

Yes but it's also more to the left. Left is good. The data points seem to follow some kind of hyperbole curve, and the tq4 point is clearly below that curve, hard to say for tq3.

Reply

[-]

a_beautiful_rhind@reddit

Left is slightly smaller. The scale is from 10-20mb. You'd want worse quality to save 1-3mb?

Reply

[-]

stddealer@reddit

It's barely worse. It's more smaller than it is worse.

Reply

[-]

ReturningTarzan@reddit

This is the right take, really. TurboQuant isn't even new, it's from April 2025 and didn't cause a stir back when the paper was released because it's not a technique designed for online K/V cache compression. It's meant for vector databases, and it's only "turbo" in comparison to other vector DB quantization schemes that use expensive clustering clustering algorithms. It's only the blog post that advertised it as a revolutionary new *online* K/V quantization scheme, and they're not basing that on anything from the paper. In fact the claims aren't sourced at all. The VRAM requirement for K/V caching is a well understood problem. To mitigate it, researchers and developers have come up with *many* different techniques over the years, many of which are in regular use today: - Grouped Query Attention (GQA): Reducing the number of key/value pairs and assigning the same key head to multi query heads reduces the cache size by a factor of about 8 before you start to lose quality - Multi-headed Latent Attention (MLA): As used by DeepSeek etc., cache a compressed latent state that maps back to keys/values in real time. Reduces the cache size by a factor of 10 to 20 in practice - Linear attention: Get rid of the K/V cache altogether, at least for most layers. 100% VRAM reduction in the limit (uses a fixed-size recurrent state instead) - Quantization: Store keys/values in lower precision. Bunch of different takes on this, some claiming even better compression than TurboQuant. In practice, many are already roughly equivalent already, as your chart illustrates That's not to say TQ isn't an improvement *in some ways*. It's just a small, incremental one, as your chart suggests. It also doesn't come for free. The blog post says "zero overhead" but the paper makes it clear that this is talking about *storage* overhead and it's comparing to Product Quantization and RabitQ, not to commonly used online techniques like the methods used in llama.cpp already. Essentially they say "with this, your vector database will be more precise than PQ or RabitQ without needing more space, and you can build your index faster."

Reply

[-]

stylehz@reddit

Hey OP could share the hugging face model page? Thanks

Reply

[-]

FishDeenz@reddit

Atomic chat app seems cool, playing around with it now!

Reply

[-]

a_beautiful_rhind@reddit

Did llama.cpp not support q4 cache on macbooks? Going from like 4 bit to 3 bit context did that much for you? With nobody writing any PPL/KLD numbers or comparing to anything else? The ones I saw in ik_llama github issues were less than exciting.

Reply

[-]

JsThiago5@reddit

Turbo is 3 bits but keep coherence of the model, compared to Q4, which degrades it too much to be useful

Reply

[-]

a_beautiful_rhind@reddit

Proof? All the PPL tests people are running it comes out higher than on Q4.

Reply

[-]

gladkos@reddit (OP)

heard 3bit was quite poor, decided not to go

Reply

[-]

lemon07r@reddit

Isnt turboquant 3bit?

Reply

[-]

Odd-Ordinary-5922@reddit

3 and 4 bit

Reply

[-]

lemon07r@reddit

neat

Reply

[-]

gladkos@reddit (OP)

depends on the initial model. we took 4b

Reply

[-]

Ok-Drawing-2724@reddit

Running 20k context on a MacBook Air is impressive. If you’re planning to use it for OpenClaw, ClawSecure helps check for any risky behaviors first. Keeps things simple and safe. Which Mac model did you test it on exactly?

Reply

[-]

AcePilot01@reddit

Now show it in real time and not sped up lmfao. Guarantee you that was a 20 min think at least. lol

Reply

[-]

Substantial-Cost-429@reddit

this is wild. running large context on macbook air was basically impossible before. turboquant is really changing the local ai game we been building out ai setups and configs for running models efficiently and this is exactly the kinda thing the community needs. just crossed 100 stars on our repo [github.com/caliber-ai-org/ai-setup](http://github.com/caliber-ai-org/ai-setup) this week which has a bunch of local model setup guides too if anyone tries this on older M chips would love to hear how it goes. also come hang in our discord [discord.com/invite/u3dBECnHYs](http://discord.com/invite/u3dBECnHYs) if u want to compare notes on local setups

Reply

[-]

Substantial-Cost-429@reddit

this is massive for local AI coding setups. being able to run a 9B model with good context window on a regular MacBook Air changes the whole accessibility story. most people dont have a beefy GPU box, they got a laptop and this kinda compression makes local LLM coding actually viable for everyday devs. super interesting experiment

Reply

[-]

TopTippityTop@reddit

Is it actually lossless in terms of quality of output?

Reply

[-]

Regular-Forever5876@reddit

Almost, less then 1%. I am writing a full review on my blog right now 😁

Reply

[-]

Feeling_Ad9143@reddit

I am using qwen3.5:9b with 32Kb context on 5070 12Gb. It would be awesome to use 128K context instead.

Reply

[-]

Slasher1738@reddit

Need it in lm studio

Reply

[-]

v01dm4n@reddit

u/neilmehta24 when? :)

Reply

[-]

Slasher1738@reddit

Probably when it goes official in llama.cpp

Reply

[-]

punkgeek@reddit

which should take only about a week.

Reply

[-]

CallOfBurger@reddit

Hehe I'm happy to have bought a RTX 3090 on a whim. It ran Qwen 3.5:27b smoothly but now the bigger model will work too :D

Reply

[-]

eugene20@reddit

Try [rotorquant ](https://www.reddit.com/r/LocalLLaMA/comments/1s44p77/rotorquant_1019x_faster_alternative_to_turboquant/)next 😄

Reply

[-]

Voxandr@reddit

He is not developer, he just steal other people work for fame

Reply

[-]

ThiccStorms@reddit

wow!

Reply

[-]

WithoutReason1729@reddit

Your post is getting popular and we just featured it on our Discord! [Come check it out!](https://discord.gg/PgFhZ8cnWW) You've also been given a special flair for your contribution. We appreciate your post! *I am a bot and this action was performed automatically.*

Reply

[-]

PANIC_EXCEPTION@reddit

This is gonna be a beast when it eventually gets ported to MLX Unfortunately that seems to be at the very end of their published roadmap, but it will happen eventually

Reply

[-]

EvolvingSoftware@reddit

https://x.com/prince_canuma/status/2036611007523512397?s=46&t=dUCVh9akIWxxNUIkrDJwJg he did it just not sure if he’s releasing it

Reply

[-]

abhishek_satish96@reddit

Were you able to run any benchmarks and confirm the quality loss if any?

Reply

[-]

gladkos@reddit (OP)

tested multiple prompts and got similar results. Google claims 90% lossless, we’ll see

Reply

[-]

sordidbear@reddit

> 90% lossless So is it lossless or lossy? The little bit I've read says no quality loss.

Reply

[-]

ANR2ME@reddit

If it's not 100% accurate then it's lossy 😅

Reply

[-]

gladkos@reddit (OP)

almost no quality loss

Reply

[-]

Jethro_E7@reddit

What does this mean exactly? What if we have 12gb vram gfx 3600?

Reply

[-]

-_Apollo-_@reddit

How many tokens could you fit without kvcache quant before? What about at q8 kvcache?

Reply

[-]

Trysem@reddit

Nvidia & Windows Supposed to be, but Google and Apple took over

Reply

[-]

Cunnilingusobsessed@reddit

What was all involved in patching llama.cpp? Im sure that wasn’t all that straight forward?

Reply

[-]

kiwibonga@reddit

Knowing that there was a functional PR on github the day it was announced, I assume they yanked that.

Reply

[-]

punkgeek@reddit

yeah - mostly they just stole other people's work and tried to (mostly) pass it off as theirs.

Reply

[-]

iamapizza@reddit

Could you link to the pull request, any idea if it'll work with gpu too?

Reply

[-]

Awkward_Sympathy4475@reddit

How does it help in image generation, does quality improve or speed

Reply

[-]

gladkos@reddit (OP)

didn't try with images yet. major improvement comes with large context requests.

Reply

[-]

jrlomas@reddit

Am I the only one that thinks this is about as good quantization? I mean this whole compression system works well because it implies some combination of: 1. there is a lot of redundancy in the information 2. the entropy of the high precision fs16 is mostly noise I am not saying this isn't an improvement, but I'm curious IF Q3 for KV cache works just as well for most models without any additional computation from the compression mechanism. Would love to see an implementation of this algorithm instead "KV Cache is 1 Bit Per Channel" [https://arxiv.org/abs/2405.03917](https://arxiv.org/abs/2405.03917)

Reply

[-]