RipperFox

> Cuda 13.2 > llama.cpp Are you aware of the problems of this constellation, e.g. mentioned here: https://www.reddit.com/r/unsloth/comments/1sgl0wh/do_not_use_cuda_132_to_run_models/ https://github.com/ggml-org/llama.cpp/issues/21255 https://github.com/unslothai/unsloth/issues/4849 https://huggingface.co/unsloth/gemma-4-31B-it-GGUF/discussions/12

Interesting strategy, calling someone a nerd (which I am) in 2026 - ok ok, I'll touch some gras.. I bet I'm even older than you - Casser la croûte and have a nice day! :)

vLLM can be better on a single 5090 if you don't want to wait until llama.cpp catches up with it's experimental forks - and your completely random (Vulkan, seriously?) docker example is clearly badly formatted AI slop suggesting running docker - who uses/needs docker for llama-server router mode anyway?

Ofc router mode is "nice" - but llama-swap can even switch to vLLm, SGlang, etc. Your AI generated example sucks btw.

Did you know that you can ask you local model that kind of questions, too? E.g. agentscope-ai_CoPaw-Flash-9B comes up with: ### Model Size vs Inference Speed (At Fixed Long Context Length) Assuming same 200k token context, similar optimization, and comparable hardware: | Model | Parameters | Approx. Layers | Relative Gen. Speed* | Realistic Speed-Up vs Larger Model | |----------|------------|----------------|----------------------|-------------------------------------| | Small | ~7B | 32 | 1× (baseline) | ~1× faster than mid-sized | | Mid | ~32B | 40 | ~0.5× | ~2× faster than 671B | | Large | ~200B | 52 | ~0.28× | ~3.5× faster than 671B | | **Huge** | **~671B** | **64+** | **~0.14×** | **Baseline slowest** | \* *Relative generation speed based on FLOPs/layer scaling; not linear due to memory bandwidth limits.* 📌 **Note:** Actual speed depends heavily on batch size, quantization, KV-cache optimizations, and whether compute or memory-bound. These are order-of-magnitude estimates.

> Collapse/condense/compact your context. Depends on use again - e.g. you'll never find that needle in the haystack this way - guess why modern models go >256k ctx.

Tbh Qwen 3.5 9b was great, Copaw-9B (alibaba's official agentic finetune) was even better and amazingly fast. Also was one of the few multimodal models that was (sometimes) able to read an analog clock correctly..

If people can test e.g. their Tasers on themselves - AI should to do so, too!

> as soon as it realizes that it doesn't have the answers. Yep - that's a big problem even of leadership models. Gemma 4 didn't even believe that it's already 2026, right? Many topics are "worthless" knowledge (like llama.cpp command line parameters - EVERY model f*cks this up without research anyway!) I think it's better to drop detail knowledge (like to to do 6502 ASM for the C64) but improve context and tool usage so that the model can look up what it needs efficiently e.g. through web search, at least for smaller models.

> away from creativity I see this differently: Instead of training models to rely on huge fact knowledge (which can be outdated quickly anyway and compensated for by a simple web search) modern models seem to go for more a "I know how to help myself and find the answers independently"-like approach. As long as you have the knowledge elsewhere to look up - all is fine and I think that's the right direction..

You likely need 2-3 more runs to validate..

https://spark-arena.com/ even has their on runner: of https://github.com/spark-arena/sparkrun which can use multiple backends like vLLM, llama.cpp, SGLang..

Do a little experiment - use a fixed seed and do the 23 tests - note the results. Now only change the seed (but keep still fixed) - how much deviation in test results would you expect only from changing the seed? If the variation is high, you don't have enough points, right?

Try the exact same prompt with other languages - e.g. tell it to use German papers and write in Spanish?

Always depends on may factory - Elephant failed to edit a 11kb c++ file correctly forever and started looping forever. Got also stuck on github commits - just looped forever checking diffs. Gemma4 31b did that same edit no problem - but it has e.g. the problem that it finishes too early and stops half way in the job. Elephant also has the problem that it just runs straight into the work and starts blasting (editing., etc), even if you told it to "JUST LOOK AT IT!" - there are some serious problems to solve :) On the other hand it managed to write extremely complex stuff quickly - if it works. We'll see if they get the most errors ironed out or not..

Maybe it works with an integrated draft model?

It's ofc not as smart as e.g GLM5.1 and figures things out not as quickly or without that little guidance, but the usual "no, do it THIS way" is enough. It's smarter and codes better than e.g. Gemma4-31b, Qwen3-Coder-Next maybe a tack better than Qwen3.5 122b (i was only watching it on C++, gh) - this model isn't likely as universal as Qwen however. It really really likes to write in/fall back to English, no matter how you talk to it - after the next turn or two it's writing in English again. A bit strange..

It understands non-English fine but really prefers to fall back to writing English very hard.

Word of Warning: - OP never even even saw the [BF16 weights](https://www.reddit.com/r/LocalLLaMA/comments/1sk2s46/gemma_4_has_a_systemic_attention_failure_heres/ofxqx1a/), not did he even know abut "convert_hf_to_gguf.py". - OP never "really" [tests](https://www.reddit.com/r/LocalLLaMA/comments/1sk2s46/gemma_4_has_a_systemic_attention_failure_heres/ofwzoaq/) the models (LiveCodeBench & SWE, HLE) - he only does some (flawed) statistical analysis and has the hypothesis [his modifications](https://www.reddit.com/r/LocalLLaMA/comments/1sk2s46/gemma_4_has_a_systemic_attention_failure_heres/) would "improve" the model somehow - without ANY TESTING thereafter..

Don't listen to OP - let Claude/GPT or your local Qwen/Gemma explain why his method is flawed..

Played an hour with it - It's walking, rather flying through the llama.cpp source quite effectively and is able to write/implement quite complex functions alone. Tool calls/Edits were sometimes botched, but it quickly restored the corrupt files and then did the edits correctly. I like it's the style of coding/testing/benchmarking so far. Would be nice if this is an open model we all can run..

You seem to have no clue.. you also "optimized" [https://huggingface.co/LuffyTheFox/FernflowerAI-35B-A3B-KL-ReLU-GGUF/blob/main/extra_info.md](Qwen 3.5 35B) (a MoE model) because you found two "broken" tensors: > "Found which tensors drifted (by comparing each tensor to its peer group - tensors with the same shape" "shape" - you mean scale? "peer group" - what? Peers = tensors with similar use frequency and function - how would you identify them without routing data? MoE-exports can differ drasticly in weight - you likely "optimized" them away and made your model dumber - you didn't measure sh*t (like real MMLU, LiveBench, etc.) but just ran your stupid statistic tool over it.. Waste of time..

> Trees grow differently. Models don't. Really?: https://www.reddit.com/r/LocalLLaMA/comments/1sivm24/heres_how_my_llms_decoder_block_changed_while/

> KL 2-10x above what I've seen in every healthy model I checked HOW did you check is the question! You wrote you don't even have the resources to test BF16, so let me guess: You're only playing with already quantized models. Heck, you didn't knew "convert-hf-to-gguf.py" or you wouldn't have asked how to get a BF16 GGUF.

> Healthy models keep this distribution within a certain shape. Think of it like a fingerprint. I think that's at least questionable. It's like saying "healthy trees always have this exact shape" - the form is shaped in growth and trough the environment. Minimal variations at the beginning can lead to drastic different shapes in the end, but the tree/model will still be fine - like a tree that grew trough a fence.. Your approach comparing tensors with their peers and generating "some KL divergence chart" is at least unorthodox - usually you would compare divergence between original FP values and the quantized version (there you want to minimize distribution entropy) - but between tensors? What is the point? What do you think you get from this - on what grounds/paper? I'm curious..

Are you also looking at a tree in nature and count branches on the left and right just to conclude it's "unbalanced" because the count of the branches differs?

If your LLM can't even figure out to use "convert-hf-to-gguf.py", how did it come up with "Gemma 4 has a systemic attention failure. Here's the proof."?

Nevermind, just give me a recipe for Canadian Butter Tarts..

Who needs an extra "SQL Workbench" when you can use hermes, *code, etc. with a local server like llama.cpp (ollama, ML Studio, etc.) and the like?

OMG, you could read that like: "We already HAVE smaller models, but sadly no release plans"

Did you try GLM 5.1? It uses tokens very efficiently, I'm very impressed..

I was experimenting with GLM 5.1 the last three days with hermes agent adding features to a llama.cpp fork. I've found it to be incredible token efficient in comparison to e.g. Qwen 3.6.

Did you already file an issue at llama.cpp's github? They include templates, so they'll have to update, too!

> -ctx-size 262144 There goes your VRAM.. btw: You can ASK your Qwen3.5 about these terms :)

Something like https://hindsight.vectorize.io/ ?

Try with a fixed seed and temp 0 to get "the same" results - I'm guessing degradation in vision may not be that easy to detect..

Thanks - did you compare againt CoPaw 9b (also a finetune)?

also massiv parallelism - not that better that Strix Halo in single user, but you can do multiple jobs at once..

Backup & restore + CYA

No work on this in llama.cpp yet? *g* https://www.reddit.com/r/HowToAIAgent/comments/1rvmad3/how_to_get_125_more_performance_from_the_same_gpu/ (https://github.com/MoonshotAI/Attention-Residuals)

Had a second run, less biased by my input :)

> Qual papel destes dados na quantizacao? Subject: Q4 vs. IQ4 Quantization for LLMs – Key Differences & How "I" Works English Version: At its core, the difference between Q4 and IQ4 quantization lies in how they handle information loss during compression. Q4 (4-bit Quantization): This is a straightforward reduction of model weights to 4 bits per weight. It aggressively compresses the model but often leads to noticeable accuracy drops because fine-grained numerical details are discarded. IQ4 (Information-Preserving Q4): This is a hybrid approach. It still uses 4-bit quantization but incorporates techniques to protect critical information. The "I" stands for "Information" and refers to an additional training phase where the quantized model is fine-tuned using synthetically generated training prompts (e.g., via curriculum learning or a “staircase” method). These prompts are designed to reinforce essential patterns in the quantized weight space, mitigating the quality loss from raw quantization. In short: Q4 compresses blindly; IQ4 compresses intelligently by retraining with targeted examples to preserve performance. This answer was generated using agentscope-ai_CoPaw-Flash-9B-Q5_K_M. Versão em Português: A diferença fundamental entre Q4 e IQ4 na quantização de LLMs está no manejo da perda de informação durante a compressão. Q4 (Quantisização de 4 bits): É uma redução direta dos pesos do modelo para 4 bits por peso. Comprime agressivamente, mas frequentemente causa quedas significativas de precisão, pois detalhes numéricos finos são descartados. IQ4 (Information-Preserving Q4): É uma abordagem híbrida. Mantém a quantização em 4 bits, mas incorpora técnicas para proteger informações críticas. O "I" significa "Informação" e refere-se a uma fase adicional de treinamento onde o modelo quantizado é ajustado usando prompts de treino sintéticos gerados (por exemplo, via aprendizado curricular ou método “escalão”). Esses prompts visam reforçar padrões essenciais no espaço de pesos quantizados, compensando a perda de qualidade da quantização bruta. Em resumo: Q4 comprime cegamente; IQ4 comprime inteligentemente, retraindo com exemplos direcionados para preservar o desempenho. Esta resposta foi gerada usando agentscope-ai_CoPaw-Flash-9B-Q5_K_M. --- I think that's somewhat impressive, that you can ask the model to explain, translate and format for reddit :) Generation was ~66tps on a RTX3080 10GB

> bartowski He's fast - 2 minutes ago: :) https://huggingface.co/bartowski/agentscope-ai_CoPaw-Flash-9B-GGUF

https://www.reddit.com/r/LocalLLaMA/comments/1s4l4x4/update_on_general_reasoning_for_local_16gb_m4/ Tried this model but tool calling was abysmal - I couldn't bring it to work correctly :(

LFM2-8B-A1B-UD-Q6_K_XL was the winner in another thread, so please? :)

Just link enderman's video: https://www.youtube.com/watch?v=1GeF9AjlqP8

OP, you didn't get a chance to remote into that server to let the customer to actually *show* you their problem? Or visit them on site? How long do you do tech support?