Formal-Exam-8767

LLM holy wars

Maybe they mean church based around the model, like "Church of Qwen" or "Church of Gemma"?

I am not talking about technical running, but practical useful running. You can technically run 400B models off SSD with 8GB RAM, but is that usable for anything?

With 24GB you can forget about running LLMs.

"only save the relevant stuff about me"

No, for two reasons: 1. Too expensive for what it provides 2. ARM-based CPU

Can you do a Q8 bench since Q4 quality-wise is not really usable for anything.

> if prices are going down dramatically You don't have to worry about that, prices definitely won't go down dramatically or even go down.

Are they connected via NVLink? I've been reading here that tensor-split will be slow without NVLink.

https://docs.vllm.ai/en/stable/features/quantization/#supported-hardware

I dug some 3090 results for comparison: /r/LocalLLaMA/comments/1rqljv4/benchmarked_all_unsloth_qwen3535ba3b_q4_models_on/

Yes, there appears to be lots of room for improvement.

B70 has theoretical memory bandwidth of 608.0 GB/s and this does not even reach 150.0 GB/s if my math is correct?

ROCm is 10 years too late.

How would a slightly modified DGX Spark using LPDDR5X have 600GB/s memory bandwidth?

arch

SXM2?

Where is this misinformation coming from? How much data is actually exchanged between two cards that it would require fast interconnect?

So they plan to sell device with 20% performance degradation out-of-the-box?

I thought being ARM-based limits it's usability for anything outside AI. Do current Windows games support ARM?

So, how many years are you willing to wait for an LLM to finish training on a single 8GB VRAM card?

That is just how auto-complete works, it completes what came before. Don't attribute intelligence to a system without any.

> Not looking for "buy a 5090 instead" Buy 16x 3090 instead.

No amount of compute can help in a memory bandwidth bottlenecked use-case. RTX 5080 - 16GB @ 960.0 GB/s RTX 3090 - 24GB @ 936.2 GB/s

And they would love to pass on electricity bill to the customer.

Yeah. I would not underestimate data center cards even if they are old.

MI50 Bandwidth 1.02 TB/s RX 9070 XT Bandwidth 644.6 GB/s

> "local is cheaper" I never saw this argument here.

How much RAM do you have? I don't think it pre-allocates context on start.

Can KV cache be spliced? What if you kept question KV-cache, spliced out reasoning part, and glued rest of response KV-cache to end of question KV-cache?

So AMD GPU cards work with Apple Silicon?

Tell me about the features of "Gamma 8 43B A5B" LLM model. > The model string "Gamma 8 43B A5B" represents a specific technical configuration of an open-weights Mixture-of-Experts (MoE) Large Language Model (LLM). > > While the naming convention breaks down into standard machine learning nomenclature, the precise string reflects a model from the extended ecosystem of open-weight architectures (often iteratively released by community developers or institutions expanding on foundational lines like Google's Gamma/Gemma or similar MoE structures). [1] > > The key features and architecture can be accurately parsed from its naming blueprint: > > ## 🛠️ Structural & Technical Specifications > > * Total Parameters (43B): The model contains a total of 43 billion parameters within its complete architecture. This positions it firmly as a medium-to-large localized model, capable of high-tier reasoning, complex coding tasks, and strong contextual synthesis while remaining deployable on localized enterprise hardware or multi-GPU home setups. > > * Active Parameters (A5B): The "A5B" designation stands for 5 Billion Active Parameters. As a Mixture-of-Experts (MoE) model, only a sub-network (expert layers) totaling roughly 5 billion parameters is activated for any single token processing step. > > * "Gamma 8" Baseline: This points to the version or generation cycle (Iteration 8) of the underlying architecture baseline, building upon the optimization patterns of highly efficient open-weight models. [2, 3, 4, 5, 6] > > ## ⚡ Key Architectural Advantages > > * Blazing Inference Speed: Because it only computes 5 billion parameters per token instead of the full 43 billion, the model delivers the execution speed and low latency of a much smaller 5B model. > > * High-Capacity Intelligence: Despite its fast processing speed, it retains the knowledge depth, reasoning capabilities, and broad vocabulary of a massive 43B parameter model, because the remaining specialized parameters are dynamically routed as needed. > > * Resource Efficiency: It drastically lowers hardware serving costs (VRAM compute cycles during generation) compared to dense, non-MoE 40B+ models, making it ideal for cost-efficient enterprise or local offline deployments. [2, 7, 8, 9] > > ## 🎯 Primary Use Cases and Capabilities > > Given this specific 43B/A5B MoE design footprint, the model features high proficiency in: > > * Agentic Workflows: Rapid token throughput allows it to execute multi-turn agentic loops (thought-action-observation cycles) without choking system speed. > > * Coding & Logic Synthesizing: The deep 43B parameter pool enables rich syntactic understanding of dense, multi-file codebases and mathematical constraints. > > * Highly Scalable Local Deployments: It serves as an optimal solution for organizations requiring strong data sovereignty and offline operations without the hardware budget for massive dense clusters. [2, 10, 11, 12, 13] > > To help tailor this, are you looking to deploy this model locally, or are you comparing it against a dense model configuration (like a standard 8B or 70B model) for a specific task?

Ask it about Gemma 5, 6, 7, 8, 9.

> make their own hardware? Sometimes, that is not the right move, don't forget what happened to 3Dfx...

If you got 12 of these BC-250 boards together into a 4U rackmount server chassis (like the ASRock 4U12G), that would be 192GB and no issues with cooling right?

MTP is new, yes. But using smaller models as draft models for larger ones is not. Support for that landed in llama.cpp in 2024: https://github.com/ggml-org/llama.cpp/pull/10455

Speculative decoding is nothing new, and there is always a desire to overcome memory bandwidth bottleneck.

I see, so that is not the reason why Intel and AMD don't want to put more than 2 memory channels in consumer CPU/MBOs?

How many layers does it have? I've been told that to achieve more than 2 memory channels you need multiple PCB layers because of the traces and that's what makes multi-channel MBOs expensive.

The main issue of running LLMs is not the amount of memory or even compute but memory bandwidth. You can ignore the amount of memory as it should be given that you have enough memory, if you don't, you can't run it, period. MoE have less active parameters so they better utilize memory bandwidth and require less compute since they are working with lower amount of data than dense models.

Am I missing something here, is "IQ4_KS" not supported on stock llama.cpp?

On-prem depending on your budget: * NVIDIA GB200 NVL72 * NVIDIA HGX B300 * NVIDIA HGX B300 * NVIDIA DGX H100

If the choice is between 5060ti 16GB and 5070 12GB, it makes no sense to go with less VRAM since you want really see any speed benefits that 5070 has over 5060ti. To me, it only makes sense to consider 5060ti 16GB or 5070ti 16GB.

But can the model even respond differently? If it has a system prompt with specified knowledge (to model this probably means all knowledge, both baked-in and from context) cutoff date, anything beyond is fiction.

Thanks for sharing. How does it compare to Vulkan?

Define useful. Considering that people are mostly using chats bots the wrong way (as **authoritative** source of truth), I am not surprised they don't find local LLMs useful.

Estimate is just and estimate. With MoE it is even harder to precisly estimate. And PP is close to impossible since it depends on actual architecture and ops not just size.

Why only DIY? Wouldn't prebuilts and brand names be even more expensive now?

Say it's an electric heater. You can demonstrate that it blows hot air out.

What is the minimal interconnect bandwidth requirement so it does not affect token generation speed?