Is there any way to estimate tokens per second given VRAM and such? The calculators don’t have every model.

Posted by Borkato@reddit | LocalLLaMA | View on Reddit | 7 comments

For example if I find GiggleBox-Super-Cool-Mega-SLOP-XL-24B.Q3629272\_K-x-ULTRA-super-DPO-LX-MegasXLR-Voldemort-GodU-Homelander-XFiles-Hopped-Into-A-Coffee-Shop.gguf and I know I have 24GB VRAM is there a way I can estimate the T/s I’ll get running it?

Reply to Post

7 Comments

[-]

TokenRingAI@reddit

You don't need a model calculator at all, your speed for a dense model is roughly your GPU memory bandwidth divided by your VRAM size. 1TB/sec memory bandwidth, 10GB vram size, 100 tokens a second. Multiply the speed by approximately 7 for an MOE model, with a top limit of around 200 tokens per second before latency starts capping the model performance.

[-]

Borkato@reddit (OP)

This is quite interesting, will try this, thank you!!

[-]

nagareteku@reddit

Tokens/second = Bandwidth (Gbps) / Parameters (billions) / Quantization (bits per parameter) Assuming that each token requires one pass of the model and that bandwidth is the bottleneck, we can divide the bandwidth by the parameter size in bits to get the token generation speed. However, this excludes input prompt processing, compute as well as context (You can add about 1B to the "Parameters" calculation per 1000 tokens) to estimate how much it will slow down generation.

[-]

nagareteku@reddit

LLM inferencing is memory bandwidth limited, while training is more compute limited. Bandwidth acts as an upper bound. This is also why unified memory in M-series Macs and Ryzen AI mini PCs are not as fast as VRAM, which is not as fast as HBM. Faster memory also costs more than slower memory, which is also why HBM costs so much, and this goes even to DDR5 costing more than DDR4 for the same size. Amongst the hardware favoured by hyperscalers, the B200 has 192GB of HBM3e at 8192 GB/s. For GPUs, the RTX Pro 6000 has 96GB of GDDR7 memory at 1792 GB/s. Running most 70-120B models at a usable speed of 10-20 tps. Unified memory? M3 Ultra has up to 512GB at 819 GB/s to run your R1 Q4 at a few tokens a second. DDR5 RAM? 12 channel DDR5-6000 in high-end Epyc builds can load in multiple TB of RAM, at 576 GB/s. Cheaper servers on 8 channel DDR4-2400 or 4 channel DDR5-4800 gets 230 GB/s. Two channels of DDR5 gives bandwidth similar to four channels of DDR4. Consumer builds usually are limited to 2 channels. 2 channel DDR5 as mentioned above barely get over 100 GB/s. 2 channel DDR4 around 51.2 GB/s for DDR4-3200. This is why CPU inferencing is slow. In this case, DDR3 is half the speed of DDR4 and is 25.6 GB/s for two channels of DDR3-1600. At these bandwidths, DDR3 channels run like sticks of fast, high end PCIe 5.0 SSDs, too slow for modern LLM usage. The Samsung 9100 Pro does 14.8GB/s read, while DDR3-1866 (PC3-14900) does 14.9 GB/s.

[-]

Borkato@reddit (OP)

This is extremely helpful. I’m going to try it with my hardware. Is the “+1000 as 1B” heuristic for prompt processing AND context or just context?

[-]

Electrical_Tax4470@reddit

Honestly the best bet is just to eyeball it based on similar sized models you've run before - those wild finetune names don't really change the inference speed much from the base model size If it's actually 24B params and you've got 24GB VRAM you're probably looking at like 15-25 t/s depending on your GPU but YMMV with quants

[-]

Borkato@reddit (OP)

Huh, I didn’t consider the idea of comparing it to the base models. I should download just the bases and compare so I can have a general idea. In fact I don’t even know if I did ever download just the base models, I bet they’re better on many tasks!