Math don't check out.

Posted by Ok-Internal9317@reddit | LocalLLaMA | View on Reddit | 13 comments

Say I own a blackwell, where prompt ingestion speed is 2500tok/s; which is very fast. My context window in opencode usually is 120K, 120000/2500=48seconds 48 second I must wait for each code edit, which could've been opening a file or a simple 20 token linux command. This is unuseably slow no?

Reply to Post

13 Comments

[-]

TokenRingAI@reddit

It doesn't reprocess the prompt each time, it is cached. On an RTX 6000, each turn takes a half second or so to process. It's extremely fast.

[-]

o0genesis0o@reddit

If the prompt caching works, you only need to process the new content within that turn, not the full 120k. So it should be seconds per turn.

[-]

FatheredPuma81@reddit

Yea I'm getting deja vu from a conversation with some guy that said Mi50's are unusable because prefill is slow. Meanwhile llama.cpp has a setting for prompt cache chunking (you really want this programs that delete portions of context like OpenCode) and a setting to store old prompt caches to RAM. You have to seriously be filling your context up a lot with unique content for prefilf speeds to be a noticeable issue...

[-]

alphatrad@reddit

It's because your math is completely wrong.

[-]

Makers7886@reddit

Since you didn't give details or config info I'd say it's in your details and config info.

[-]

StardockEngineer@reddit

Why are people downvoting you for stating the obvious? We have no info.

[-]

Makers7886@reddit

That was with me editing for softer tone as well. It's like every technical subreddit now has the lowest effort non-information question. Like sending a doctor a foot pic and saying "why bad".

[-]

StardockEngineer@reddit

That’s not how it works. Context length is not divided by 2500. Actual context is. Length is the max. You’ll start where from 0 to 30k (Claude Code). And with caching subsequent prompts are calculating the difference, not the whole thing again.

[-]

Ha_Deal_5079@reddit

honestly running 120k context all day is the problem not the solution. i keep sessions focused n restart when switching tasks way faster

[-]

suicidaleggroll@reddit

In addition to caching, you won’t sit at 120k context at all times with opencode. It will start at 0 and build up as you work. When it gets high enough, 60-90k, opencode will compress the context, meaning it will have the LLM write notes to itself about where it’s at and what it’s doing, then it’ll purge and reload, dropping context usage down to maybe 10k.

[-]

jacek2023@reddit

Check logs for "forcing full prompt re-processing", that's the main issue I am having with agentic coding and llama.cpp (I am working on workarounds)

[-]

FullstackSensei@reddit

Man, I run Qwen 3.5 397B Q4 on Mi50s with 100t/s PP on a good day and I don't usually wait more than 10 seconds per change after initial prompt ingestion. Prompt caching is a thing. Going by roo code, my average session is 10M+ context tokens, while llama-swap usually reports 200k, if not less.

[-]

FoxiPanda@reddit

Yep, you're missing the effect of caching. On the first turn, let's say you load in 20000 tokens (a big system prompt in an agentic harness). This will take your 20000/2500 = 8 seconds for the response tokens (may include thinking so maybe not the *final* response though) to start flowing out. After that though, your first 20000 tokens *should be* cached assuming you don't invalidate your cache somehow. Then, let's saw you write a 500 token prompt on your second turn. You don't reprocess the 20000 - they're already processed. You only process the 500 new. 500 / 2500 = 0.2 seconds Response tokens (including thinking tokens) start flowing out less than a second later.