For everyone that uses OpenCode / Pi - Heres your promptprocessing fix!

[-]

jacek2023@reddit

Thanks for sharing. It would be very helpful if someone could test it on their setup. I’ve been testing it a lot over the last few days, but only on pi + Qwen 3.6 27B

Reply

[-]

Fyi, it doesn't seem to work so great with GH Copilot (qwen3.6-27b); no checkpoints are made during initial prompt processing, only when it is 100% completed, and with an initial context size (tools, instructions, large project, etc) of 28K, this causes all new contexts (All agents and new chats) to be quite slow. The old checkpoint system, set to at least 4096, mostly covered this use case. But on Pi, so far it's pretty sweet! Not seen any reprocessing

Reply

[-]

No_Algae1753@reddit (OP)

Been testing it so far (ive been the qwen3.5 122b user). The latest changes you have made seem to have fixed it completely for me.

Reply

[-]

jacek2023@reddit

try experimenting with \`--checkpoint-min-spacing-n-tokens 256\` (bigger number -> fewer checkpoints)

Reply

[-]

Caffdy@reddit

> --checkpoint-min-spacing-n-tokens 256 what does that flag do?

Reply

[-]

jacek2023@reddit

minimum spacing between context checkpoints

Reply

[-]

No_Algae1753@reddit (OP)

Is this to avoid unecessary checkpoints being made on smaller prompts / tool call?

Reply

[-]

jacek2023@reddit

It’s a minimum distance between them. Originally, there was a hardcoded value of 64, but if prompt processing is let's say 1000 t/s, then 64 feels too small, so I am testing 256

Reply

[-]

thejacer@reddit

Is there a pp to distance ratio you recommend?

Reply

[-]

Kike328@reddit

what is promptprocessing?

Reply

[-]

thirteen-bit@reddit

Time that it takes for LLM to process the prompt (system prompt + all previous messages) when it switches between parallel requests (e.g. coding agent sub-agent returns to orchestrator and you wait until orchestrator model processes all 70000 tokens that were in the cache before calling sub-agent). Or your coding agent runs and you ask the same model something in the chat ui in a separate app - when coding agent resumes it must re-process entire prompt. Basically look for something like this in the logs (I'm using mailine llama.cpp, linked PR improves the context checkpoints): 31.00.512.014 I slot launch_slot_: id 1 | task 10385 | processing task, is_child = 0 31.00.512.041 W slot update_slots: id 1 | task 10385 | n_past = 40183, slot.prompt.tokens.size() = 40183, seq_id = 1, pos_min = 40182, n_swa = 0 31.00.512.042 I slot update_slots: id 1 | task 10385 | Checking checkpoint with [40082, 40082] against 40182... 31.00.541.678 W slot update_slots: id 1 | task 10385 | restored context checkpoint (pos_min = 40082, pos_max = 40082, n_tokens = 40083, n_past = 40083, size = 149.626 MiB) 31.00.800.915 W slot create_check: id 1 | task 10385 | erasing old context checkpoint (pos_min = 8769, pos_max = 8769, n_tokens = 8770, size = 149.626 MiB) 31.00.927.480 I slot create_check: id 1 | task 10385 | created context checkpoint 32 of 32 (pos_min = 40254, pos_max = 40254, n_tokens = 40255, size = 149.626 MiB) 31.01.472.531 W slot create_check: id 1 | task 10385 | erasing old context checkpoint (pos_min = 9123, pos_max = 9123, n_tokens = 9124, size = 149.626 MiB) 31.01.715.309 I slot create_check: id 1 | task 10385 | created context checkpoint 32 of 32 (pos_min = 40766, pos_max = 40766, n_tokens = 40767, size = 149.626 MiB) 31.01.750.105 I begin: ngram_mod occupancy = 37526/4194304 (0.01)

Reply

[-]

jacek2023@reddit

this is what happens after you send your message, before you see the first reply token

Reply

[-]

caetydid@reddit

Thanks! I was unsure whether it is a server issue or caused by the interaction between llamacpp and pi agent? So, do I get it right, that checkpoints break and the entire context has to be preprocessed again instead of being reused every once in a while? Because that is surely what it seems like for me when using pi with my local Qwen 3.6. I assume this will speed up my vibe coding experience quite a bit because I often use between 50 and 100k context, and hope this will be merged in main soon.

Reply

[-]

jacek2023@reddit

Yes, the goal is to speed up the "agentic coding experience". Look at the llama-server logs and notice the "prompt processing" lines, if these occur without good reason, the user is wasting time / electricity / sanity.

Reply

[-]

Ok-Measurement-1575@reddit

Not sure I have a PP issue in opencode?

Reply

[-]

nonerequired_@reddit

For that purpose I am (forced to) using claude code instead of opencode. It causes lots of prompt reprocessing issues

Reply

[-]

hdmcndog@reddit

Not sure why that would help much, to be honest. Yes, OpenCode does _some_ tool call pruning, eventually, but it’s still fairly conservative, so I wouldn’t expect prompt reprocessing until relatively deep in a session. Pi doesn’t have this problem at all, it tries to maximise cache hits by not adjusting prior messages at all. So you could try with Pi, and the experience should be similar to what you see with Claude Code, I guess. But even with Pi, there still sometimes is prompt reprocessing, so some tuning on the llama.cpp side, like what OP is trying to do is probably necessary and helpful.

Reply

[-]

nonerequired_@reddit

I will try pi. I heard it is a bare-bones kind of thing. That is why I was avoiding it

Reply

[-]

No_Algae1753@reddit (OP)

Set yourself up for the incomming pp jokes

Reply

[-]

ECrispy@reddit

can you avoid this by turning off checkpoints? and whats the danger with that?

Reply

[-]

jacek2023@reddit

Let's say you have 100k tokens of chat history and then send "thank you". Without checkpoints, how much of that prompt do you think has to be reprocessed before generation can start?

Reply

[-]

ECrispy@reddit

ah ok so checkpoint is a sort of cache of the kv cache with prompt processing? i'd assume its done automatically at end of message

Reply

[-]

jacek2023@reddit

Initially I was using Gemma 31B with OpenCode. First I realized that OpenCode does that [https://github.com/anomalyco/opencode/issues/23595](https://github.com/anomalyco/opencode/issues/23595), so I switched to pi, then I realized that Gemma is removing thinking from the prompt history. So in that case "100k tokens" from the next message is not same as "100k tokens" from before, so some reprocessing is always required, without the checkpoint everything must be reprocessed (so you must wait minutes!)

Reply

[-]

hdmcndog@reddit

Could you go into a little more detail in what situation you experience the prompt processing happening, all the time? In my surface level tests, I don’t see big issues so far. But it might totally depend on the usage pattern. I usually just have a large prompt processing happening at the start of the session, for the system prompt, and then on large file reads etc. Otherwise, it seems to be pretty smooth for me.

Reply

[-]

jmager@reddit

I've been using this branch all week and rebuilding it daily, and it indeed fixes the checkpointing issues.

Reply

[-]

MuDotGen@reddit

Oh, what is your setup? Pi/Opencode? Hardware? Model? Is it preventing all the prompt re-processing you were experience before?

Reply

[-]

MuDotGen@reddit

I have had a bunch of checkpoint issues with prompt cache using Qwen3.6-35B-A3B with Pi. Would this help fix it perhaps?

Reply

[-]

ECrispy@reddit

I assume these changes get downstream into bun-llama/ik-llama etc?

Reply

[-]

sophlogimo@reddit

"open". So it is not fixed- or what do you mean?

Reply

[-]

No_Algae1753@reddit (OP)

Well it is obviously not merged it is still a PR. I just build the latest llama.cpp with this PR and the prompt processing issue I dealt with is gone. Try it out for yourself and see if it works.

Reply

[-]

Maximum@reddit

What is the issue with it?

Reply

[-]

IslamNofl@reddit

I tested it myself a week ago, and it doing the job

Reply

[-]

anthonyg45157@reddit

How does this issue manifest or show itself in pi? I don't think I've had any issues with prompt processing but I haven't fed any super large files or anything recently

Reply

[-]

sixx7@reddit

Here is exactly what it means, summed up in a log line: > [2026-05-21 16:18:50 TP0] Prefill batch, #new-seq: 1, #new-token: 2048, #cached-token: 46656 In that example, if your backend is not caching prompts, it has to process all 46,656 tokens again. Think about long running conversations or multi-turn agents. The context can build up to hundred(s) of thousands of tokens. If they are cached, it's basically instant. If not, all that time has to be spent processing them again. Caching your prefill tokens is the literal biggest performance improvement you can make if it's not being done. Works fine in vLLM and SGLang. Apparently broken in llama.cpp

Reply

[-]

No_Algae1753@reddit (OP)

If you are using llama.cpp try longer context. You will encounter promptprocessing

Reply

[-]

ang3l12@reddit

Is this only in opencode / pi? or any harness?

Reply

[-]

jacek2023@reddit

for everything

Reply

[-]

jacek2023@reddit

You must wait longer after typing your prompt because the last usable checkpoint is far away. Sometimes you have to wait a few minutes because the prompt is processed from the start ("forcing full prompt reprocessing...")

Reply

[-]

wren6991@reddit

OpenCode itself is also just a bit of a shitshow with prefix stability. My favourite issue is that it puts the current date in the system prompt and re-evaluates it every turn, so you get a full prompt cache flush if you're using OpenCode at midnight.

Reply to Post

40 Comments