Technical question can you mask/hide parts of the KV cash for a request.
Posted by Noxusequal@reddit | LocalLLaMA | View on Reddit | 3 comments
We have the following idea have two llm agents interact but each also has an internal monolog.
It would suck to reload the whole context between each llm request.
So there are 2 questions: 1. Can you incramentally update the kv cache ? Only adding the last line of dialog that was produced in the last prompt. 2. Can you hide parts of the kv cache ? So that we dont have to reload it between the agents taking turns. Since we could just hide the part of the kv cash that refers to the internal monolog of the other agent.
Do any implementation like this exsist or if not would that even be technically possible ?
nero10579@reddit
What software are you inferencing with
tronathan@reddit
I think if you submit two queries, and they share a lot of the header, that the KV cache is generally smart enough to reuse what cache it can. As soon as two queries diverge though, all tokens after that will not have cache hits, so those will be a bit slower.
I'm kinda hand-waving here but I think the KV cache will Just Work and you can send different requests with simlar/overlapping prompts, as long as your agent-specific instructions are at the end of your prompt, you should be good.
Someone please correct me, i'm probably oversimplifying.
Noxusequal@reddit (OP)
That would be very sweet if true :)