Well, this has been a while. Obviously, ignore my model recommendations, they're way outdated by now.

So, if I get this right, if my conversation hits the token limit, older stuff gets cut off? Have you tried using summaries or anything like that to keep track of longer chats? Would love to hear more about your experience!

Correct, the oldest tokens from the context get truncated as new ones are added that would go above the set limit.

Summaries were tried but found to be fairly ineffective. For one, the summary is only ever as good as the model that writes it and, more importantly, by their nature, they don't contain details.

For instance, let's say you were Roleplaying with a LLM and in the beginning of the story, you went to the beach. If that beach episode gets dropped from the context, a summary would probably include it but not whether you had ice cream or waffles. If the conversation comes back to the time at the beach, the model will hallucinate what you had.

A more robust way to attempt to fix the context is vector storage which attempts to reinsert the most relevant parts of the conversation into the context. This banks on it recognizing that the ice cream is now once again relevant and on the model to fish that information out of the now huge context.