Small and fast function calling models

llama-cpp-python seems to enable llama.cpp's prompt caching (kv cache cache) by default. If you used llama.cpp directly you have to enable it explicitly. So I would expect llama-cpp-python to be faster after the first generation, once the prefix is cached including the long system prompt If gpu is available, make sure you're using it (n_gpu_layers=-1 in Llama constructor) Does the both functions require arguments? How many tokens does LLM need to generate? If one doesn't take arguments, you could turn it into a classification that takes only 1 token Once you move on to the second generation, do you still need the previous prompt and everything?