As far as people seem to be concerned, almost everyone who says a model is crap, they always seem to evaluate a model by how it works by just giving it a few prompts. I never see anyone passing a system prompt that actually could help them. And I’m not meaning the typical example of telling it is a whatever type of expert. I’m meaning something that explains the environment and the tools it can use or anything like that.

I’ve learned that the more information you pass in a system prompt before you say anything to a model, the better the model seems to respond. Before I ask a model to do anything, I usually give it an overview of what tools it has, and how it could use them. But I also give it permission to experiment with tools. Because one tool might not work, but another may accomplish the task at hand.

I give the model the constraints of how it can do the job, and what is expected. And then in my first message to the model I lay out what I want it to do, and usually and invariably with all of that information most models generally do what I want.

So why does everyone expect these models to just automatically understand what you want it to do, or completely understand what the tools that are available if they don’t have all of the information or the intent? Not even a human can get the job done if they don’t have all of the variables.