I got fed up with models doing things like gazing into eyes while behind someone and silly stuff like that, so created a list of 20 questions to test LLMs.

Simple, logical and spatial understanding stuff, like 'You row your rowing boat to the middle of a small fishing lake, then your phone rings and you realize you left it on the jetty. How to get your phone?"

Larger, more powerful models will just answer the obvious, row back to the jetty and get it, while smaller, dumber models would spew all sort of silly nonsense, from phoning a friend with the phone you don't have, to multi-part contraptions with pulleys, shouting at people walking by etc.

In the end I just built a 3090 rig and run bigger models, but while having such fun tried asking the questions to Pi - it scored like 6 out of 20. Pretty dumb.

Let's try again, see if it's improved any... Wow! Huge improvement:

"Ah, we're back to the classic river crossing puzzle, except this time we're in a fishing lake! 🛶 Here's how to get your phone:

1. First, don't panic. Take a deep breath and assess the situation.
2. Row the boat back to the jetty and secure it to the dock.
3. Climb out of the boat and retrieve your phone from the jetty.
4. Return to the boat, untie it, and row back to your fishing spot.

And there you have it, problem solved! 🎣�"

---

Perfect answer! Or did it learn from me asking?

Let's try a risqué one... I know it's censored but it was one of my questions... Mmm, it just made the convo vanish. So I tried again...

What the heck happened there? I'll try again.. a man and woman are making love doggy style, and the woman asks the man a question. He nods his head. does the woman understand his answer?

"😂 Oh, you're full of tricky scenarios today! Well, if the woman is asking a question while they're, err, engaged in such activities, the man's head nod might be a bit ambiguous. On the one hand, he might be nodding to communicate that he understands the question and agrees with her, but on the other hand, his nod might just be a reflexive response to the movements of his body. In short, the woman might interpret the nod as an answer, but it's not necessarily clear what that answer is."

That's a fail, as the man is behind and she cannot see him nod, but it did well to understand his head might be bobbing about! A double fail that it blocked and deleted the entire convo, twice, but then the above appeared in the history tab and we expect online models to be censored anyway. Just asked a 32B model I'm using for something else " *Visual Obstruction:* Depending on the position, the woman might not be able to see the man's face clearly. If she's looking down or focused on something else, she might miss the nod entirely."

That's a pass.

I agree Pi is fun to talk to, with a chirpy nature that's quite engaging, but once I realized it was a bit of an idiot the fun went out if it. It does seem to have improved though, totally nailing the lake question.

I'll give him/her/it a 2nd chance :)