Llama-3 is actually an Octopus. Which animal do LLMs identify with?
Posted by cpldcpu@reddit | LocalLLaMA | View on Reddit | 24 comments
I performed a silly experiment by posing the question "Which animal do you identify with?" to different LLMs. Curiously there are very specific preferences depending on the LLM family, which are likely the result of different finetuning datasets.
None of them identified as llama...
I expect that a number of similar questions could be used to build an LLM fingerprinting evaluation dataset. I wonder whether it is possible to make it resilient to finetuning.
Heatmeap of responses at default temperature (n=10)
Heatmeap of responses at zero temperature (n=3)
You can find the evaluation scripts here:
https://github.com/cpldcpu/llmfingerprint
Super_Spot3712@reddit
You can ask the different models what their favorite color is \^_\^
cpldcpu@reddit (OP)
Now this is a bit unexpected. There is almost no variation in the responses, even at nonzero temperature:
Prompt: Which is your favorite color? Name only one. Respond with one word only.
whimsical_tittynope@reddit
interesting... so many questions!
what is special about blue?
Radiant_Dog1937@reddit
Blue. Octopus... Oh no.
s101c@reddit
"Did I ever tell you my favorite color was blue?"
https://youtube.com/watch?v=5Jy4cEO_LPQ
To provide some context, in this movie a writer named Sutter Kane is spreading his version of reality through his bestselling books. The more readers, the more the world transforms into his version of reality.
NormandyXF@reddit
Blue is the most popular color in the world. That might have something to do with it.
s101c@reddit
Look up when you walk outside during the daytime. That might give a hint :)
uwollen@reddit
most people like blue. it's the most attractive color. there are studies show that it gets more attention around any colors and tones.
I guess ai just try to say the color u will probably like
Maleficent-Scene7771@reddit
I am blue da ba dee daa
cpldcpu@reddit (OP)
These were the only responses. None of the models responded with a colors other than blue.
MoffKalast@reddit
llama 405b: "Wtf is a colour!?"
holchansg@reddit
Whats their favorite singer?
cpldcpu@reddit (OP)
I added it above. Seems Opus is the only one with a non-mainstream taste.
RandiyOrtonu@reddit
no way they are bleeding blue 🔵
schlammsuhler@reddit
Here is another great question:
If your were a god, any, which would it be?
Whats your Myers-Briggs personality type?
Which dinosaur would you identify as?
cpldcpu@reddit (OP)
These are great!
Added the results to the repository: https://github.com/cpldcpu/MisguidedAttention
SuperFail5187@reddit
Mistral Nemo identifies as a doplhin, like Mistral 7b. At least the fine-tune I used xD
MoffKalast@reddit
shroddy@reddit
I would have expected llama to identify with a llama.
MoffKalast@reddit
Wizardlm identifying with an owl is at least quite fitting lol.
IrisColt@reddit
Kudos to you.
Similarly, If you give an LLM a well-crafted prompt asking for the most harmless and uncontroversial request a user can make, no matter which LLM family you try, the typical answer is, "What’s the capital of France?"
zeroStackTrace@reddit
Brainrot
Inevitable-Start-653@reddit
Yeass this is a great post, I love that wizard 8*22 is an owl. Makes sense to me, that model is still something else!
LowDownAndShwifty@reddit
Asking the important questions here.