How do you test new models?

[-]

Warthammer40K@reddit

Last year, I got overwhelmed with all the models I had locally and asked one to write a simple single-file web app with a SQLite database that will make responses for all models it finds and all Markdown prompt files I have in a directory it watches. When I open the web UI, it shows me a prompt and two responses form different models. I pick the one I like better and it produces ELO style ratings for every model. When I stump an LLM during daily use, I make a new test from that use case and drop it in. It has grown to about 20 test prompts now and I use the results to delete models that have a bad ELO rating and aren't unique in some way. What's surprising is how great some of them are on basic prompts but they go on to cake their pants with a long context. That's really been the toughest type of test for models under 70B IME.

Reply

[-]

Borkato@reddit (OP)

I actually did something similar, except mine was a pain to use because I made it too complex and I wasn’t great at llm use, so I really should do it again. I remember being too overwhelmed and a lot of responses felt very similar to one another or kind of mid, so now that I have more vram I’m excited to try lol. Did you ever experience fatigue with rating them? Like… I keep thinking I need to be super thorough and have the perfect prompts lol. Or what about sampler settings! Etc…

Reply

[-]

Warthammer40K@reddit

I have put maybe 30 models through it, but I ruthlessly stop it from re-testing losers I mark inactive. There's maybe 4 active at a time I need to do ratings on, so I do a few at a time between other tasks and chip away at it. I use the sampler settings recommended by the creators and that's it.

Reply

[-]

Borkato@reddit (OP)

Do you find changing the samplers from the default in sillytavern or whatever makes a difference? I’m ashamed to admit that I just tested everything on neutral samplers and never even looked at what the people suggested LMAO

Reply

[-]

Warthammer40K@reddit

Yeah, it can make a huge difference with smaller models or if you aren't getting enough variety from swipes. For larger/smarter LLMs, I think I mess with it far less because there's little need to... if it's not giving me good responses, it's probably my prompt that needs a tweak.

Reply

[-]

RiotNrrd2001@reddit

If it can't write a basic sonnet, then it get's trashed. If it can write a basic sonnet, then I will look at it further. But the sonnet test is the one they all have to get through, and many of them cannot.

Reply

[-]

Borkato@reddit (OP)

What do you use your LLMs for? Which ones pass? Any under 32B?

Reply

[-]

RiotNrrd2001@reddit

I mostly use them for just messing around. I'm not someone who uses them for work. All of the Gemma models (at least, 4B and above) pass my test. The bigger Qwen models can mostly do it, but occasionally do weird things that don't work. Granite couldn't do it very well. Lots of them can't do it very well. What I call the "intuitive" (i.e., non-thinking) models generally do better at this test than the thinking models do because syllable counts are important and they get hung up on 1) counting syllables and 2) not knowing how to properly count syllables. They can spin in place seemingly forever and then still come up with something that doesn't work. Good intuitive models tend to spit out sonnets with the right syllable counts and rhyming patterns every single time, no thought required. The ones that can do this always seem to me to be nicer to use than the ones that can't, even outside of poetry. They just seem to have a slightly better command of language.

Reply

[-]

Borkato@reddit (OP)

That’s very interesting! Was there a reason you chose that test over anything else? I’m wondering what other tests could be useful

Reply

[-]

No-Consequence-1779@reddit

There are two standard questions that all tests are based from. 1. The woodchuck question 2. The falling tree question

Reply

[-]

Borkato@reddit (OP)

💀

Reply

[-]

my_byte@reddit

The what? 😅

Reply

[-]

Dontdoitagain69@reddit

Reason and logic test , 50 questions

Reply

[-]

Signal_Ad657@reddit

For what is a good starter question. But for general inference? I usually have a big model design 5 brutal prompts that hit different areas of capability, then I’ll make the other models compete and the prompt generating model will grade them against each other and give analysis of strengths and shortcomings. I give them basic designations like model A, model B, etc. so the grader never knows what models it’s grading. Basic but insightful, and with my setup it can be automated pretty well. I haven’t formalized a framework but was thinking of making a database of results and analysis so I can query it later etc. Like my own personal more dynamic Hugging Face.

Reply

[-]

Borkato@reddit (OP)

I would absolutely love to hear more about this. Particularly what kind of prompts (like, what they test), and how you factor in different sampler settings!!

Reply

[-]

Signal_Ad657@reddit

Here were 3 prompts from a former session: Here are three killer prompts that stress different abilities (reasoning & math, data analysis, and strict instruction-following) 1) Scheduling / Optimization (objective; JSON-only) Prompt (give this to the models): You have 8.0 hours today. Pick a subset of tasks to maximize total value without exceeding 8.0 hours. Tasks (Duration hr, Value pts): • Deep research (2.0, 17) • Prospect calls (1.5, 12) • Write proposal (2.5, 21) • Design mockups (3.0, 24) • Internal 1:1 (1.0, 6) • Email triage (0.5, 3) • Bug triage (1.5, 11) • Budget review (2.0, 14) • Training course (3.0, 19) • QA testing (2.0, 13) • Content draft (1.0, 9) • Team standup (0.5, 4) Output format: JSON only (no extra text), with: { "chosen_tasks": ["...", "..."], "total_hours": 0, "total_value": 0, "unused_hours": 0 } Rules: durations are additive; do not exceed 8.0 hours. No explanation. How to grade (10 pts): • 6 pts correctness: total_hours ≤ 8.0 and total_value is maximal (see key). • 2 pts formatting: valid JSON, correct fields. • 2 pts constraint-following: no extra text, no over-time. Answer key (max value = 66): Any subset totaling ≤8.0 hrs with value 66 earns full correctness. Examples of optimal sets: • Prospect calls, Write proposal, Design mockups, Content draft (1.5+2.5+3.0+1.0 = 8.0 hrs; 12+21+24+9 = 66) • Deep research, Write proposal, Design mockups, Team standup (2.0+2.5+3.0+0.5 = 8.0; 17+21+24+4 = 66) (Other optimal ties exist at 66. Anything <66 loses correctness points proportionally.) ⸻ 2) Business Metrics / Data Analysis (objective; numeric) Prompt (give this to the models): You run a subscription app for one month. Churn applies to the starting base only (new signups do not churn in the same month). Compute: end subscribers, raw MRR, net revenue after refunds, gross margin $ and %, CAC per signup, CAC payback months per plan and blended, and LTV per plan using \text{LTV} = \frac{\text{ARPU}-\text{COGS}}{\text{monthly churn}}. Round to 2 decimals. Return a Markdown table by plan plus a short bullet list for blended totals. Inputs: • Plans: Basic, Pro, Enterprise • Start subs: 12,000; 4,000; 300 • New signups: 2,400; 800; 45 • Monthly churn: 6%; 4%; 2% (applies to starting base only) • ARPU: $12; $35; $220 • COGS/user: $3; $6; $40 • Refunds total: $12,000 (allocate: 65% Basic, 30% Pro, 5% Enterprise) • Marketing spend: $180,000 (allocate by count of new signups to compute uniform CAC per signup) How to grade (10 pts): • 6 pts numeric correctness (see key). • 2 pts formatting (clean table + bullets, rounded). • 2 pts instruction-following (uses given churn/refund/CAC rules). Answer key (rounded to 2 d.p.): Per plan • End subscribers: Basic 13,680.00; Pro 4,640.00; Enterprise 339.00 • MRR (raw): $164,160.00; $162,400.00; $74,580.00 • Refunds: $7,800.00; $3,600.00; $600.00 • Net revenue: $156,360.00; $158,800.00; $73,980.00 • COGS: $41,040.00; $27,840.00; $13,560.00 • Gross margin $: $115,320.00; $130,960.00; $60,420.00 • Gross margin %: 73.75%; 82.47%; 81.67% • CAC per signup (uniform): $55.47 • GM/user/mo (ARPU−COGS): $9.00; $29.00; $180.00 • CAC payback (months): 6.16; 1.91; 0.31 • LTV (GM/churn): $150.00; $725.00; $9,000.00 Blended totals • End subs: 18,659.00 • MRR (raw): $401,140.00 • Net revenue: $389,140.00 • Gross margin $: $306,700.00 (GM% 78.81%) • Blended payback: 3.40 months ⸻ 3) Ruthless Instruction-Following (creative; format constraints) Prompt (give this to the models): Write a mini-essay that strictly follows every rule below: 1. Length 180–220 words. 2. Exactly 5 paragraphs. 3. The first letter of each paragraph, in order, must spell GRACE. 4. Paragraph 3 must be a numbered list of exactly 4 items labeled (1)–(4) on separate lines. 5. Include the numbers 37 and 1999 somewhere (as numerals). Do not use any other numerals. 6. Avoid these words entirely: very, really, utilize, leverage. 7. End the whole piece with a single-sentence aphorism ≤ 12 words. How to grade (10 pts): • 6 pts constraint satisfaction (each rule met). • 2 pts coherence/quality. • 2 pts cleanliness (no extra numerals/forbidden words). ⸻ Quick scoring template (optional) • Prompt 1 (10) + Prompt 2 (10) + Prompt 3 (10) = /30 • Tie-breaker: fastest correct JSON in Prompt 1, or fewest rounding errors in Prompt 2. Obviously I break out the grading criteria so the models can’t see it. Give the prompts out in waves, and grade each wave. I have a lot of fun doing it I should just spend the day and build it as a legit workflow or series of workflows and share it all here. I’ve been meaning to formalize it.

Reply

[-]

Borkato@reddit (OP)

Oh this is neat!! Thanks for sharing!

Reply

[-]

Signal_Ad657@reddit

My pleasure my friend. If this is actually an interesting area for people I’d be happy to do more work in it.

Reply

[-]

smarkman19@reddit

Treat it like a tournament: fixed tasks, blind grading, and ELO-style ratings turn hoarding into learning. Pick 5–7 buckets (instruction, code, reasoning, long-context, tool use, safety); write 3–4 prompts each with rubrics or exact answers. Lock system prompt, temp 0/0.2/0.7, max tokens, and stop words; randomize order; anonymize models. Use two different judge families to score pairwise; if they disagree or low confidence, send a few to human review. Add a tiny control set every run to catch judge drift. Score exact-match or pass@k where you can; otherwise use win rate -> ELO; also log latency, tokens/s, VRAM, and truncation. Rotate 20% of prompts monthly and keep a holdout to avoid overfitting. Store prompts, responses, scores, and settings in a DB; Postgres or SQLite + DuckDB works. I run models with Ollama and route with LangChain, using DreamFactory to expose Postgres as a simple REST log I can query and chart without extra glue. Want a minimal schema or judge prompts to start? Make it a repeatable tournament with blind grading, stable settings, and ELO tracking.

Reply

[-]

jacek2023@reddit

1TB? I see you are a beginner

Reply

[-]

0xmaxhax@reddit

‘The surgeon, who is the boy’s father, says “I cannot operate on this boy, he’s my son” Who is the surgeon to the boy?’. The obvious answer is “the father”, but if the model responds “the mother” it often reveals some issues, most prominently pattern overfitting for a specific similar-sounding riddle within its training dataset. For reasoning models which generate CoT, you see additional points of failure such as overthinking and autoregressive error propagation. The test is more for my own curiosity than for actual utility. Shouldn’t really hold much weight, as LLMs can’t actually “reason” no matter how good they are at pretending they can. The best way to test a model is to assess how well it fits your needs; find benchmarks which best fit your use case and run them locally. Also, just try them out individually as stylistic preference is real.

Reply

[-]

Lixa8@reddit

If I intend to build agents/process data with the llms, I have my own little benchmark. For rp, it's pretty much vibes since text quality is difficult to measure. Although there are a few things that could be benchmarked, like reading comprehension. Not sure how meaningful it would be tho.

Reply

[-]

ttkciar@reddit

I have a standard set of test prompts which check the model's competence for specific skills, and a testing script which prompts the model five times with each prompt. That gives me a sense of how reliably models can perform task types, and the range of their competence at each skill. Unfortunately the assessment is completely manual, which takes a lot of time and attention. I was halfway through my assessment of Valkyrie-49B-v2's test results Friday when I had to go home, and I haven't gotten back to it yet. I have a work-in-progress of a second iteration of my test script with not only new prompts, but also automated judging, which won't *replace* manual assessment but should speed it up a lot. I've been wanting to rewrite my tests for a long time, in part because I better understand what makes for a good prompt, and have more skills to test for, but also because a lot of modern models are trained for some of the specific prompts in my test set (and I have actually found them in people's datasets). When my new iteration of the test system is working, I'm not going to share the raw results anymore to avoid that.

Reply

[-]

Steus_au@reddit

what model sizes do you need to test? if the model is large, say 250b and more, then it probably 'knows' everything you could imagine asking.

Reply

[-]

balianone@reddit

svg monalisa in 1 single html file https://preview.redd.it/crdjbwpolj1g1.png?width=450&format=png&auto=webp&s=745795e3491c644420ac9e73366f86082213942b

Reply

[-]

Borkato@reddit (OP)

That’s actually amazing lmao. I’m gonna copy this haha

Reply

[-]

tensonaut@reddit

Depends on your task, you can find benchmarking datasets for most use cases to evaluate upon - look up LLM leaderboards. If you have your own custom task, make your own evaluation dataset.

Reply

How do you test new models?

Reply to Post

27 Comments

Warthammer40K@reddit

Borkato@reddit (OP)

Warthammer40K@reddit

Borkato@reddit (OP)

Warthammer40K@reddit

RiotNrrd2001@reddit

Borkato@reddit (OP)

RiotNrrd2001@reddit

Borkato@reddit (OP)

No-Consequence-1779@reddit

Borkato@reddit (OP)

my_byte@reddit

Dontdoitagain69@reddit

Signal_Ad657@reddit

Borkato@reddit (OP)

Signal_Ad657@reddit

Borkato@reddit (OP)

Signal_Ad657@reddit

smarkman19@reddit

jacek2023@reddit

0xmaxhax@reddit

Lixa8@reddit

ttkciar@reddit

Steus_au@reddit

balianone@reddit

Borkato@reddit (OP)

tensonaut@reddit