Any advice for testing similar versions of the same model?

Posted by Borkato@reddit | LocalLLaMA | View on Reddit | 5 comments

For example a heretic version vs the standard vs unsloth vs one merged with something else - are there any particular things to look out for?

Reply to Post

5 Comments

[-]

Velocita84@reddit

Normal benchmarks to test real world degradation, KLD to test output token distribution divergence

[-]

If it's for comparing different quant/finetune for quality drop, you can just use Huggingface/Lighteval. This is how to run with local mode. https://www.reddit.com/r/LocalLLaMA/comments/1po4wwe/run_various_benchmarks_with_local_models_using/

[-]

DinoAmino@reddit

Lol. I remember that. You posted it after I showed you how to run it. https://www.reddit.com/r/LocalLLaMA/s/Wy0OlaJyXa

[-]

Borkato@reddit (OP)

Oh this is lovely thank you!

[-]

FuckingMercy@reddit

Do benchmarking on your own data, if you want to be methodical about it that's the only way to go. If you want a half-moon answer, try to test "edge case behaviour" like uncommon languages. stuff that you know were not super common in the training and post training datasets...