I have been doing some experiments with LLMs for classification, specifically leveraging logprobs as proxy measures of uncertainty. These are very neatly exposed by LocalLLaMA (and some API-based LLMs), but I feel they are still quite under-explored.

In my latest article (part of a series), I look at a few applications:

* Identifying noisy samples: Using entropy to find noisey samples

* Detecting distribution shifts: Using the log margin as a signal for when your data source changes

* Threshold tuning: Using log probs to balance the Precision vs. Recall trade-off

Full write-up here: https://gerardsimons.com/articles/llm-as-classifier-part-3

I’m very keen to hear everyone's thoughts and experience with this, and possible other applications. One thing I’ve noticed is how wildly these values can differ from problem to problem and model to model, which can make it a rather noisy signal to calibrate.