Drummer's Precog 24B and 123B v1 - AI that writes a short draft before responding

Yes and no. Yes, it's a pitfall of LLM inference in general, but hallucination is probabilistic, with hallucination growing exponentially more likely with token count, and "thinking" infers a lot of tokens. Review of basic probability: If P1 and P2 are the probabilities of two events occurring, then the probability of both events occurring is P1 * P2. If P3 is the probability of another event occurring, then the probability of all three events occurring is P1 * P2 * P3. If there are N events, and their probabilities are all about the same, then the probability of N events all happening is P^N If the probability of an event occurring is P, then the probability of N events *not* occurring is 1-P, and the probability of N events all not occurring is (1-P)^N Of interest here is the probability of hallucination, call it H, which would make the probability of no hallucination 1-H. If the probability of a given inferred token being a hallucination is P (ignoring variations of probability between tokens as a simplification), then the probability of none of N tokens being a hallucination is (1-P)^N which makes the probability of hallucination anywhere in inference is 1-(1-P)^N To see how this can snowball, let's put some hypothetical numbers into that. Say the typical answer is 300 inferred tokens, the typical "thinking" phase inferrs 500 tokens, and the probability of any given token being a hallucination is P = 0.00002. That would make the probability of hallucination without thinking 1-(1-0.00002)^300 = 0.006, or 0.6%. The probability of hallucination during the thinking phase would be 1-(1-0.00002)^500 = 0.010, or 1.0%. The probability of hallucination any time during the thinking + answering inference would be 1-(1-0.00002)^800 = 0.016, or 1.6%. Thus hallucination during thinking + answering would be about 2.7x more probable than hallucination while inferring an answer without thinking (though, remember, these are made-up numbers, used to illustrate the *shape* of exponential functions). Just as exponential functions are sensitive to large N, so are they sensitive in the opposite sense to small N, so by keeping the length of the "thinking" phase very short, TheDrummer has largely mitigated the problem. The probability of hallucination in inferring short thinking + answer is much closer to the probability of hallucination in inferring just the answer. Illustrating that, let's say the typical thinking phase infers just 100 tokens, using the same P as before. Probability of hallucination in thinking + answer: 1-(1-0.00002)^400 = 0.008, or 0.8%, which is only 1.3x the likelihood of hallucinating in the answer alone. I'll note, here, that a variation of Self-Critique can frequently catch and correct hallucinations, so pipelining models can also be used to mitigate the risk: * Infer just the "thinking" phase (easily enough done with an llama.cpp grammar), * Infer a critique of the "thinking" content, * Infer a rewriting of the "thinking" content based on the critique, * Inject the rewritten "thinking" content into the `<think></think>` part of the prompt, and infer the reply based on that. On one hand, Self-Critique has the advantage of not only catching (most) hallucinations, but also improving the merit of the inferred content just in general. On the other hand, all this extra inference takes a lot more time, whereas keeping the "thinking" phase short results in inference taking *less* time. It's all tradeoffs.