It is very difficult to say for certain. I am using FP8 and Q6_K right now, mostly because Q6_K is slightly faster than Q8_0, and shouldn't be any worse. For instance, here is unsloth showing results for the 35b-a3b: https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs which suggests that mean K-L divergence is less than 0.01 from 5 bits onwards, and my guess is that dense model is less sensitive to quantization than 3b active MoE.

Very roughly, it seems that each additional bit in weights will halve the mean K-L divergence, until we are at 6 bits and improvement seems to stop, despite we are not yet near zero. If we extrapolate the early part of the graph, from e.g. 2 to 4 bits, we can see that rate of improvement is slowing down, and that 5 bits are improving less than that linear trend, and 6 bits only very slowly, and assuming that trend continues, then q8_0 is much bigger but again only very slightly more faithful to the original.

It also behooves to remember the implication of a logarithmic y-axis: the original model's point would reside at the negative infinity in this scale, and by the time bf16 is used, the divergence is zero and that is what you would get. However, even the slight and random perturbations in the model weights due to the quantization seems to cause enough error that K-L divergence can't get much better than somewhere in 0.001 and 0.002. I personally do not think that these differences are very significant at 6 bits and beyond, and in fact task performance is usually reasonable even down to 4 bits, despite I can see with a 4-bit model that the thinking output has become more confused and model no longer accurately seems to be able which was its own output from user's command, and begins to make more tool call errors and restates paths incorrectly, etc.