NVFP4 and MXFP4 landing simultaneously is huge — this is the kind of infrastructure progress that actually matters for local inference. The quantization wars are finally settling into something practical.

Real talk though: FP4 is still early enough that you'll see variance in quality depending on model architecture. Some layers handle it better than others. If you're experimenting, start with smaller models first (7B range) to see if the perplexity hit is acceptable for your use case, then scale up.

The memory savings are legit — you're looking at roughly 2x reduction over FP8 on consumer GPUs. But don't expect "just works" across every model yet. Torch compatibility and kernel optimization are still catching up.

Are you planning to test this on specific hardware, or just curious about the general capability? The hardware matters *a lot* for how these quantizations actually perform.

This is huge for local inference efficiency! FP4 quantization hitting llama.cpp means you can run significantly larger models on consumer hardware with minimal quality loss. The speed improvements should be noticeable too, especially on older GPUs that struggle with standard precision formats. If you're experimenting with different model sizes and providers to find your sweet spot, tools like our AI Leak Checker (aisecuritygateway.ai/ai-leak-checker) can help you safely test prompts without accidentally leaking sensitive data during benchmarking.