Idk is it worth like 5-10 grand?

As disappointing as it is, my answer is: it depends.

1. Do you have that kind of money to dedicate to AI? Keep in mind, we have all the way from 10 year olds to 80+ year olds (probably lower and higher) posting here on reddit; major differences in economic situations and time to compound between the 2 groups.
2. If you do have that kind of money, do you want 96GB VRAM that's super freaking fast with a ton of compute? Or half a terabyte of memory to run a massive model with way worse prompt processing and inference speeds (mac studios)? Or some middle ground with worse memory bandwidth but better compute and more memory (epyc w/12-channel DDR-5)? Or do you want to run 3 5090's and do tensor parallelism but eat the power bill from it?

So - it depends. At that price point (say 5-10k), there's a lot of directions to go, and it seems like each offering kind of has its own niche based on some hardware limitations and a balance of tradeoffs between speed, size, power consumption, and compute.

MOE is a godsend for sure, but tbh I remain more impressed with gemma-3-27b as a model for pretty much anything other than code gen, and the qat version of that is, while not quite where you'd need with 12gb VRAM, still quite modest at 16.8GB at Q4 (link) or just over 14GB at 3.0bpw with exllamav3.

12GB of VRAM on a 4070 makes a ton of sense for gaming; that's a great amount of footprint there. Just when it comes to LLMs and needing VRAM for these super sparse, redundant models, turns out using a GPU isn't exactly what they were designed for. So it's pretty amazing we can get as far as we have with the current general-purpose GPU architecture, but just take a peek at what groq is doing for inference or google with their TPUs and realize we're all kind of hammering square pegs into round holes when it comes to our approach to inference right now.