I had this same exact idea a while back, but when implementing it I ran into several issues.

One issue is that the way LLMs actually embed and retrieve text. LLMs were trained on normal language with syntax, connectors and structure. If you strip sentences down to these compressed telegraphic fragments, you remove the cues the embedding model uses to understand meaning. This makes retrieval based on semantic embedding harder and more mistake prone.

LLMs are generative. Embedding models are not. As someone else mentioned, if your stored chunks become overly compressed then retrieval becomes noisy or wrong all together which forces the language model to hallucinate more often. I don't see how your solution resolves the issue of worse semantic clustering and noisier nearest neighbor results.

Based on how embedding works, when splitting text into 2 to 5 word fragments it invariably changes granularity. Embedding models will treat very short sentences differently from normal prose. So the result was that it is not actually compressing text, it is altering its information geometry.

You say that "no hallucination occurs because facts are preserved" but the issue isn't about facts. These models don't know or care about facts. They function based on relationships.

Have you done comparison studies showing traditional RAG vs this method?

Does the compressed text embed into the same vector neighborhood as the original paragraph?