RAG, Done Properly: Why Chunks Are the Wrong Unit of Knowledge

The chunk is a parsing convenience, not a knowledge unit

The chunk-as-unit assumption is the fundamental bug in most RAG pipelines. A chunk is a parsing convenience that became a retrieval assumption. It has no idea boundary, no version context, and no access state.

Retrieval stacks have spent years patching that mismatch: rerankers, hybrid search, threshold tuning, prompt engineering. All of it is downstream of the real problem.

You are tuning retrieval to compensate for a broken unit of knowledge. Fix the unit upstream and most of the downstream problems disappear on their own.

Why chunks fail

A chunk is structurally neutral. The splitter cuts wherever the token count runs out, so you can retrieve half a table, a conclusion without its argument, or a claim stripped of the context that justifies it.

Enterprise corpora have dozens of near-identical versions of the same material. Top-K returns five copies of the same paragraph, current and deprecated mixed together. The LLM produces confidently wrong answers because the retrieval layer never promised it a clean set of claims.

Most stacks have nothing between the document parser and the vector store. That gap is where the three problems, idea boundaries, version state, and access control, compound.

The fix: embed question-answer packets

Instead of embedding text windows, embed question-answer packets. Each IdeaBlock contains one question, its validated answer in 2 to 3 sentences, and typed governance fields: clearance level, version state, and source.

When you embed a question-answer packet instead of a text window, your embedding represents a single atomic claim, not a chunk of narrative that happens to contain it.

Access control works differently too. A sales engineer querying the same index as a legal reviewer gets a different dataset not because the retrieval layer filters it, but because the blocks themselves carry the access boundary.

On a benchmark of 17 documents and 298 pages, IdeaBlocks reduced the average cosine distance from 0.3624 to 0.1585. Semantic distillation took 2,042 raw blocks down to 1,200 canonical blocks. Word count dropped from 88,877 to 44,537. The distilled dataset outperformed the undistilled version by 13.55% on vector accuracy. None of this required changing the retrieval algorithm, reranker, or embedding model.

Why deduplication is not a cleanup step

Fifteen near-duplicates of the same paragraph create fifteen competing vectors in the same region of embedding space. Retrieval distributes probability mass across all of them, pulling the match score down for the canonical version.

IdeaBlocks clusters by cosine similarity at an 80 to 85% threshold across 3 to 5 iterative rounds. Near-duplicates are merged via a specially tuned LLM.

The result is one clean vector per claim instead of fifteen competing ones. Retrieval gets sharper because the embedding space is no longer polluted by duplicates.

The seven-stage pipeline

What changes downstream

Query matching shifts from probabilistic, hoping the right paragraph floats up, to structural, with direct question-to-answer matching. Governance moves from logic bolted onto the orchestrator to typed fields on each block.

Access control moves from retrieval-layer filters to blocks that carry their own access boundary. Updates change from finding all duplicate passages, which is intractable at scale, to updating one IdeaBlock that propagates to every app that consumes it.

The system gets simpler, not more complex. You remove downstream patches because the upstream unit is finally correct.

When this is worth doing

Browse AI infrastructure tools and LLM tools, or read the companion guide on inference engines compared.