RAG Chunking: The Step That Quietly Breaks Everything

Chunking is the least glamorous part of RAG, and it is the one that decides whether the whole thing works.

Nobody demos their chunking strategy. It gets a single line in the setup script and no further thought. Then the system fails in production and the team blames the model, the embeddings, the vector database, anything but the quiet decision they made at the very start about how to cut their documents into pieces.

What Chunking Is

A model cannot retrieve a whole hundred-page document. So before indexing, you split documents into smaller pieces, chunks, and each chunk becomes a unit that can be retrieved on its own. When someone asks a question, the system pulls the chunks that match, not the whole document.

That makes each chunk a promise. It promises that the piece, on its own, carries enough meaning to answer a question. Break that promise and retrieval breaks with it, because a chunk that has lost its context cannot be rescued by anything downstream.

Why Fixed-Size Chunking Fails

The default approach is fixed-size chunking. Cut every 500 tokens, or 800, and move on. It is simple, and it is the root cause of retrieval failures in most pipelines.

The problem is that documents do not organize themselves in 500-token blocks. A fixed cut splits sentences mid-thought, tables mid-row, and rules mid-clause. I have watched a compliance rule get severed by a chunk boundary, so the retrieved piece held half the rule and the model answered from the half it got. The chunk was relevant. It was also incomplete, which made it worse than useless, because it looked like a real answer and quietly left out the condition that changed everything.

What to Do Instead

Better chunking respects the structure of the document.

One. Semantic chunking splits on meaning, keeping related sentences together and breaking where the topic actually shifts, so a chunk is a complete thought rather than an arbitrary window.

Two. Proposition-based chunking goes further, turning content into self-contained statements that each stand alone and answer a question on their own.

Three. Hierarchical chunking preserves the document's structure, so a chunk knows which section and heading it belongs to, and the context above it is not lost.

The good news is that this is cheap. Switching off fixed-size chunking costs almost nothing and lifts retrieval accuracy immediately. It is the highest-return change most teams can make, and most have not made it because it sits in the unglamorous setup step nobody revisits.

The Deeper Problem

Step back and chunking is really about one thing. Does the meaning of your document survive being cut into pieces?

A good chunk preserves meaning. A bad chunk destroys it, quietly, at index time, before anyone can see. Everything downstream, retrieval, reranking, generation, is working with whatever survived the cut. If the meaning did not survive, no amount of clever retrieval can recover it. You cannot retrieve a rule that was split in half. As a whole, coherent thing, it no longer exists in your index.

This is exactly why I built CORDspec around preserving meaning as data moves and gets transformed, with the Envelope Fidelity Score to measure how much survived. Chunking is the first place RAG either keeps the meaning or loses it. Treat it like the foundation it is, not the throwaway line it usually gets, and you fix more of your RAG problems in an afternoon than a bigger model would fix in a quarter.

How to Fix Your Chunking This Week

Chunking is the rare RAG fix that is both high-impact and cheap. Three moves.

One. Stop using fixed-size chunks. That single change removes the most common source of retrieval failure.

Two. Switch to semantic or hierarchical chunking, so a chunk is a complete thought that keeps its place in the document.

Three. Test it on real questions. Take twenty questions you know the answers to and check whether the right, complete chunk comes back. If it does not, your chunking is still cutting meaning.

This is the highest return on effort available in a RAG pipeline, and it lives in the step everyone treats as setup and forgets.

Structure before tools. Strategy before speed. Systems before scale.

The AI Operator Framework is the methodology for designing, governing, and scaling AI operations inside a business, including the data and citation fidelity layers most companies skip. Explore the full framework at JamilAshkar.com/framework. Start with the free course at JamilAshkar.com/course.