How RAG Works, Step by Step
RAG is five steps. Most explanations stop at three, which is exactly why most RAG breaks.
The tutorial version goes: embed your documents, retrieve the top matches, generate an answer. That version runs a demo. It does not survive production, because it skips the two steps that decide whether retrieval is any good. Here is the full pipeline, honestly, with the failure points marked.
Step One: Index
Before you can retrieve anything, you prepare your data. Documents get split into chunks, and each chunk gets converted into an embedding, a list of numbers that represents its meaning. Those embeddings go into a vector database.
This step looks like plumbing, and teams rush it. That is a mistake. How you chunk decides what can ever be retrieved. Split a rule across two chunks and no query will ever pull the whole rule, because it no longer exists in the index as a whole thing. The index is the foundation. Everything downstream inherits its flaws, and no clever retrieval recovers what indexing threw away.
Step Two: Retrieve
At question time, your query gets embedded too, and the system finds the chunks whose embeddings sit closest to it. This is the nearest-neighbor search that vector databases are built for. It is fast, and it is approximate.
Approximate is the key word. Vector search finds what is mathematically similar, which is not always what is actually relevant. A query about one project can pull the near-identical document for a different project, because the phrasing and structure look alike. Retrieval gets you into the right neighborhood. It does not get you to the right door.
Step Three: Rerank
This is the step the tutorials skip, and it is the one that separates a demo from a production system.
A reranker takes the pool of chunks that retrieval returned and re-scores them on how well they actually answer the question, using a model that reads the query and the chunk together rather than comparing precomputed embeddings. The standard production pattern is to retrieve a large pool, rerank it, and pass only the top few to the model. Hybrid search with a reranker cuts error rates by roughly 69 percent compared with naive vector-only retrieval. Skipping this step is the single most common reason production RAG underperforms, and it is an afternoon of work to add.
Step Four: Augment
Now you build the prompt. The top reranked chunks get inserted into the context, along with your question and instructions. This is the augment in Retrieval-Augmented Generation.
There is a limit here people ignore. Stuff too much retrieved text into the context and the model's attention degrades. It reads the early chunks well and the later ones poorly. More context is not better context. The discipline is retrieving broadly, reranking hard, and passing only the highest-quality evidence, so the model sees a small clean set instead of a large noisy one.
Step Five: Generate
Finally, the model reads the assembled context and writes the answer. If the previous four steps did their job, this step is easy and reliable. The model is not reasoning from memory. It is summarizing evidence you handed it.
Which is why blaming the model when RAG fails is almost always wrong. By the time generation happens, the outcome is mostly decided. The answer was determined at index and retrieval, two steps nobody was watching, and the model is just the last mile.
Where the Steps Fail
Trace the failures backward and they cluster at the start. Bad chunking at step one. Approximate matching at step two. No reranking at step three. By step five the model is doing its best with evidence that was already wrong.
Every one of those early failures is a meaning problem. Meaning lost in chunking, meaning compressed out of embeddings, meaning that never survived from the document to the retrieved chunk. That is the layer I work on with CORDspec, and the Envelope Fidelity Score exists to measure exactly this, how much meaning made it through the pipeline intact. Fix the early steps and generation takes care of itself. Obsess over the model and you are polishing the one step that was never the problem.
Where Most Teams Should Focus
If the five steps feel like a lot, here is the shortcut. Two of them decide almost everything, and they are the two the tutorial skipped.
One. Chunking, at index time. Get this wrong and nothing downstream can recover, because the right answer never made it into the index as a whole thing.
Two. Reranking, between retrieval and generation. Add it and you cut error rates dramatically for an afternoon of work.
Fix those two and most RAG systems go from unreliable to production-grade. The model, the part everyone obsesses over, is the step you should touch last.
Structure before tools. Strategy before speed. Systems before scale.
The AI Operator Framework is the methodology for designing, governing, and scaling AI operations inside a business, including the data and citation fidelity layers most companies skip. Explore the full framework at JamilAshkar.com/framework. Start with the free course at JamilAshkar.com/course.
