Retrieval-Augmented Generation: What RAG Actually Buys You

Every few months someone announces that RAG is dead. Usually it's the week after a new model ships with a bigger context window, and the argument is always the same: why bother retrieving anything when you can just paste the whole corpus into the prompt?

I've built enough of these systems to have a boring answer. RAG isn't dead. A specific, lazy version of it is, and honestly it had it coming.

Let me back up.

What RAG is, without the metaphor

The pitch usually arrives wrapped in a metaphor about a student looking things up in a library. It's not wrong, but it hides the part that matters. A language model only knows what it saw during training. That knowledge is frozen at a cutoff date, it has no idea about your internal documents, and it will state all of it with the same serene confidence whether it's right or not.

Retrieval-Augmented Generation is the unglamorous fix: before the model answers, you go fetch relevant text from somewhere it can actually trust — a document store, a database, a search index — and you put that text in front of the model as part of the prompt. The model then answers using what you handed it rather than whatever it half-remembers.

That's the whole idea. Retrieve first, generate second. Everything else is implementation detail, and the implementation detail is where all the interesting failure lives.

The version that deserved to die

The demo that made everyone excited in 2023 was three steps. Chunk your documents, embed them into a vector database, and at query time pull the top few chunks by cosine similarity and drop them into the prompt. It worked in the demo. It works in your demo too. Then you point it at a real corpus and the wheels come off.

Pure vector similarity is semantic, which sounds like a feature until it quietly drops the one document that used the exact product code the user typed, because "semantically" that code looks like noise. It has no notion of whether the chunk it retrieved is actually the best answer or merely a plausible-looking one. And chunking — the least discussed, most consequential decision in the whole pipeline — will happily slice a table in half or sever a sentence from the caveat that reverses its meaning.

None of this is a reason to abandon retrieval. It's a reason to stop pretending naive retrieval was ever the finished product.

What good retrieval actually looks like now

The pattern that has settled in for production systems is less elegant and considerably more effective.

You retrieve with hybrid search — dense vector similarity for meaning, plus old-fashioned keyword search (BM25) for the exact terms, then merge the two rankings together. This isn't nostalgia for keyword search; the measured gap is real. Hybrid consistently beats semantic-only retrieval by high single-digit points on ranking quality, mostly by catching the exact-match cases pure embeddings fumble.

Then you rerank. The first retrieval pass is optimised for speed over a large index, so it's approximate by design. A cross-encoder reranker takes those candidates and scores each one against the query properly, jointly — slower, but far more accurate about what's actually relevant. The rule of thumb people have converged on is almost suspiciously tidy: retrieve twenty, rerank down to five, send three to five to the model. Retrieve wide, then be ruthless.

That reranking step is the highest-leverage thing most teams are still missing. It's also the least exciting to put on a slide, which is probably why.

But what about the giant context window?

This is the real argument, so let's take it seriously. Frontier models now ship with million-token context windows. Why retrieve a handful of chunks when you can hand the model the entire knowledge base and let it sort things out?

Three reasons, and none of them are ideological.

Cost is the blunt one. Feeding a large corpus into the prompt on every query isn't slightly more expensive, it's a different order of magnitude — the comparisons floating around put a retrieval query near a hundredth of a cent against roughly ten cents for the stuff-everything approach. Run that at any real volume and the difference stops being a rounding error and starts being the budget.

Latency is the second. Prefill scales with how much you put in the prompt. A model that has to read a hundred thousand tokens before it says anything feels like it's thinking hard, in the bad way.

And attention quality is the quiet one. Even when a model technically accepts a million tokens, its ability to reason carefully across all of them degrades as you fill the window. Burying the three relevant paragraphs inside a hundred thousand irrelevant ones is not a neutral act. You paid more, waited longer, and made the problem harder.

The consensus that's emerged is the sensible one: use retrieval to decide what the evidence set should be, and use the long context window to reason over that bounded set. Retrieval picks the exhibits. Long context reads them. They were never really competitors.

Where it earns its keep

RAG is worth the trouble when the answer depends on information the model couldn't have memorised: your internal wiki, this quarter's numbers, a policy that changed last week, anything specific and current and yours. Customer support over a live knowledge base, question-answering over legal or medical corpora where the source matters as much as the answer, any assistant that needs to cite where it got something — that's the home turf.

It is not worth the trouble when the model already knows the answer perfectly well, or when the task is reasoning rather than recall. Bolting a retrieval pipeline onto a problem that didn't need one is its own kind of overengineering, and I've watched teams do it because RAG was the thing you were supposed to have that quarter.

The part nobody demos

The failure modes are all operational, and they're all boring, which is exactly why they bite. Retrieve a wrong or stale document and the model will fold it confidently into an answer — you haven't removed hallucination, you've handed it better source material to be wrong with. Your retrieval quality is capped by your data quality and your chunking, both of which need maintaining long after the launch demo is over. And every stage you add — hybrid search, reranking, the fetch itself — is another few hundred milliseconds and another thing that can page you at 2am.

RAG doesn't make the model smarter. It makes it better informed, which is a different and more useful thing. The intelligence was always going to be the model's problem. Retrieval just decides what the model gets to look at before it opens its mouth — and most of the time, that turns out to be the decision that actually mattered.