Retrieval-augmented generation gets described like a spell: connect the model to your documents and it suddenly knows your business. Then a team wires one up, asks it a question their own handbook answers on page two, and watches it come back with something confidently vague.
The disappointment is predictable. The framing was wrong from the start: RAG isn't a capability you switch on. It's a data pipeline, and it behaves like one. The quality of what comes out is set almost entirely by the plumbing between the question and the model.
The model is rarely the problem
When a RAG system gives a bad answer, the instinct is to blame the LLM or reach for a bigger one. In our experience, the failure usually happened earlier: the right passage was never retrieved, so the model answered without it. No model can cite a paragraph it never saw. Before touching anything else, look at what retrieval actually returned for the failing question. It's humbling, and it redirects the work to where the leverage is.
Your documents are messier than you think
Every company believes its documentation is roughly fine. Then you index it and discover three conflicting versions of the vacation policy, a critical process that exists only as a screenshot, and headers that say "FINAL_v2_updated". A retrieval system faithfully surfaces all of it, contradictions included. A lot of what gets called prompt engineering is actually just cleaning up the source material: deciding what's current, what's authoritative, and what should never be indexed at all.
Chunking is an editorial decision
How you split documents decides what the system can find. Split a contract clause across two chunks and neither half means anything; lump a whole policy into one chunk and retrieval loses precision. There's no universal setting, because the right split follows the structure of the material. Contracts want clauses, handbooks want sections, invoices barely want chunking at all. This is editorial work. Someone has to understand the documents.
Measure it like software, not vibes
Teams evaluate RAG by chatting with it for ten minutes and forming an impression. That impression doesn't survive contact with real users, who ask questions nobody anticipated, in phrasings nobody tested. We keep a plain list of real questions with known correct answers, and every change to the pipeline runs against it. It's not glamorous. It's also the only way to know whether yesterday's clever tweak made things worse.
Where that leaves the magic
None of this diminishes what these systems can do. When retrieval is right, the answers are right, the citations are real, and people stop pinging a colleague for things the handbook already covers. It just doesn't come from the spell. It comes from the plumbing, which is good news for anyone willing to do the work. Plumbing is a thing you can fix.