Retrieval-augmented generation is easy to describe. Find the passages relevant to a question, give them to a model, and ask it to answer from them. The term comes from a 2020 paper by Lewis and colleagues, which paired a language model with a retriever over a document index, and the pattern has since become the usual way to make AI systems answer from an organisation's own knowledge.
It is also the source of most disappointing AI assistants. The demo works on twenty documents; production has twenty thousand, with versions, permissions, duplicates and questions nobody anticipated. When the answers go wrong, the model gets the blame. Often the cause is upstream: the right passage was never retrieved, or it was retrieved and buried. RAG is a retrieval problem before it is a generation problem.
This note is the architecture. The build — inventory, ingestion, permissions, rollout — is in Building a Production AI Knowledge System.
Two paths
A RAG system has two paths that run at different times and fail in different ways.
| Indexing path (when content changes) | Answering path (on every question) |
|---|---|
| Connect to approved sources | Understand the question: rewrite, split, classify |
| Parse and clean; keep structure | Apply filters: permissions, validity, scope |
| Chunk along the content's own structure | Retrieve candidates: semantic and keyword |
| Enrich: heading path, source, owner, effective date, access rules | Rerank and keep a few |
| Embed and index, with a version | Assemble context: deduplicate, order, fit the budget |
| Propagate changes and deletions | Generate under a contract: cite or abstain |
| Check citations; log what was retrieved, for whom |
Most teams invest in the left column once and the right column never. Both need owners, monitoring and tests.
Decisions that matter more than the vector database
The choice of vector store gets most of the attention and makes the least difference. These decisions make more.
Chunking
Split content along its own structure — sections, clauses, steps, table rows — not at a fixed character count. Keep each chunk's heading path with it: a passage that says "this does not apply" means nothing without the heading that says what "this" is. A useful refinement is to retrieve small, precise chunks but pass the model the larger section they belong to.
Hybrid retrieval
Semantic search finds passages that mean the same thing in different words. Keyword search finds exact terms: product codes, clause numbers, names. Real questions need both. Run them side by side and combine the results; reciprocal rank fusion, described by Cormack and colleagues in 2009, is a simple and robust way to merge ranked lists without tuning weights.
Reranking
First-stage retrieval is built for speed and recall. Take a generous set of candidates and rerank them with a slower, more precise model, then keep only the few best. Reranking is often one of the cheapest improvements to a disappointing RAG system.
Filters before similarity
Permissions, validity dates and scope belong in the query, applied before similarity ranking, not checked afterwards. Filtering after retrieval can return nothing when the top results were all restricted, and it means restricted content has already been handled by the ranking step. The model is never asked to keep a secret, because it never sees one.
The context budget
More context is not free and not always better. Every passage costs tokens and time, and research by Liu and colleagues found that models use information at the beginning and end of a long context more reliably than information in the middle. Pass a few strong passages in a deliberate order, not everything that scored above a threshold.
The answer contract
The model's instruction is simple and strict: answer from these passages and cite them, or say the answer is not in the approved knowledge. An assistant that abstains honestly is worth more than one that is right most of the time and confidently wrong the rest.
Atlas note — When a RAG system gives a bad answer, I look at the retrieval log before I look at the prompt. If the right passage was missing, stale or ranked too low to be used, no prompt will fix it.
RAG is one tool, not the whole architecture
Retrieval over text is the right pattern for policies, procedures, product documentation and other prose that changes occasionally. It is the wrong pattern for several things it is often used for.
| The question needs | Better pattern |
|---|---|
| A live figure: an order's status, a balance, stock | A tool that queries the system of record; never an indexed copy |
| An answer from a small, stable set of documents | Put them in the context directly, with caching; no retrieval needed |
| Several lookups that depend on each other | Retrieval as a tool the agent can call more than once, refining its query |
| Relationships: what else depends on this policy or product? | A graph of entities alongside the text index |
| What the agent learned from its own work | Memory, with provenance and expiry — see Memory is not a database |
For agents, the third row matters most. Instead of retrieving once before every answer, give the agent a search tool with clear parameters — query, scope, date range — and let it decide when to search and how to refine. This handles multi-step questions far better, at the cost of more steps to evaluate and bound.
Freshness and versions
A knowledge system is only as current as its slowest update.
- Incremental indexing. Changes reach the index quickly; nobody waits for a nightly rebuild to correct a wrong policy.
- Deletion propagates. When a source document is removed or expires, its chunks and any cached answers go with it. Retention follows the source.
- Embedding versions. Vectors produced by different embedding models are not comparable. Changing the embedding model means re-embedding the whole corpus into a new index, testing it, and switching over deliberately — so plan for two indexes to exist side by side.
- Record the version. Log which index version, which passages and which model produced each answer, so any answer can be reconstructed later.
Evaluate retrieval and generation separately
If you only score final answers, you cannot tell which half failed. Keep a set of real questions with their correct answers and the passages that support them, and measure:
- Retrieval — does the supporting passage appear in the candidates, and how high is it ranked?
- Faithfulness — is every statement in the answer supported by the passages given?
- Citations — do the citations point to the passages that actually support the claim?
- Abstention — when the answer is not in the sources, does the system say so?
- Permissions — do deliberate probes for restricted content come back empty?
The Guild's Evaluation Lab scorecard includes factual accuracy, citation quality and retrieval accuracy among its quality dimensions, and the Agent Production Readiness Kit shows where knowledge tests sit in a full go/no-go decision. Re-run the set after every change to sources, chunking, embeddings, prompts or models.
Security
Two risks are specific to RAG. The first is indirect prompt injection: retrieved text can contain instructions, and the model must treat it as content, never as commands — OWASP lists prompt injection first among its risks for LLM applications. The second is the vector store itself. OWASP's category for vector and embedding weaknesses covers unauthorised access, leaks between users or tenants who share a store, attacks that reconstruct source text from embeddings, and poisoned content, and it recommends permission-aware vector stores. Treat the index as a copy of the most sensitive document in it, and protect it accordingly.
The note Agent permissions and tool boundaries covers how retrieval tools fit into an agent's wider boundaries.
How it fails
| Failure | What you see | What fixes it |
|---|---|---|
| Missing passage | Confident answer from general knowledge | The answer contract; abstention tests; coverage of sources |
| Buried passage | Right document retrieved, wrong answer | Reranking; smaller context; ordering |
| Lost heading | A clause applied to the wrong case | Structure-aware chunking with heading paths |
| Stale content | Last year's policy quoted | Incremental indexing; deletion propagation; owners |
| Conflicting sources | Different answers to the same question | One owner per topic; retire duplicates |
| Permission leak | Restricted content in an answer | Filters before similarity; probe tests |
| Silent drift | Quality falls with no change anyone remembers | Versioned indexes; regression runs |
An example
Illustrative example, not a client engagement.
An internal assistant answers questions about leave policy. Asked how many unused days a part-time employee can carry over, it gives the full-time figure. The retrieval log shows why: the policy was split every few hundred characters, and the passage with the carry-over rule had lost its heading, "Part-time staff". Re-chunking along the document's sections, with heading paths attached, fixes this question and others like it. No change to the model or the prompt was needed.
In the Guild's AI engineering work, this is the Knowledge Foundation: a source inventory, permission-aware RAG, and rules for freshness and citation. In the Citadel, approved context and sources form a layer of their own, separate from the agents that use them — which is exactly where a RAG system belongs.
