Retrieval-augmented generation, or RAG, is an application pattern that retrieves relevant external information for a user task and supplies selected context to a generative model. The model then produces an answer conditioned on that context rather than relying only on its learned parameters.
Adding a vector database does not make answers factual. The system can index the wrong content, retrieve an irrelevant passage, omit a decisive clause, cross a permission boundary or generate a claim that the passage does not support. End-to-end quality hides whether the failure came from retrieval or generation.
RAG is a controlled evidence pipeline. Manage source authority, permissions, parsing, retrieval, context assembly, citations and answer policy as distinct stages, and evaluate each stage with questions drawn from the real workflow.
Architecture
RAG has multiple failure points
The ingest path discovers content, checks access, parses structure, segments text, attaches metadata and updates the index. The query path interprets the task, filters permitted content, retrieves candidates, reranks passages, assembles context and generates an answer. Citation rendering and feedback happen after generation.
A failure at an early stage propagates. If the table parser loses a column, no retriever can recover the missing relationship. If the permission filter is late, sensitive content may already be exposed. If the source link is lost during chunking, the answer cannot provide useful provenance.
| Stage | Failure example | Diagnostic |
|---|---|---|
| Ingestion | Current file not indexed | Is the authoritative source present? |
| Retrieval | Required clause absent | Was evidence in the candidate set? |
| Context | Near-duplicates crowd out detail | Did the prompt receive enough evidence? |
| Generation | Claim exceeds the passages | Does each material claim follow? |
| Citation | Link opens wrong version | Can the user verify the source? |
Boundaries
RAG improves access to evidence, not automatic truth
RAG is useful for knowledge that changes, belongs to the organisation or needs citations. It does not replace deterministic databases for exact transaction state, policy engines for permissions or calculation code for financial results. Those systems can be exposed as tools when their output is needed.
Retrieval also cannot resolve a source that is wrong or genuinely ambiguous. The application should represent source authority and conflict, and it should decline to synthesize certainty when the evidence does not support it. Human owners remain necessary for high-impact interpretations.
- Authorize before adding context.
- Preserve source version and exact location.
- Evaluate retrieval before blaming the model.
- Validate citations for support, not mere overlap.
- Use abstention for missing or conflicting evidence.
What good looks like
Useful outcomes from retrieval-augmented generation
- Answers use current and permitted source material.
- Users can inspect citations that actually support material claims.
- Retrieval failures are distinguished from generation failures.
- Content changes can be indexed and tested without retraining the model.
- The system abstains when authoritative evidence is missing or conflicting.
Operating model
How to run the work
- 01
Define source authority and access
Identify which repositories are authoritative for each question and who may see them. Preserve document identity, version, effective date and ownership. Apply user permissions before or during retrieval so a relevant but unauthorized passage never reaches the model.
- 02
Prepare retrievable content
Parse headings, tables, lists and metadata without discarding structure. Choose chunk boundaries that preserve complete meaning and retain links to the source location. Detect duplicates, superseded documents and failed extraction before indexing.
- 03
Retrieve and assemble context
Use lexical, semantic or hybrid search with metadata filters and, where useful, reranking. Inspect whether the required evidence appears in the selected context, not merely somewhere in the corpus. Allocate context to diverse supporting passages rather than repeated near-duplicates.
- 04
Generate, cite and evaluate
Instruct the model to answer within the supplied evidence, expose uncertainty and cite source locations. Validate citation entailment and structured fields. Evaluate retrieval recall, context precision, answer correctness, unsupported claims and abstention separately.
Evaluation
Questions that change the decision
- Which source is authoritative when repositories conflict?
- What chunk and metadata preserve the meaning needed for this task?
- Should retrieval be lexical, semantic, hybrid or tool-based?
- How much evidence must be present before the system may answer?
- What citation granularity lets a reviewer verify the claim quickly?
Failure modes
Where teams lose control
Unauthorized content enters the prompt through retrieval.
Poor parsing breaks tables and separates qualifiers from their clauses.
Top-ranked passages can be relevant in topic but insufficient for the answer.
The model can cite a real source that does not entail its statement.
Stale duplicates outrank the current authoritative document.
Measurement
Measure the finished job
Measure the completed workflow, including review effort and exceptions. Output volume on its own is not evidence of a better process.
- retrieval recall for required evidence
- precision and diversity of selected context
- answer correctness and completeness
- citation entailment for material claims
- unsupported answer and appropriate abstention rate
- index freshness and extraction failure rate
Questions
Common questions
What does RAG mean in AI?
RAG means retrieval-augmented generation. An application retrieves relevant external information and gives it to a generative model as context for the answer.
Does RAG stop hallucinations?
No. It can provide better evidence, but retrieval may fail and the model may still make unsupported claims. Evaluate retrieval, answer correctness, citations and abstention separately.
Is RAG the same as a vector database?
No. A vector index can be one retrieval component. A complete RAG system also includes source governance, parsing, permissions, query handling, ranking, context assembly, generation, citations and evaluation.
When should a company use RAG?
Use it when answers need current, private, domain-specific or citable knowledge. Prefer direct database queries or deterministic tools for exact transactions, calculations and rule enforcement.
Sources
Primary references
About the author
Tony Kim
Tony writes about applied AI, dependable product engineering and the systems that turn complex response work into controlled delivery.
Zeke
AI product engineering for moving a software brief into a reliable production product.
Product leaders, founders and engineering teams. Start with the workflow, constraints and evidence you already have.
See Zeke→