RAG application development combines a language model with a governed retrieval layer that selects relevant source material at request time, supplies it as context and preserves evidence for the generated answer or action.
A convincing prototype can be built from a folder and a vector database in days. Production fails on different questions: which version is authoritative, whether the user may see the source, how tables and scans are parsed, what happens when evidence conflicts, and how the team knows an answer is grounded rather than merely plausible.
RAG is an information product, not a prompt accessory. Retrieval quality, source governance, answer policy, interface and operational feedback must be designed together. The objective is not to make every question answerable. It is to answer supported questions usefully and refuse, qualify or escalate the rest.
Architecture
Separate retrieval, answer policy and generation
Retrieval determines which evidence is available. Answer policy decides what may be concluded from it. Generation turns the permitted conclusion into useful language or structured output. These layers can use models, but they need separate interfaces and tests. Otherwise a prompt change can quietly alter access handling, refusal behaviour and business logic at once.
Preserve a response trace with user context, query transformation, eligible source set, retrieved passages, ranks, policy decision, model version, citations and outcome. Sensitive payloads may require redaction or limited retention, but the product still needs enough observability to diagnose why a failure occurred and whether other answers share it.
| Layer | Question | Evidence of quality |
|---|---|---|
| Ingestion | Did the authoritative content and structure survive? | Versioned parse fixtures and source-coordinate checks |
| Retrieval | Did required permitted evidence reach the context? | Judged test questions and recall by source class |
| Answer policy | Was the evidence sufficient for this response mode? | Expected answer, qualification, refusal and escalation labels |
| Generation | Does each material claim follow the supplied evidence? | Claim-to-citation review and contradiction tests |
| Product | Did the user complete the real task safely? | Task outcome, correction effort and incident signals |
Evaluation
Build the test set before optimizing the demo
Collect real question shapes from intended users and create cases for direct facts, synthesis, comparison, temporal queries, tables, ambiguous wording, no evidence, conflicting evidence and restricted content. Each case should identify the evidence that must be found and the acceptable response behaviour. Synthetic expansion can increase coverage after domain owners establish the core cases.
Report metrics by slice. A system may work for short policies and fail on scanned annexes, or answer English questions well while losing German compound terms. Track regressions against a fixed set and add production failures as reviewed cases. Do not tune on the same examples used to claim final quality.
- Separate retrieval relevance from answer correctness.
- Include questions whose correct result is no answer.
- Test every permission role with identical query wording.
- Measure citation support at claim level, not page presence.
- Keep a held-out release set and explicit regression budget.
Delivery
Select an engineering partner through a production slice
A credible engagement begins with one bounded task, real sources, real access distinctions and a measurable acceptance set. The first vertical slice should include ingestion, retrieval, answer policy, interface, evaluation and observability. A chat screen backed by manually cleaned documents proves too little about ongoing operation.
Ask how the partner handles source deletion, permission changes, reindexing, model replacement, cost ceilings and incident review. Require code, infrastructure definition, evaluation data, runbooks and ownership transfer. The organization should be able to change a model or retrieval component without rebuilding the entire product or losing historical comparability.
- Use a domain where wrong answers are visible and correctable.
- Supply messy representative documents rather than a curated demo folder.
- Define acceptance on task outcomes and failure handling.
- Exercise one source update and one access revocation during the pilot.
- Require exportable traces and evaluation fixtures for handover.
What good looks like
Useful outcomes from RAG application development
- The system retrieves current, permitted and contextually relevant passages rather than merely similar chunks.
- Users can inspect citations that resolve to the exact source version and location supporting a material answer.
- Unsupported, conflicting and insufficient evidence produce distinct behaviours instead of fluent completion.
- Offline test sets and production feedback connect model changes to measurable user-task outcomes.
- Ingestion, index, model and prompt changes can be released, observed and rolled back independently.
Operating model
How to run the work
- 01
Bound the user task and answer contract
Define who asks, what decision follows, which sources are authoritative and what a useful answer contains. Classify questions the system should answer, decline or route to a person. Establish latency, citation, freshness, confidentiality and consequence requirements before choosing models or retrieval infrastructure.
- 02
Engineer the source and permission pipeline
Inventory repositories, formats, ownership, access rules, update signals and document authority. Parse structure such as headings, tables, lists and page coordinates instead of flattening everything into undifferentiated text. Carry source identity, version, validity and access metadata through every derived chunk and index.
- 03
Design retrieval as a measurable system
Create representative questions and relevance judgments. Compare lexical, dense and hybrid retrieval, query rewriting, metadata filters and reranking against those cases. Tune chunk boundaries to the source and task. Measure whether required evidence reaches the context, not only whether a similar passage appears near the top.
- 04
Generate within an explicit evidence policy
Give the model the question, permitted passages, response structure and rules for citation, uncertainty and conflict. Require material claims to map to sources. Keep deterministic business rules outside free-form generation. When evidence is absent or contradictory, return a useful limitation and next action instead of inventing continuity.
- 05
Evaluate, release and operate the product
Test retrieval, grounding, answer usefulness, citation correctness, permission enforcement, robustness and latency separately. Run adversarial and access-boundary cases. Release behind observable interfaces, sample production traces with appropriate privacy controls and route failures into source, retrieval, policy or interface improvements.
Evaluation
Questions that change the decision
- Which source or owner is authoritative when documents disagree or contain superseded positions?
- Must permissions be enforced before retrieval, after retrieval or at both stages?
- What evidence threshold permits a direct answer, a qualified answer or an escalation?
- Which parts require deterministic computation or workflow rather than language generation?
- How will source, embedding, retriever, reranker, prompt and model versions be traced per response?
Failure modes
Where teams lose control
Good generation can disguise poor retrieval, making a plausible answer look grounded when the needed passage was absent.
Document-level permissions applied after chunking can leak restricted facts through indexes, caches or citations.
Naive fixed-size chunks break tables, definitions and exceptions away from the clauses that give them meaning.
One aggregate quality score hides whether failures come from retrieval, grounding, usefulness or unsafe tool behaviour.
Automatic ingestion without authority and expiry rules can make an outdated draft outrank the approved source.
Measurement
Measure the finished job
Measure the completed workflow, including review effort and exceptions. Output volume on its own is not evidence of a better process.
- recall of required evidence at the context boundary for representative questions
- citation precision and percentage of material claims fully supported
- correct refusal or escalation rate on missing and conflicting evidence
- permission-boundary violations across adversarial access tests
- task completion, correction effort and user acceptance by question class
- end-to-end latency, cost, source freshness lag and production failure rate
Questions
Common questions
What is included in RAG application development?
It includes source ingestion, parsing, metadata, permissions, indexing, retrieval, reranking, answer policy, generation, citations, evaluation, interface integration, deployment and operational monitoring.
Does RAG prevent AI hallucinations?
No. RAG can provide current evidence and make claims inspectable, but generation can still misread, omit or exceed the evidence. Grounding tests, response policies, citations and human escalation remain necessary.
When should a company build a RAG application?
Use RAG when the task depends on changing or private knowledge that can be retrieved at request time. If the task is deterministic, has no governed sources or cannot tolerate model uncertainty, another architecture may be better.
How is enterprise RAG different from a chatbot prototype?
Enterprise RAG must maintain source authority, permissions, freshness, evaluation, observability, failure handling and ownership over time. A prototype often demonstrates only retrieval and generation on a clean sample.
Sources
Primary references
About the author
Tony Kim
Tony writes about applied AI, dependable product engineering and the systems that turn complex response work into controlled delivery.
Zeke
AI product engineering for moving a software brief into a reliable production product.
Product leaders, founders and engineering teams. Start with the workflow, constraints and evidence you already have.
See Zeke→