LLM application development designs and engineers a software product in which a language model interprets, retrieves, transforms or generates information inside explicit business, security and operational boundaries. The product includes the surrounding data, workflows, interfaces, evaluations, authorization and runtime controls, not only the selected model or prompt.
A prototype can look impressive because a person selects clean examples, repairs context and tolerates occasional mistakes. Production receives ambiguous requests, malicious documents, missing evidence, conflicting instructions, long inputs, concurrent users, model changes and unavailable dependencies. If the team cannot state what correct behavior means, replay a failure or restrict a tool call, more prompt tuning will not create a reliable application.
Start with a bounded user decision and an observable acceptance contract. Design deterministic software around probabilistic inference. Models may propose classifications, answers or plans; application code should enforce identity, authorization, data scope, schemas, state transitions, side-effect limits and release policy. Evaluate the complete system on representative cases before launch and continuously after changes. Model flexibility is valuable only when failure is visible, containable and recoverable.
Product scope
Define the decision before designing the model interaction
“Build a chatbot” is not a product contract. A useful scope states who is trying to accomplish what, which material they can use, what the system returns and what happens next. Summarizing a policy for exploration is different from deciding whether a claim is compliant. Drafting an email is different from sending it. The same model capability can therefore require radically different controls depending on the downstream consequence.
Write acceptance examples in product language. For an evidence assistant, specify that each material claim needs a source the user is allowed to read and that contradictory sources trigger a visible conflict. For an extraction product, define the canonical schema, required provenance and acceptable unknown state. Include cases the product must refuse. This contract keeps model choice subordinate to user value and gives design, engineering, security and operations a common object to test.
- Name the user, decision, input, output and next system.
- Separate generated suggestion from authorized action.
- Specify unknown, conflict and escalation behavior.
- Define irreversible and high-impact operations as separate gates.
- Make every launch claim traceable to an evaluation result.
Quality system
An evaluation corpus is part of the product, not a launch ceremony
Model benchmarks say little about a specific workflow. Build an evaluation corpus from the distributions the application will face. Include ordinary requests so optimization does not only serve spectacular edge cases, and include difficult examples that expose document quality, ambiguity, long context, multilingual terminology, missing sources and unsafe instructions. Label the properties that matter rather than assigning one vague quality score.
Evaluate components and the full journey. Retrieval metrics can reveal whether the right evidence was available; answer scoring can assess support and completeness; tool simulations can test authorization and parameter safety; end-to-end review can measure whether a user completes the task. Store model, prompt, retrieval, index, tool and policy versions with each run. When a production incident appears, reproduce it, fix the control and add a safely minimized regression case.
| Layer | Question | Representative measure |
|---|---|---|
| Retrieval | Was usable evidence found? | Recall and access-correct relevance |
| Generation | Is the output supported and complete? | Claim support and rubric result |
| Structure | Can software consume it safely? | Schema and invariant pass rate |
| Action | Was the permitted operation proposed? | Authorization and argument validity |
| Outcome | Did the user finish the job? | Task success, correction and escalation |
Security
Treat model instructions as influence, not enforcement
OWASP identifies prompt injection as a core LLM-application risk and notes that retrieval or fine-tuning does not remove it. Any text the model reads can contain competing instructions, including a document retrieved from a trusted repository. Delimiters and prompts may reduce accidental confusion, but they cannot serve as authorization. The application must assume the model can be persuaded to propose something outside policy.
Put enforcement in deterministic layers. Filter retrieval by the authenticated user before content reaches the model. Validate output before rendering or execution. Bind tools to narrow server-side identities and recheck authorization for every call. Do not store credentials in prompts. Separate a proposed action from its execution and require human confirmation for material effects. Test both direct attacks and indirect instructions embedded in documents, images and tool results.
- Treat retrieved and uploaded content as untrusted.
- Resolve identity and authorization outside the model.
- Use least-privilege, task-specific credentials for tools.
- Validate arguments and results at every integration boundary.
- Preserve a human gate for consequential external actions.
Operations
Design for change, partial failure and replay
An LLM application depends on model endpoints, indexes, storage, policy services and external tools. Each can slow, fail or return a changed shape. Define timeouts, bounded retries, idempotency and fallbacks by operation. A read-only summary can often be retried; a payment or message cannot be repeated blindly. Return a clear pending or failed state rather than letting the model narrate success that the downstream system did not confirm.
Observability should reconstruct behavior without becoming an uncontrolled data lake. Capture version identifiers, timing, token use, retrieval references, validation results, tool status and user outcome. Minimize or redact sensitive payloads according to the legitimate debugging need. Use canary releases and compare quality, latency and cost before broad rollout. A reliable team can answer which version produced a result, why it received those sources and whether the external action actually occurred.
- Version prompts, models, indexes, schemas, tools and policy.
- Make side-effecting calls idempotent and separately confirmed.
- Distinguish model completion from business completion.
- Set latency and cost budgets per successful user outcome.
- Keep rollback and replay available for every release.
What good looks like
Useful outcomes from LLM application development
- The first release solves a named job for a named user with explicit non-goals and escalation paths.
- A versioned evaluation set represents common, difficult, ambiguous, multilingual and adversarial cases.
- Retrieved context carries source identity, access scope, freshness and citation back to the user.
- Structured outputs are validated before they reach databases, interfaces or downstream software.
- Tools execute under least privilege with server-side authorization, bounded parameters and human approval for material actions.
- Traces connect input, context, prompt, model, output, validation, tool result and user outcome without leaking unnecessary sensitive data.
- The team can change model, prompt or retrieval logic and compare quality, latency, cost and risk before rollout.
- Users receive a useful abstention or handoff when evidence, confidence or a dependency is insufficient.
Operating model
How to run the work
- 01
Define the job and acceptance contract
Describe the user, trigger, input, expected output, downstream decision and cost of error. List non-goals, prohibited actions and cases that require a person. Convert desired behavior into examples and measurable acceptance criteria before choosing an architecture.
- 02
Create the evaluation corpus first
Gather safely representative cases across normal traffic, edge cases, poor documents, multiple languages, unavailable evidence and adversarial instructions. Label expected facts, citations, formats, decisions and abstentions. Version the dataset and keep a protected holdout for honest comparison.
- 03
Design context and trust boundaries
Decide what comes from the user, system, retrieval, memory and tools. Treat external text as untrusted data rather than authority. Enforce access before retrieval, minimize context, mark provenance and separate instructions from content. Define retention and deletion for prompts, derived data and traces.
- 04
Engineer outputs and actions
Use constrained schemas for machine-consumed output and validate types, ranges, references and business invariants. Resolve authorization in application code. Give tools narrow credentials, idempotency and rate limits. Require explicit approval before sending, purchasing, deleting, signing or changing high-impact records.
- 05
Operate through measured releases
Run offline evaluation, security tests, load tests and human review before a canary release. Observe quality, latency, cost, abstention, overrides and incidents by version. Retain replayable cases, roll back regressions and add production failures to the evaluation set after safe review.
Evaluation
Questions that change the decision
- Is an LLM necessary for the variable part of the task, or will rules, search or conventional machine learning be more reliable?
- What is the smallest user outcome that can be evaluated independently?
- Which sources may enter context and how are authorization, freshness and provenance enforced?
- Which outputs are advisory, which update state and which require accountable approval?
- Should the system retrieve, fine-tune, use tools or combine these patterns?
- What latency, availability, cost and data-handling constraints determine model and deployment choices?
- How will a user inspect, correct, contest or reverse a result?
- What evidence is required to promote a new model, prompt, index or tool version?
Failure modes
Where teams lose control
A fluent answer can be factually wrong, unsupported or misaligned with the user’s actual decision.
Direct and indirect prompt injection can alter behavior through user input, retrieved files, pages or tool results.
Retrieval without access filtering can disclose material the requesting user could not open at source.
Placing secrets or authorization logic in a system prompt confuses instruction with a security boundary.
Unvalidated model output can become code, queries, markup or actions with conventional injection impact.
An agent with broad tools can amplify a small interpretation error into an irreversible external action.
Model or provider updates can change behavior without an application-code change.
Observability can itself collect sensitive prompts, documents and outputs beyond a legitimate need.
Average accuracy can hide a severe failure concentrated in one language, user group or high-impact case.
Unbounded context and retries can produce unstable latency and cost under real load.
Measurement
Measure the finished job
Measure the completed workflow, including review effort and exceptions. Output volume on its own is not evidence of a better process.
- task success on the versioned evaluation set and protected holdout
- claim-level support and citation correctness where evidence is required
- critical error rate by use case, language and risk category
- appropriate abstention and escalation on unsupported or ambiguous cases
- structured-output validation failures and repaired responses
- unauthorized or over-broad tool calls prevented before execution
- user corrections, overrides and reopened outcomes
- end-to-end latency by percentile and successful dependency path
- model, retrieval, storage and observability cost per completed outcome
- regressions detected before rollout and mean time to safe rollback
Questions
Common questions
What is included in LLM application development?
It includes product scope, data and context design, model integration, retrieval, evaluation, interfaces, structured outputs, security, tool authorization, observability, deployment and operations. The prompt is one versioned component inside that system.
Should we use RAG or fine-tuning?
Use retrieval when the application needs current, attributable knowledge and access-aware sources. Fine-tuning can shape behavior or specialize recurring patterns. Neither automatically solves authorization, prompt injection, factual support or evaluation, and some tasks need neither.
How do you test an LLM application?
Build a representative, versioned corpus with expected facts, sources, formats, actions and abstentions. Test retrieval, generation, schemas, tool calls and user outcomes separately and end to end, including multilingual, ambiguous and adversarial cases.
Can an LLM application take autonomous actions?
It can propose and perform bounded actions, but authorization belongs in application code. Use least-privilege tools, validated parameters, idempotency, audit and explicit human approval for consequential, irreversible or externally visible actions.
Sources
Primary references
- AI Risk Management Framework resources National Institute of Standards and Technology
- LLM01:2025 Prompt Injection OWASP GenAI Security Project
- LLM Prompt Injection Prevention Cheat Sheet OWASP Foundation
About the author
Tony Kim
Tony writes about applied AI, dependable product engineering and the systems that turn complex response work into controlled delivery.
Zeke
AI product engineering for moving a software brief into a reliable production product.
Product leaders, founders and engineering teams. Start with the workflow, constraints and evidence you already have.
See Zeke→