AI model evaluation is the systematic measurement of a model or model-enabled system against defined tasks, contexts, risks and acceptance criteria. It uses representative test cases, human or automated judgements and operational measures to support a release decision.
A public benchmark rarely represents a company workflow. A high average can hide severe failure in one language, document type or customer segment. Subjective spot checks reward eloquence and make regressions difficult to detect. Testing only the model also ignores retrieval, prompts, tools and user interaction.
Evaluate the smallest unit that informs a decision and the complete workflow that creates value. Version cases, rubrics, systems and results; report critical slices and uncertainty; and connect every score to an explicit release, rollback or improvement action.
Measurement
Evaluate components and the end-to-end system
Component tests isolate causes. Retrieval evaluation asks whether required evidence was found. Generation evaluation asks whether the answer follows the supplied evidence. Tool tests validate selection and arguments. Policy tests verify abstention, escalation and permissions.
End-to-end evaluation asks whether a real user can complete the work correctly and efficiently. A component can improve without improving the workflow, and a strong final answer can hide unsafe intermediate behavior. Both levels are needed for a defensible release.
| Layer | Question | Example measure |
|---|---|---|
| Retrieval | Was required evidence selected? | Recall and context precision |
| Generation | Does output follow the evidence? | Correctness and unsupported claims |
| Tool use | Was the right action requested? | Selection and argument validity |
| Workflow | Did the user achieve the outcome? | Accepted completion and time |
| Operations | Is it viable at scale? | Latency, cost and recovery |
Governance
A score matters only when it changes a decision
Set thresholds before reading the new result. A release may require no regression on critical failures, a minimum overall success rate, a ceiling on unsupported claims and bounded cost. Different risk tiers can use different gates and human review levels.
Document exceptions with owner, evidence, scope and expiry. A waived failure should create a mitigation or rollout constraint, not disappear. Re-evaluate after changes to model, prompt, source corpus, retrieval, tools or policy because any of them can alter behavior.
- Version cases, expected results, rubrics and systems.
- Keep a protected holdout set.
- Report confidence and sample size with scores.
- Inspect critical slices before the average.
- Link thresholds to release and rollback actions.
What good looks like
Useful outcomes from AI model evaluation
- Acceptance criteria reflect real user outcomes and failure consequences.
- The evaluation set represents routine, difficult and adversarial work.
- Model, retrieval, tool and end-to-end failures can be diagnosed separately.
- Releases are compared against a fixed baseline and critical slices.
- Production feedback adds evidence without silently rewriting the benchmark.
Operating model
How to run the work
- 01
Define the evaluation decision
State whether the evaluation selects a model, approves a release, compares prompts or monitors drift. Define the unit of success, material failure and acceptable thresholds. Include latency, cost, privacy and operational behavior when they affect the user outcome.
- 02
Build a representative case set
Sample from real task distributions with appropriate permission and de-identification. Add boundary, rare, multilingual and adversarial cases. Preserve expected evidence and evaluation notes. Keep a protected holdout set so repeated optimization does not overfit every known example.
- 03
Choose reliable measures
Use deterministic checks for schemas, citations and calculations. Use expert rubrics for nuanced correctness and completeness. Calibrate reviewers with examples, measure disagreement and blind comparisons where possible. Do not replace a material business criterion with an easy proxy.
- 04
Run gates and monitor change
Record model, prompt, retrieval, tool and code versions. Compare with baseline, inspect slice regressions and investigate failures. Block release when critical thresholds fail. In production, monitor accepted outcomes and sample reviewed cases, then add new failure patterns through a controlled process.
Evaluation
Questions that change the decision
- What deployment or product decision will the evaluation support?
- Which user and failure distributions must the case set represent?
- Can the metric measure the required quality or only a convenient proxy?
- Which slices require their own minimum rather than an overall average?
- What regression triggers a block, rollback or human-only mode?
Failure modes
Where teams lose control
Optimizing on the test set produces apparent improvement that does not generalize.
Averages conceal catastrophic failure in a small high-risk slice.
Uncalibrated judges reward style over factual correctness.
Changing the rubric and system simultaneously destroys comparability.
Production thumbs-up data can reflect user fatigue rather than correctness.
Measurement
Measure the finished job
Measure the completed workflow, including review effort and exceptions. Output volume on its own is not evidence of a better process.
- task success and critical failure rate
- performance by language, source type and risk slice
- reviewer agreement and unresolved judgement rate
- citation, schema and tool-call validity
- latency and cost per accepted outcome
- regression frequency and rollback time
Questions
Common questions
What is AI model evaluation?
It is the systematic testing of a model or AI-enabled system against representative tasks, risks and acceptance criteria to support selection, release and monitoring decisions.
What metrics should be used for LLM evaluation?
Use task-specific correctness and critical failure measures plus relevant citation, schema, tool, latency, cost and human correction metrics. No single generic metric covers every workflow.
Can another LLM grade model outputs?
It can assist when calibrated against expert judgements and monitored for bias, but it should not be the sole authority for high-impact or subtle domain correctness. Use clear rubrics and disagreement review.
When should AI evaluation run?
Run it before initial release, after material changes to models, prompts, data, retrieval, tools or policies and continuously on sampled production behavior with appropriate privacy controls.
Sources
Primary references
About the author
Tony Kim
Tony writes about applied AI, dependable product engineering and the systems that turn complex response work into controlled delivery.
Zeke
AI product engineering for moving a software brief into a reliable production product.
Product leaders, founders and engineering teams. Start with the workflow, constraints and evidence you already have.
See Zeke→