How to build an AI evaluation dataset for a real product

Evaluation contract

Define what one test case means in the product

Write the task contract before collecting examples. Define the input unit, all context visible to the system, expected output or action, downstream user, decision consequence and reference sources. State whether the system may abstain, ask a question or return several valid options. For retrieval, a case might pair a query with an authorized corpus and relevance judgments. For document extraction, it might include the source file, schema, field-level references and acceptable normalization. For a drafting product, one “perfect answer” is usually weaker than a rubric covering factuality, completeness, grounding, instruction adherence and harmful commitments.

Define the failure taxonomy and acceptance logic. Separate omission, incorrect fact, unsupported claim, wrong scope, unsafe action, privacy exposure, instruction failure and latency. Name failures that require zero tolerance in the evaluated release and those that use statistical thresholds. Record who owns the product policy and who can adjudicate edge cases. The dataset does not create policy; it encodes a particular version of it. When reviewers cannot decide because the product behavior is undefined, resolve the product rule rather than hiding disagreement inside a label.

Sampling

Represent production and preserve critical cases without confusing the two

Describe the intended population: time period, channels, document sources, formats, languages, user groups, jurisdictions, complexity, quality and known seasonality. Draw a representative sample using those dimensions and retain selection weights or counts. If production does not yet exist, use a clearly labelled proxy and document how it differs. Avoid selecting only cases already solved by the team or documents that parse cleanly. Include negative and out-of-scope inputs so the system is evaluated on whether it should act, not only how well it acts on ideal work.

Create named challenge slices for rare but important behavior: very long inputs, missing evidence, conflicting instructions, minority languages, unusual layouts, adversarial content, sensitive data and high-consequence decisions. Deliberate oversampling makes those failures observable but does not estimate their prevalence, so report challenge results separately. For every item, preserve source, collection date, permission, purpose, transformations and exclusions. Limit access and retention according to the data, and remove unnecessary identifiers while retaining only attributes required for meaningful evaluation.

• Characterize the population before drawing examples.
• Label proxies and document how they differ from production.
• Include negative and out-of-scope behavior.
• Report deliberately oversampled challenge slices separately.
• Preserve permission, provenance, transformation and retention.

Reference judgments

Treat annotation as a measurement process, not clerical labeling

Write instructions with definitions, decision rules, examples and escalation cases. Distinguish objective source facts from product policy and subjective preference. For open-ended outputs, use structured rubrics and source-linked rationale rather than asking reviewers to rank style. Pilot with qualified annotators from the actual domain. Measure agreement by label and slice, inspect confusion and revise the instructions. High agreement on easy cases does not prove consistent judgment on the failures that matter.

Capture individual judgments before adjudication where independence matters. Store the label, rationale, cited evidence, reviewer role, confidence and instruction version. Disagreement can reveal missing context, ambiguous source material, different expertise or unresolved policy. A named adjudicator should resolve according to documented authority and may mark the item indeterminate or split the case instead of forcing a majority label. Model-based judges can help scale selected comparisons, but validate them against competent human judgment and avoid using the same model family as unquestioned authority.

Lifecycle

Protect evaluation integrity while keeping the dataset alive

Separate data by purpose. Development examples are visible for prompt, model and workflow iteration. A protected acceptance set supports release decisions and receives restricted access. A stable regression set tracks known behavior across versions. A rolling set samples recent production to detect drift. Use grouping so near-duplicate documents, templates, customers or time-linked cases do not cross splits and create leakage. Search both exact and semantic overlap. Record every evaluation access because repeatedly viewing failures during tuning converts protected data into development knowledge.

Version the item set, task contract, annotation policy and slices together. Report metrics with the exact version, sample counts, uncertainty and limitations. Do not remove difficult failures merely because a policy changed; record the reason and preserve history. Add production failures through a reviewed intake that checks rights, deduplication, representativeness and taxonomy. Periodically refresh the rolling set and decide whether the protected benchmark still represents the product. A higher score on an obsolete dataset is not improvement in the operating system.

• Separate development, acceptance, regression and rolling purposes.
• Group related inputs before splitting and check semantic leakage.
• Restrict and log access to protected evaluation.
• Version cases, policy, annotations and slice definitions together.
• Use reviewed production evidence to evolve the portfolio.