AI software engineering is the discipline of designing, building, testing, releasing and operating software whose behavior depends partly on statistical models. It extends conventional software engineering with representative evaluations, data and model provenance, uncertainty-aware interfaces, runtime safeguards and continuous measurement of behavior in the actual context of use.
A model demonstration proves that a capability can appear under selected conditions. It does not prove that a product will behave acceptably across real users, shifting data, malicious input, failed dependencies and consequential actions. Teams get into trouble when they treat a model response as the system, hide probabilistic behavior behind deterministic-looking interfaces, or release a changed model without evidence comparable to a code release.
Engineer the complete sociotechnical system, not an isolated model call. Put identity, authorization, state, validation and side-effect policy in deterministic software. Define expected behavior through representative cases and risk-specific tolerances. Every release should carry traceable evidence for code, data, model, prompt, configuration and evaluation. Production feedback must improve the test system without turning users into unconsenting experiments.
System contract
Specify behavior where ordinary software specifications stop
Conventional software can often be tested against exact outputs. An AI component may produce several acceptable answers, and a plausible answer may still be materially wrong. The specification therefore needs examples, rubrics and tolerances tied to the user’s decision. Define what must be present, what must never appear, when sources are required and when the system should say that it cannot proceed. Separate harmless variation in wording from a failure of facts, authorization or action.
Write the contract at the system boundary. A correct model classification is not a successful outcome if the application attached the wrong customer record or if a downstream worker could not contest it. Include data quality, interface, human-review and fallback behavior. NIST’s AI Risk Management Framework organizes risk work around governing, mapping, measuring and managing the system in context; that is a useful reminder that technical accuracy is only one property of a trustworthy deployment.
- Define purpose, context, users and prohibited uses.
- Classify failures by consequence, not only frequency.
- Specify support, uncertainty and escalation behavior.
- Test the complete user and downstream-system journey.
- Make acceptance criteria observable before implementation.
Architecture
Build a deterministic control plane around probabilistic capability
A model should not be the source of truth for identity, entitlement, workflow state or approval. Authenticate the user in application code, filter data before retrieval and validate every structured result against types and business invariants. A tool call is a proposal until server-side authorization accepts its actor, resource and parameters. Consequential operations need idempotency, confirmation and a durable outcome from the target system. The model’s statement that an action succeeded is never proof that it did.
Reproducibility requires more than a Git commit. Record the application version together with model identifier, provider configuration, prompt, retrieval index, data snapshot or feature pipeline, policy bundle and tool schema. Create a software bill of materials and supplier inventory appropriate to the system. NIST’s Secure Software Development Framework treats secure practice as part of the development lifecycle; AI dependencies add artifacts and behavioral change channels, but they do not remove the need for protected environments, reviewed changes and remediated vulnerabilities.
| Concern | Probabilistic component | Deterministic software |
|---|---|---|
| Interpretation | Propose meaning or classification | Apply accepted schema and policy |
| Knowledge | Compose from provided context | Enforce access, freshness and provenance |
| Action | Suggest tool and arguments | Authorize, validate, execute and confirm |
| State | Describe a possible next step | Own the canonical transition and audit |
| Failure | Express uncertainty or abstain | Select fallback, escalation and rollback |
Delivery
Treat evaluation evidence as a first-class release artifact
Unit tests remain necessary for parsers, permissions, transformations and integrations, but they cannot establish the behavioral envelope of a model-dependent system. Maintain evaluation sets representing routine traffic, difficult inputs, missing evidence, distribution edges, multiple languages and adversarial attempts. Score the properties that matter separately: factual support, completeness, action safety, format validity, calibration and user outcome. A weighted average must never cancel a critical security or safety failure.
Associate every candidate release with its evaluation report, limitations, approval and deployment plan. Compare against the current production version, not against a vague impression. Use shadow evaluation for read-only paths and a small canary when real interaction is needed. Define automated stops for severe indicators and a human owner for ambiguous evidence. Release prompts, retrieval policies and model configuration through the same controlled path as code because each can change observable behavior.
- Keep a protected holdout for honest comparison.
- Report performance by risk class, case family and language.
- Require zero tolerance for defined critical failures.
- Link every change to evaluation and approval evidence.
- Canary with bounded exposure and a verified rollback path.
Operations
Learn from production without surrendering control
Production monitoring should begin with user outcomes. Track whether the task completed, whether a person corrected or overrode the result, whether supporting material was valid and whether an external action was confirmed. Add model and infrastructure signals such as token use, retrieval quality, latency, tool errors and refusal rate. Segment by release and case family so an apparently stable average cannot hide a concentrated regression. Capture only the content needed for the declared diagnostic purpose and apply retention and access controls.
When an incident occurs, contain exposure first: disable a tool, route to manual review, revert a model or stop the feature. Preserve version and event evidence, then reproduce the behavior with a minimized case. Determine whether the defect belongs to data, prompt, model, retrieval, policy, integration, interface or operating process. Repairing the wrong layer creates brittle patches. Add an approved regression case, document the decision and verify recovery. Mature AI engineering is visible in how quickly the team can explain and safely change a failing system.
- Monitor completed outcomes rather than model responses alone.
- Segment trends so localized harm is visible.
- Minimize sensitive telemetry and control its retention.
- Contain before diagnosing an active incident.
- Turn validated failures into durable regression evidence.
What good looks like
Useful outcomes from AI software engineering
- The product has a bounded purpose, named users, prohibited uses and an explicit behavioral contract.
- Architecture separates probabilistic inference from deterministic identity, authorization, state and execution controls.
- Code, model, data, prompt, index, policy and configuration versions can be traced to every material result.
- An evaluation portfolio covers ordinary, difficult, multilingual, adversarial and high-impact cases before release.
- Release decisions use severity-weighted evidence rather than one average accuracy number.
- Failures lead to a safe state, useful user explanation, escalation or rollback instead of fabricated completion.
- Runtime telemetry detects quality, safety, latency, cost and distribution changes without collecting unnecessary content.
- Operational incidents produce minimized regression cases, control improvements and accountable follow-through.
Operating model
How to run the work
- 01
Write the system contract
Name the user outcome, context of use, inputs, outputs, non-goals and unacceptable consequences. Distinguish advice from execution and define which cases require evidence, abstention, human review or refusal. Convert those statements into observable acceptance examples.
- 02
Design the control plane and provenance
Place authentication, authorization, data scope, schemas, state transitions, budgets and side-effect rules outside the model. Version every behavioral dependency and record the lineage needed to reproduce a result without retaining unnecessary sensitive payloads.
- 03
Build an evaluation portfolio
Create representative sets for capability, safety, security, reliability and subgroup behavior. Use component tests, system tests, simulations and human review. Protect holdouts, document rubrics and establish severity-specific release thresholds.
- 04
Release as a reversible experiment
Generate a release record linking changes to evaluation evidence, security review and operational readiness. Use shadow traffic, canaries or bounded cohorts where appropriate. Monitor leading indicators, cap exposure and preserve a tested rollback or feature-disable path.
- 05
Operate and learn under control
Measure actual task outcomes, corrections, escalations, drift, incidents, latency and cost. Triage by user impact. Reproduce failures with minimized cases, repair the right layer and add regressions to the evaluation portfolio before expanding exposure.
Evaluation
Questions that change the decision
- Which part of the user outcome benefits from probabilistic inference, and which parts should remain deterministic?
- What behavior is unacceptable even when aggregate quality is high?
- Which data, model, prompt, retrieval and policy versions form one releasable system?
- What evidence is required for each risk tier and deployment stage?
- How will users recognize uncertainty, inspect support and correct a result?
- Which actions may run automatically, which need confirmation and which are prohibited?
- What telemetry is necessary to diagnose behavior without creating excessive data exposure?
- How can the team disable, roll back or degrade the capability when a dependency or model misbehaves?
Failure modes
Where teams lose control
A polished interface can imply certainty that the underlying system does not possess.
One benchmark average can conceal rare catastrophic failures or weak performance for a language or case family.
Training-serving skew can make offline results unrepresentative of production.
Model, provider or retrieval changes can alter behavior without a conventional source-code change.
Untrusted content can steer a model toward data disclosure or unsafe tool use.
Broad tool credentials can turn an interpretation error into an external side effect.
Retries can duplicate an action, increase cost or obscure the original failure.
Telemetry can collect confidential inputs and outputs beyond the need for diagnosis.
User corrections can be mistaken for consent to reuse sensitive material in training.
A dependency that fails slowly can exhaust latency and cost budgets before a fallback activates.
Measurement
Measure the finished job
Measure the completed workflow, including review effort and exceptions. Output volume on its own is not evidence of a better process.
- task success and critical-failure rate by release, case family and language
- claim support, source validity and evidence coverage where grounding is required
- appropriate abstention, escalation and refusal on insufficient or prohibited cases
- schema, invariant and authorization failures blocked before execution
- production corrections, overrides, complaints and reopened outcomes
- behavioral regressions detected before release versus after exposure
- end-to-end latency, availability and dependency failure by percentile
- cost per successful outcome rather than cost per model request
- mean time to detect, contain, reproduce and safely repair an incident
- percentage of material results reproducible from recorded version lineage
Questions
Common questions
How is AI software engineering different from machine learning engineering?
Machine learning engineering often emphasizes data pipelines, training, model evaluation and serving. AI software engineering covers the complete product: interfaces, authorization, workflow, integrations, behavioral evaluation, secure delivery, observability, human oversight and operations. Mature teams need both disciplines.
Can normal software tests validate an AI application?
They validate deterministic components but not the full behavioral envelope. Add representative evaluation sets, rubrics, adversarial cases, subgroup analysis, human review and end-to-end outcome tests. Keep critical failure thresholds separate from aggregate quality.
What must be versioned in an AI system?
Version code, model, provider settings, prompts, data or feature pipelines, retrieval indexes, policy, tool schemas and relevant infrastructure configuration. A material result should be traceable to the combined system release that produced it.
When is an AI system ready for production?
It is ready for bounded production when its purpose and limits are explicit, representative evaluations meet risk-specific thresholds, security and privacy controls are verified, users have safe failure paths, operations can observe outcomes, and the team can contain and roll back a regression.
Sources
Primary references
- Secure Software Development Framework Version 1.1 National Institute of Standards and Technology
- NIST AI Risk Management Framework Playbook National Institute of Standards and Technology
- AI Risk Management Framework Core National Institute of Standards and Technology
About the author
Tony Kim
Tony writes about applied AI, dependable product engineering and the systems that turn complex response work into controlled delivery.
Zeke
AI product engineering for moving a software brief into a reliable production product.
Product leaders, founders and engineering teams. Start with the workflow, constraints and evidence you already have.
See Zeke→