How to monitor production AI behavior, not just uptime

Trace design

Trace the product decision without logging everything

Map the actual path from request to consequence. Include preprocessing, routing, retrieved sources, prompt template, model and parameters, tool calls, policy checks, postprocessing, displayed output, user action and downstream change. Not every system uses all parts. Give each relevant component a deployable version and correlate the events so an investigator can reconstruct which behavior produced an outcome. A model version alone is insufficient when retrieval filters, system instructions or a tool schema can materially change the answer.

Apply data minimization to observability. Determine which raw inputs or outputs may be retained, for what purpose and duration, with which access and redaction. Where content cannot be logged, preserve derived attributes, references, consented samples or secure escalation capture sufficient for the approved investigation. Document what cannot be reconstructed. Separate operational logs from restricted audit evidence. Monitoring does not create permission to retain everything a user submits, and privacy protection should be designed before the incident that makes detailed evidence desirable.

Behavior quality

Combine fast proxies with slower product truth

Operational indicators arrive quickly: availability, latency, retries, fallback, retrieval count, tool failure, structured-output validity and cost. Deterministic assertions can catch prohibited formats, missing citations, schema violations or policy rules. Model-based evaluators may provide scalable signals for carefully validated dimensions. None of these automatically proves the user decision was correct. Calibrate every proxy against qualified human assessment and downstream outcomes, and monitor the proxy itself when models or policies change.

Sample production behavior for structured review. Use random sampling to estimate ordinary quality and targeted sampling for critical slices, new versions, low confidence, complaints, overrides and unusual inputs. Preserve the sampling design so targeted cases are not reported as prevalence. Use the same task contract and failure taxonomy as offline evaluation, while allowing new production failures to be classified. Where ground truth arrives later, connect it carefully and note selection: only some recommendations may be acted on, making observed outcomes a biased subset.

• Use operational signals for speed, not as proof of correctness.
• Validate automated evaluators against competent human judgment.
• Combine random prevalence samples with targeted challenge samples.
• Report severe failures separately from aggregate quality.
• Account for delay and selection in downstream outcome data.

Change detection

Treat drift as a question about the system, not a model diagnosis

Compare current input characteristics, task mix, language, sources, document formats, user populations, retrieval corpus, output attributes, intervention and quality with the approved baseline. Choose features that have a credible relationship to product behavior and can be monitored lawfully. Statistical distribution change can signal investigation, but a changed feature is not automatically harmful. Seasonal traffic may be expected, and a stable aggregate can conceal deterioration in one important slice.

Investigate across layers. Did traffic composition change? Was a prompt, model, embedding, index, tool, policy or user interface deployed? Did an upstream source change its schema or quality? Did reviewers change guidance? Is the quality metric itself stable? Reproduce affected traces when permitted and rerun a protected regression set. Compare a candidate rollback or control adjustment on the same cases. Record the causal hypothesis and evidence. Avoid declaring “model drift” when the actual failure is a stale knowledge source or changed business rule.

Control loop

Predefine containment and turn reviewed incidents into better controls

Set thresholds by consequence, not only frequency. One disclosure of restricted information may justify immediate containment while a small change in stylistic preference does not. Define who can limit traffic, increase human review, disable an action or tool, switch to a safer route, revert prompt or model, or stop the feature. Preserve configuration and relevant evidence, notify required owners and communicate uncertainty honestly. The rollback path must be tested before release and include data or index compatibility, not just application code.

After containment, verify recovery on incident cases, regression data and live samples. Identify contributing controls without reducing the cause to operator error. Update monitoring, product boundaries, tests, runbooks and user guidance as appropriate. A reviewed production case can enter a challenge or regression set only after permission, provenance, deduplication and split review. Track recurrence and whether the corrective action moved another risk. Monitoring is complete when it supports governed response and learning, not when it produces the largest dashboard.

• Set action thresholds according to failure consequence.
• Name authority for scope limits, review increases and rollback.
• Test rollback across models, prompts, data and tools.
• Verify recovery on both incident and representative cases.
• Govern the addition of production evidence to evaluation.