An AI model migration is a controlled product change that replaces a model, provider, endpoint, deployment mode or model family while preserving required user outcomes, security controls, integrations and operational reliability.
A model rarely behaves like a drop-in library. Applications absorb provider-specific message formats, tool calling, safety behavior, context limits, token accounting, streaming events, rate limits and error semantics. Prompts and evaluations may be tuned to the current model. A simple endpoint switch can therefore change answer meaning, refusal behavior, tool execution, cost and support load even when both models claim similar capability.
Migrate the system behavior, not just the API. First state why the change matters and which existing behavior is intentional. Expose hidden coupling, build a provider-neutral product contract where it creates leverage, and evaluate candidate systems on representative work. Shift bounded traffic only when operational controls and rollback are real. Portability is the ability to make an informed change, not the fiction that every model is interchangeable.
System contract
Define the behavior that must survive the model change
Exact output equivalence is usually the wrong objective for a generative system. Define invariant properties instead: the task completes, material claims have valid support, prohibited data is not exposed, tools receive authorized arguments, structured output validates, refusals occur in specified conditions and users can recover from uncertainty. Some tasks also need tone, language or formatting constraints, but these should remain tied to a user outcome.
Create a current-state baseline before tuning the replacement. Without it, the team cannot distinguish improvement from changed taste or remember which old defects were already accepted. Compare variants on the same protected cases and report confidence and sample size. Preserve examples of disagreement for expert review. The migration decision should name both the gain being pursued and any known tradeoff being accepted.
| Layer | Evidence | Typical gate |
|---|---|---|
| Behavior | Task outcome, support, refusal and correction | No critical regression |
| Integration | Tool calls, schemas, streaming and error handling | Contract suite passes |
| Operations | Latency, load, quotas, availability and fallback | Capacity target met |
| Risk | Data path, access, supplier terms and monitoring | Owner approval |
| Economics | Usage, engineering, review and exception cost | Business case holds |
Portability
Abstract contracts deliberately, not universally
A narrow adapter can isolate transport, authentication, request construction, response events, retry semantics and usage reporting. Tool and structured-output interfaces can expose application contracts independent of a provider. This separation improves testing and makes future comparison cheaper. It also creates one place to enforce timeouts, tracing, redaction and fallback policy.
Do not hide every model difference. Context handling, multimodal input, constrained decoding, caching and reasoning controls may create real product value. Represent such features as explicit capabilities and let the product choose them knowingly. A universal chat interface that silently drops unsupported behavior creates the appearance of portability while moving risk into production. Document graceful degradation and reject configurations that cannot satisfy a required capability.
- Separate application intent from provider request format.
- Expose capability differences instead of silently flattening them.
- Keep prompt and adapter versions in every production trace.
- Test error, timeout, quota and partial-stream behavior.
- Provide an export path for evaluation and operating evidence.
Cutover
Make the transition observable and reversible
Offline evaluation cannot reproduce every user interaction, load pattern or upstream data condition. Shadow execution can reveal integration and performance differences when duplicate processing is legally and economically acceptable. A canary then exposes a bounded production population to the candidate. Keep assignment stable enough to compare outcomes and identify which system produced every result.
Rollback is a designed state transition, not an emergency wish. Verify compatibility of prompts, caches, conversation state, retrieval indexes, tool side effects and downstream records. Define the last safe point, authority and communication path. If the candidate can perform irreversible actions, restrict or confirm them during early stages. Complete the program by deleting unused access and monitoring, not by leaving two providers active indefinitely.
- Use stages with explicit entry and exit evidence.
- Tag every trace and downstream action with the model release.
- Protect rollback from incompatible state changes.
- Stop expansion automatically on defined severe indicators.
- Close dual processing and supplier access after stabilization.
What good looks like
Useful outcomes from AI model migration service
- The migration has explicit business drivers, constraints, acceptance thresholds and decision owners.
- Provider and model coupling is inventoried across code, prompts, retrieval, tools, data and operations.
- Candidates are compared on product outcomes, critical failures, latency, capacity and total cost.
- Traffic moves through observable stages with compatibility controls and a tested rollback path.
- The old model and provider dependencies are retired only after stability and data obligations are verified.
Operating model
How to run the work
- 01
Frame the migration decision
Document the trigger: capability gap, economics, latency, availability, contractual change, deployment control or portfolio simplification. Define the affected users, tasks, jurisdictions and data classes. Establish must-preserve behavior, intended improvements, prohibited regressions, budget and deadline. Record whether the target is preselected or whether candidate comparison remains open.
- 02
Discover behavioral and technical coupling
Trace every invocation path, model configuration, prompt, retrieval step, tool schema, structured output, cache, moderation control, retry, fallback, rate limit and telemetry field. Review supplier contracts, regional processing, retention, capacity and deprecation conditions. Mark assumptions embedded in tests and user interfaces. Coupling that remains invisible will surface during cutover.
- 03
Build the migration baseline
Freeze representative cases and production-derived failures under appropriate data controls. Measure the current system by task family, risk class, language, user role and load. Separate desired behavior from known defects so the target is not required to copy every weakness. Capture cost, latency, throughput, correction effort, escalations and critical failure rates beside qualitative output.
- 04
Adapt and compare candidate systems
Implement the smallest useful compatibility layer for invocation, tool contracts, structured outputs, errors and observability. Tune each candidate within a fair, documented budget rather than copying a provider-specific prompt unchanged. Run blinded comparisons, adversarial cases and operational load tests. Investigate disagreement and attribute failures before choosing the target configuration.
- 05
Cut over, stabilize and retire
Deploy in shadow or replay where valid, then use a canary with bounded users, traffic or tasks. Monitor product outcomes and infrastructure signals by model version. Exercise rollback before widening exposure. After the stability window, remove obsolete credentials, endpoints, model-specific code and commitments, and preserve the migration evidence for future changes.
Evaluation
Questions that change the decision
- Is migration justified by measured product and operating outcomes rather than model fashion?
- Which current behaviors are requirements and which are defects or accidental provider coupling?
- Where does abstraction reduce future change cost, and where would it hide valuable model capability?
- What evidence authorizes each traffic stage and what signal triggers immediate rollback?
- When may the old provider access, data and commercial commitment be retired?
Failure modes
Where teams lose control
Copying prompts unchanged can create an unfair comparison and poor behavior on the target model.
An abstraction layer can erase useful provider features or become a lowest-common-denominator architecture.
Average quality can improve while a critical language, tool or refusal case regresses.
Dual running can duplicate sensitive data processing and cost if its boundary and end date are vague.
Rollback may fail after schemas, retrieval content or downstream state have changed incompatibly.
Measurement
Measure the finished job
Measure the completed workflow, including review effort and exceptions. Output volume on its own is not evidence of a better process.
- task success, correction effort and escalation by scenario and model variant
- critical regression count by language, user role, tool and risk class
- end-to-end latency, throughput, availability and rate-limit events
- total cost per completed business task rather than token price alone
- traffic moved through approved stages and rollback exercises completed
- provider-specific dependencies, credentials and commitments retired
Questions
Common questions
Can an application switch AI models without code changes?
Sometimes a compatible endpoint reduces transport changes, but behavior, tool calling, structured output, safety, context, errors and operating limits can still differ. Production migration requires comparative evaluation and controlled cutover even when the API shape is similar.
How do you compare an old and new LLM fairly?
Use the same representative and protected cases, give each model a documented and reasonable adaptation budget, compare complete product outcomes and review severe disagreements. Include latency, capacity, correction effort and total operating cost beside output quality.
Does a model gateway make an AI product portable?
It can isolate authentication and common invocation mechanics. Portability also depends on prompts, tools, retrieval, safety behavior, state, evaluation, data terms and operations. Required provider-specific capabilities should remain explicit rather than silently discarded.
When should the old model be turned off?
After the target passes defined offline and production gates, rollback has been exercised, the stability window closes and no required workflow depends on the old configuration. Then retire access, code, data obligations and commercial commitments deliberately.
Sources
Primary references
- AI RMF Core National Institute of Standards and Technology
- Generative AI Profile National Institute of Standards and Technology
- Secure Software Development Framework National Institute of Standards and Technology
About the author
Tony Kim
Tony writes about applied AI, dependable product engineering and the systems that turn complex response work into controlled delivery.
Zeke
AI product engineering for moving a software brief into a reliable production product.
Product leaders, founders and engineering teams. Start with the workflow, constraints and evidence you already have.
See Zeke→