Replacing the language model behind an AI feature is not a simple configuration change. This LLM model migration checklist gives engineering and product teams a repeatable way to compare API behavior, prompts, structured outputs, tools, cost, latency, safety, and rollback readiness before switching providers or model versions.
Overview
A model migration succeeds when the new model preserves the user-visible behavior that matters while improving a measurable constraint such as cost, latency, availability, capability, or maintainability. A model can appear stronger in a general benchmark and still be a poor replacement for a production feature if it formats responses differently, follows tools inconsistently, consumes more context, or triggers different safety handling.
Treat the migration as a compatibility and evaluation project rather than a prompt-editing exercise. Start by documenting the current system: model identifier, provider, SDK version, request parameters, system prompt, user prompt templates, retrieval settings, tool definitions, output schema, retry logic, logging, and escalation paths. Then define acceptance criteria in terms of outcomes. For example, a support assistant may need to cite retrieved passages, return valid JSON, call a ticketing tool only with confirmation, and preserve a calm tone. These requirements are more useful than a general instruction to “match the old model.”
Keep the old implementation available until the new path has passed offline evaluation and a controlled production rollout. A rollback is only practical if the previous model, prompt version, configuration, and dependencies can be restored independently.
How to estimate migration impact
Use a simple decision model before committing engineering time. Score the candidate model against the dimensions that affect your feature, then validate the highest-risk dimensions with real application traffic or a representative test set.
| Dimension | What to compare | Evidence to collect |
|---|---|---|
| API compatibility | Endpoints, authentication, SDK methods, parameter names, streaming, retries, and error formats | Working integration test and documented differences |
| Prompt behavior | Instruction priority, verbosity, refusal patterns, formatting, and sensitivity to examples | Paired outputs from a fixed evaluation set |
| Context and tokenization | Input limits, output limits, token counting, truncation, and multilingual behavior | Token measurements and boundary tests |
| Structured output | JSON validity, schema adherence, enum handling, and missing-field behavior | Parser results and field-level validation |
| Tools and agents | Tool selection, argument accuracy, parallel calls, confirmation steps, and retries | Recorded tool-call traces and failure cases |
| Operations | Latency, throughput, rate limits, availability, observability, and support procedures | Load tests, traces, alerts, and runbook updates |
| Economics | Input and output usage, caching, retries, routing, and infrastructure overhead | Cost per request and cost per successful task |
| Safety | Input and output filters, prompt-injection exposure, sensitive data handling, and refusal consistency | Safety test results and human review |
For cost estimation, use your own measured usage rather than a headline rate. A basic monthly estimate is:
Monthly model cost = requests × (average input tokens × input unit cost + average output tokens × output unit cost) ÷ billing unit
Extend the estimate with retry volume, tool-call turns, cached-token treatment where applicable, embedding or retrieval costs, and the cost of failed requests. Also calculate cost per successful task. A cheaper request is not necessarily cheaper if it produces more invalid outputs, extra retries, or human intervention.
For quality, create a weighted score instead of relying on one benchmark:
Migration score = quality weight × quality result + reliability weight × reliability result + cost weight × cost result + latency weight × latency result
Normalize each result to the same scale, record the weights, and keep the raw measurements beside the score. The score supports a decision; it does not replace review of severe failures.
Inputs and assumptions
Complete this compatibility worksheet for both the current and candidate models. Mark each item as compatible, requires adaptation, unverified, or blocked.
- Request contract: endpoint, authentication, model name, message roles, streaming mode, temperature or sampling controls, maximum output settings, timeouts, and retry behavior.
- Prompt contract: system instructions, examples, delimiters, retrieved context, locale requirements, and assumptions about instruction hierarchy.
- Token contract: how tokens are counted, where truncation occurs, minimum context retained for citations, and expected output headroom. For background, see LLM Context Windows Explained.
- Output contract: plain text, JSON, schema-constrained output, Markdown, citations, or downstream parser expectations. Test malformed, incomplete, and extra-field responses.
- Tool contract: names, descriptions, argument schemas, required confirmation, authorization checks, parallel-call behavior, and handling of unavailable tools.
- Quality assumptions: define what counts as correct, acceptable, unsafe, too slow, or unnecessarily verbose. Include difficult and ordinary examples, not only successful cases.
- Operational assumptions: expected request volume, concurrency, peak periods, regional requirements, logging limits, and the fallback path if the provider or model is unavailable.
Do not assume that identical prompts produce equivalent behavior. A migration may require separating instructions that were previously combined, making output schemas more explicit, changing delimiters, or moving validation into application code. Keep prompt changes versioned so a quality change can be attributed to the model, the prompt, or both.
Build a regression set from production samples that are privacy-reviewed and, where necessary, redacted. Include success cases, edge cases, adversarial inputs, long-context cases, multilingual examples, tool failures, and cases that previously caused incidents. The LLM Prompt Testing Framework provides a useful structure for turning these cases into repeatable tests.
Worked examples
Example 1: A structured support classifier
Suppose an application classifies incoming requests into a fixed set of categories and returns a JSON object containing a category, urgency, confidence explanation, and escalation flag. The migration test should not stop at whether the response “looks right.” Parse every response, validate the schema, check allowed enum values, and verify that missing or ambiguous information leads to the intended escalation behavior.
Compare the current and candidate models on the same redacted evaluation set. Record classification accuracy, invalid-JSON rate, average input and output tokens, p50 and p95 latency, retry rate, and cost per successful classification. If the candidate has better classification results but fails schema validation more often, test structured output controls, a stricter prompt, or application-side repair before approving the switch.
Example 2: A retrieval and tool workflow
Consider an assistant that retrieves policy passages and can create a service ticket. Test at least four paths: an answer supported by retrieved content, a question with no supporting passage, a request that requires a tool, and a prompt-injection attempt inside retrieved text. Verify that the model distinguishes instructions from data, cites or quotes the permitted evidence, requests confirmation before consequential actions, and refuses to invent a tool result.
Use trace-level evaluation: inspect the retrieved context, final prompt, tool choice, arguments, authorization decision, and final answer. The Prompt Injection Defense checklist can help organize these tests. A successful final answer alone can conceal an unsafe intermediate action.
Example 3: A staged production rollout
Deploy the candidate behind a feature flag. Begin with offline evaluation, then shadow traffic where the candidate response is generated but not shown to users. Next, expose a small, monitored cohort and compare task success, latency, cost, user corrections, safety events, and support tickets. Set explicit stop conditions before rollout, such as a schema failure threshold, a severe safety regression, an unacceptable latency increase, or a cost increase beyond the approved range.
Keep routing reversible at the application layer. Store model and prompt versions with each request so an incident can be tied to a deployment. Review the LLM Application Monitoring Checklist before opening the feature to all traffic.
When to recalculate
Revisit this checklist whenever the provider changes pricing, rate limits, API parameters, context behavior, safety controls, or model lifecycle requirements. Recalculate cost when traffic mix, average prompt length, output length, caching, retry rate, or tool-call frequency changes. Re-run quality and latency tests when the model identifier, SDK, system prompt, retrieval configuration, tokenizer behavior, or output schema changes.
A quarterly review is a useful minimum for actively maintained applications, but incident reviews and major release notes should trigger an earlier check. Compare the proposed change with the provider’s current documentation rather than copying assumptions from an older integration. The guide to reading AI model release notes can help identify deprecated parameters and behavior changes.
Before sign-off, record:
- Current model and candidate model, including provider and API version.
- Prompt, SDK, schema, retrieval, and tool-definition versions.
- Evaluation set, scoring method, sample size, and known limitations.
- Measured quality, validity, latency, usage, cost, and safety results.
- Approved rollout stages, monitoring owner, stop conditions, and rollback owner.
- Review date and the events that require an earlier recalculation.
Finally, obtain separate engineering and product approval. Engineering should confirm compatibility, observability, security, and rollback. Product should confirm that the behavior remains acceptable for users and business workflows. A migration is ready when both groups can explain not only why the candidate is better, but also how the team will detect and reverse a regression.
