LLM Provider Migration Guide: Move an AI Application Between Model APIs Without Breaking Outputs

Overview

LLM portability depends on separating your application from the details of any one provider. A migration can affect request construction, message roles, token accounting, streaming events, tool schemas, JSON handling, safety controls, embeddings, error behavior, and operational limits. Even when two APIs support similar capabilities, they may interpret the same prompt differently.

The safest approach is to treat the model provider as an interchangeable infrastructure component behind an adapter. Your application should send a provider-neutral request to that adapter, and the adapter should translate it into the target provider's format. Keep business logic, validation, logging, evaluation, and fallback decisions outside the provider-specific layer.

Before changing production traffic, capture representative inputs and expected outcomes from the current system. Include ordinary requests, long inputs, malformed inputs, tool-use requests, refusals, empty results, multilingual content if relevant, and cases that previously caused timeouts or validation failures. These become your migration regression set.

For related implementation details, see the LLM API error-handling guide, the guide to structured outputs and JSON Schema validation, and the prompt evaluation framework.

Provider compatibility at a glance

Checklist by scenario

Scenario 1: Migrating a basic chat or completion workflow

1. Inventory the current contract. Record the exact model identifier, message sequence, default parameters, maximum output setting, timeout, retry policy, and response fields your code reads.
2. Normalize your internal request. Use fields such as system_instruction, messages, max_output_tokens, temperature, and metadata internally, then translate them in the adapter.
3. Translate prompts deliberately. Do not assume that copying a system prompt will preserve behavior. Recheck instructions about format, uncertainty, refusal boundaries, and priority. If your prompt relies on examples, test them again; the few-shot prompting examples guide provides a useful design reference.
4. Match output handling. Decide whether your application needs plain text, parsed JSON, citations, classifications, or another typed result. Validate the result at the application boundary rather than trusting the provider response.
5. Compare representative outputs. Evaluate correctness, required fields, tone, length, refusal behavior, latency, and cost according to your own acceptance criteria.

Scenario 2: Moving an OpenAI-style integration to another provider

An OpenAI-compatible endpoint can reduce code changes, but compatibility should be treated as a convenience rather than a complete contract. Confirm whether the target supports the same message roles, content parts, tool schema, streaming events, response formats, and error fields.

Use a small adapter instead of scattering compatibility conditions throughout the application:

interface LLMAdapter {
  generate(request: NormalizedRequest): Promise<NormalizedResponse>;
  stream(request: NormalizedRequest): AsyncIterable<NormalizedEvent>;
  embed(input: string[]): Promise<EmbeddingResponse>;
}

type NormalizedRequest = {
  messages: Message[];
  tools?: ToolDefinition[];
  outputSchema?: unknown;
  maxOutputTokens?: number;
  temperature?: number;
};

The adapter should normalize provider errors into categories such as authentication, invalid request, rate limit, timeout, overloaded service, safety refusal, and response validation failure. That lets your retry and user-facing logic remain stable. For failure diagnosis, consult the LLM API error-handling checklist.

Scenario 3: Migrating tool calling or an AI agent

1. List every tool name, description, parameter, required field, enum, and validation rule.
2. Confirm how the target represents an assistant tool call and how your application returns the tool result.
3. Check whether multiple tool calls can be issued in one turn and whether your orchestration layer supports them.
4. Set an explicit maximum number of agent turns and a deadline for the entire run.
5. Test invalid arguments, unavailable tools, duplicate calls, partial tool results, and prompt-injection content returned by external sources.

Do not allow a provider migration to weaken tool authorization. The model may suggest a tool call, but your application should still enforce identity, permissions, argument validation, and business rules. Use the prompt injection defense checklist when tools access private data or take actions.

Scenario 4: Migrating a RAG application

Separate the generation migration from the retrieval migration whenever possible. Keep the existing index while comparing generation models, then evaluate a new embedding model and index as a separate change. Embedding vectors produced by one model are generally not something to mix casually with vectors from another model; verify dimensions, distance configuration, normalization, and index requirements before reusing infrastructure.

Test retrieval and generation independently. Measure whether the expected source appears in the retrieved context, whether the answer is supported by that context, and whether the model follows citation or “not enough information” rules. The RAG tutorial can serve as a broader evaluation reference.

Scenario 5: Migrating structured outputs

Define a provider-neutral schema and validate every response with a real parser. A prompt that says “return JSON” is not equivalent to a guaranteed schema-constrained response. Compare required fields, null handling, enum behavior, nested objects, escaping, refusal responses, and truncated output.

Keep repair logic bounded. If parsing fails, record the raw response securely, classify the failure, and use a limited retry or repair path rather than repeatedly sending the same request. Never silently convert invalid data into a successful business record.

What to double-check before production

• Prompt semantics: Recheck instruction priority, delimiters, examples, output constraints, and any provider-specific prompt syntax.
• Context behavior: Confirm practical input and output limits, truncation behavior, and how your code handles overlong requests. Do not rely only on a published maximum; test the complete request your application sends.
• Token and usage reporting: Verify whether usage is returned immediately, only at the end of a stream, or not in the same response shape. Update monitoring and budget alerts accordingly.
• Streaming: Test empty chunks, content boundaries, tool-call fragments, disconnects, retries, and whether the final event duplicates previously displayed text.
• Safety behavior: Compare refusals, filtered content, blocked tool arguments, and safety metadata. Treat differences as application behavior changes that need product review.
• Privacy and logging: Review what request and response data is logged, where it is stored, and whether failed requests expose sensitive prompts or retrieved documents.
• Concurrency and limits: Load-test realistic bursts, queue behavior, retry amplification, and long-running agent workflows. A retry policy should include backoff and a clear upper bound.
• Fallbacks: Decide whether a fallback provider receives the same prompt, a simplified request, or no request at all. Document how structured output and tool behavior are handled during fallback.

For coding teams, create migration tasks from observed failures rather than assumptions. The AI coding assistant prompt templates can help structure planning, debugging, refactoring, and test-generation work around the change.

Common mistakes

• Changing provider and prompt at the same time: This makes regressions difficult to attribute. Freeze the prompt first when possible, then make intentional prompt adjustments after baseline testing.
• Testing only happy-path chat: A migration that passes one conversational example may still fail on streaming, tools, schemas, long context, or provider errors.
• Assuming similar model names mean similar behavior: Treat every model change as an evaluation event, even inside the same provider.
• Hard-coding provider fields across the codebase: This creates a second migration the next time an endpoint or SDK changes. Keep translation at the boundary.
• Retrying every failure: Invalid requests, schema failures, authorization errors, and safety refusals usually require different handling from transient capacity or network errors.
• Replacing embeddings without planning reindexing: A retrieval migration may require a new vector index, backfill process, validation period, and rollback path.
• Ignoring observability: Track request identifiers, adapter version, model identifier, latency, failure category, validation result, and safe usage metrics so output changes can be investigated.
• Launching without a rollback: Keep the previous adapter and configuration deployable until the new path has passed functional, quality, and operational checks.

When to revisit

Revisit this guide before switching providers, replacing a model, upgrading an SDK, moving from a legacy endpoint, changing an embedding model, adding tool use, or investigating unexplained output regressions. It is also worth reviewing before a seasonal planning cycle or any launch that increases request volume.

Maintain a short migration record with the provider, model, adapter version, prompt version, schema version, evaluation set, known limitations, and rollback configuration. Re-run the regression set when any of those inputs change. Refresh the compatibility table and implementation examples after major API releases rather than assuming that an older integration remains equivalent.

Use this final release checklist:

1. Freeze and label the current production configuration.
2. Capture representative inputs and baseline outputs.
3. Implement the target behind a provider-neutral adapter.
4. Test messages, prompts, structured outputs, tools, streaming, embeddings, limits, and errors.
5. Compare quality and operational metrics against explicit acceptance criteria.
6. Deploy with a canary, feature flag, or limited traffic split when practical.
7. Monitor validation failures, latency, retries, refusals, tool errors, and user-visible regressions.
8. Keep rollback available until the new path is stable and the migration record is complete.

Provider portability is not achieved by hiding every difference. It comes from making differences visible, testable, and contained at a well-defined boundary.