LLM API Error Handling Guide: Diagnose Rate Limits, Timeouts, and Failed Requests

Overview

An LLM API request can fail before it reaches a model, while the provider is processing it, or after your application receives a response. Treating every failure as a generic “API error” makes incidents harder to investigate and can create new problems, such as retry storms, duplicate actions, or hidden data-quality issues.

A reliable error-handling design has five parts:

• Classification: identify whether the failure is caused by credentials, request data, limits, service availability, networking, or parsing.
• Bounded retries: retry only failures that are likely to recover, and stop after a defined limit.
• Timeout control: distinguish connection, read, and total-operation timeouts where your client supports them.
• Structured observability: log enough context to reproduce a failure without exposing secrets or sensitive prompts.
• Fallback behavior: give the user, queue, or downstream service a deliberate outcome when recovery fails.

Provider-specific status codes, headers, SDK behavior, model limits, and retry recommendations can change. Use the provider’s current documentation as the final authority, but keep your application logic organized around these durable categories.

What to track

Status codes and failure categories

The following reference is a starting point rather than a universal contract. Read the response body and provider documentation before assigning a final diagnosis.

Request and response metadata

Record a correlation ID generated by your application, the provider request ID when available, environment, endpoint, model identifier, operation name, attempt number, elapsed time, status code, retry-after guidance, input and output token counts when available, and the final disposition. These fields make it possible to compare failures across deployments and providers.

Do not place API keys, authorization headers, complete user prompts, personal data, or full model responses in ordinary logs. Prefer a redacted request fingerprint, prompt version, input-size bucket, and sampled payload stored under your organization’s access controls.

Timeouts and response parsing

An LLM API timeout may occur during connection setup, while waiting for response headers, while reading a streaming response, or because an overall deadline expired. Configure the narrowest timeout controls your HTTP client provides, and pass a request deadline through internal services so a downstream call cannot outlive the user request indefinitely.

Assume that a successful HTTP response does not guarantee a usable application result. Validate the response schema, required fields, finish state, tool-call arguments, and content type before passing data downstream. For structured output workflows, pair schema validation with a controlled repair or retry path; the guide to LLM structured outputs covers this pattern in more detail.

Cadence and checkpoints

Use bounded exponential backoff

For transient failures, delay retries progressively rather than sending them immediately. Add random jitter so many workers do not retry at the same instant. A provider-neutral formula is:

delay = min(maxDelay, baseDelay * 2 ** attempt) + random(0, jitter)

Use a small maximum attempt count and a total time budget. A retry policy should answer four questions: which errors are retryable, how many attempts are allowed, how long the whole operation may run, and what happens after the budget is exhausted.

for attempt in range(max_attempts):
    try:
        return call_llm(timeout=request_deadline)
    except Error as error:
        if not is_retryable(error) or deadline_exceeded():
            raise
        sleep(backoff_with_jitter(attempt))

This pseudocode is intentionally provider-neutral. In a production SDK, preserve the original exception, inspect structured error fields, honor a valid retry hint, and ensure cancellation interrupts the sleep and request.

Make retries safe

Retries are not automatically safe. A generation request may be repeatable from the application’s perspective, but a tool call that sends an email, creates a ticket, charges an account, or changes a record can produce a duplicate side effect if the first request succeeded but its response was lost.

For side-effecting operations, separate model planning from execution. Validate the model’s proposed action, assign an operation ID, use an idempotency key when the provider or downstream service supports it, and check the current state before repeating the action. A queue with a durable status record is often safer than retrying directly inside a web request.

Review on a monthly or quarterly schedule

At least monthly for active systems, and at minimum quarterly for lower-volume integrations, review error counts by category rather than only by endpoint. Track total requests, successful requests, retry attempts, exhausted retries, timeout rate, rate-limit rate, validation failures, average and high-percentile latency, token usage, and fallback usage. Compare the same operation across model versions, regions, deployments, and application releases where those dimensions exist.

Also record changes made since the previous checkpoint: SDK upgrades, model changes, prompt revisions, concurrency changes, new tools, altered timeout budgets, and provider configuration updates. This change log helps distinguish a traffic problem from a request-shape regression.

How to interpret changes

A rising error rate is a symptom, not a diagnosis. Use the pattern of change to narrow the investigation:

• 401 or 403 failures begin after deployment: compare environment variables, secret rotation, project selection, and permission configuration. Confirm that the application is using the intended credential without printing it.
• 400 or 422 failures rise after a model or SDK change: diff the serialized request. Check renamed parameters, unsupported features, message structure, tool schemas, and input limits.
• 429 failures rise with traffic: compare concurrency, request bursts, token volume, and retry traffic. A retry storm can make the original capacity problem worse. Add admission control or a queue instead of simply increasing retries.
• Timeouts rise while status-code errors remain stable: inspect payload size, output limits, streaming behavior, connection reuse, worker saturation, and downstream deadlines. Separate provider latency from time spent waiting for your own queue or connection pool.
• 5xx failures cluster in a short window: preserve provider request IDs, check service-status information when available, and use a circuit breaker or fallback path if the incident affects user-facing work.
• HTTP success remains high but application failures increase: inspect schema validation, truncated output, refusal or finish states, tool arguments, and content filtering behavior. This is a response-quality or contract problem, not necessarily a transport problem.

Keep a small set of reproducible test cases for authentication, invalid input, rate limiting, timeout handling, malformed structured output, and tool-call recovery. Run them in a non-production environment after significant SDK, model, prompt, or infrastructure changes. A prompt or schema revision can alter request size and response shape even when application code is unchanged. For broader prompt regression coverage, see this prompt evaluation framework.

When to revisit

Revisit this LLM API error-handling guide on a regular monthly or quarterly checkpoint, and immediately after any event that changes the request or execution environment. The most important triggers are a provider API or SDK release, model replacement, authentication-system change, new streaming or tool-use feature, traffic increase, deployment to a new region, or a production incident.

Use this practical checklist before marking an integration ready:

1. List every operation, endpoint, model setting, and downstream side effect.
2. Map each observed failure to a category and decide whether it is retryable.
3. Set connection, read, and total-operation deadlines appropriate to the user workflow.
4. Implement capped exponential backoff with jitter and cancellation support.
5. Add idempotency or state checks before retrying side-effecting work.
6. Validate response schemas, tool arguments, finish states, and content before use.
7. Capture redacted logs, correlation IDs, provider request IDs, latency, attempt count, and final disposition.
8. Test authentication failure, invalid parameters, 429 responses, timeouts, 5xx responses, and malformed output.
9. Review dashboards and retry budgets at the next scheduled checkpoint.

Keep the checklist beside the integration’s runbook, not only in application code. When a new failure pattern appears, add a reproducer, a classification rule, and a documented recovery path. That turns one incident into a lasting improvement to your LLM application’s reliability.