LLM API costs are driven by more than a model’s advertised rate. Input and output tokens, repeated context, retries, caching, model selection, traffic patterns, and monitoring choices all affect the bill. This guide gives you a repeatable way to estimate costs, set token budgets, design a practical routing strategy, and review usage without sacrificing response quality, reliability, or latency.
Overview
The most useful cost optimization work begins with measurement rather than guesswork. Before changing a prompt or switching models, establish how many requests your application receives, how many tokens each request uses, what percentage of calls fail or retry, and which requests require a high-capability model.
A simple monthly estimate is:
Monthly cost = request volume × cost per request + fixed platform and storage costs
For token-based APIs, expand the variable portion:
Cost per request = (input tokens ÷ 1,000,000 × input rate) + (output tokens ÷ 1,000,000 × output rate)
Use the pricing units and token definitions supplied by the provider you are evaluating. Some services distinguish between standard, cached, batch, or priority processing, so keep each category separate in your worksheet. Do not compare models using output quality alone; compare them on quality, token consumption, latency, error rate, and the amount of application logic needed to handle their responses.
Cost reduction usually comes from a combination of small improvements:
- Limit unnecessary input and output tokens.
- Cache repeated instructions, retrieved context, or deterministic results where appropriate.
- Route simple tasks to less expensive models and reserve stronger models for difficult cases.
- Reduce duplicate requests, avoid uncontrolled retries, and batch work that does not require immediate responses.
- Track cost and quality together so savings do not quietly increase support work or failed outcomes.
For applications that generate structured data, reliable validation can prevent expensive repair loops. See the prompt-to-JSON workflow guide for schema, validation, and repair patterns.
How to estimate
Build a worksheet with one row for each meaningful request type, such as document summarization, customer classification, retrieval-augmented answers, code generation, or agent tool calls. Estimating at this level is more useful than applying one average to the entire product.
- Count requests. Record expected daily or monthly volume for each request type. Use separate scenarios for current traffic, launch traffic, and a higher-volume case.
- Measure tokens. Capture input and output tokens from real or representative requests. Sample enough variation to reveal short, typical, and unusually large inputs.
- Apply the provider’s rates. Keep input and output calculations separate. Add any applicable cached, batch, or priority categories as separate lines instead of hiding them in an average.
- Add operational overhead. Account for retries, fallback calls, moderation or classification steps, tool calls, and evaluation traffic. A user-facing answer may trigger several model requests.
- Calculate unit economics. Divide total model spend by successful requests, completed tasks, or another business-relevant unit. This helps compare a cheap but unreliable workflow with a more expensive workflow that completes tasks consistently.
A useful worksheet might contain these columns: request type, monthly requests, average input tokens, average output tokens, retry rate, primary model, fallback model, cacheable tokens, cached-hit rate, estimated cost, quality score, and latency target.
Use weighted averages only after measuring the underlying categories. If one percent of requests consumes a large context window, an overall average can conceal the source of a cost spike. Percentiles or segmented ranges are often more informative for production planning.
Inputs and assumptions
Token budgets
Set a maximum output budget for every endpoint. The limit should be large enough for a complete answer but not so large that a vague prompt can produce unnecessary prose. Define the expected response shape, use concise instructions, and return structured fields when downstream code does not need natural-language padding.
Input budgets matter too. Remove duplicated instructions, stale conversation history, unused tool descriptions, and retrieved passages that do not support the answer. In a RAG application, retrieve fewer but more relevant chunks, then measure whether reduced context changes grounded-answer quality. The vector database and RAG comparison can help frame this trade-off.
Caching
Identify content that repeats across requests: system instructions, policy text, product catalogs, reference documents, or identical user queries. A cache is useful only when the saved computation exceeds the cost and complexity of maintaining it. Define a cache key, expiration rule, invalidation trigger, and privacy boundary.
Do not cache personalized or permission-sensitive results without verifying that the key includes the relevant user, tenant, document version, and access context. Also distinguish prompt caching from application-result caching. They solve different problems and may have different provider rules.
Routing and fallback assumptions
Use a model-routing decision table rather than routing by intuition:
- Routine extraction or classification: start with a smaller model if it meets the measured accuracy threshold.
- Long-context synthesis: compare context handling, input cost, latency, and citation quality.
- Complex reasoning or code changes: use a stronger model when evaluation shows a meaningful improvement.
- Uncertain requests: classify first, then escalate only when confidence, complexity, or risk justifies it.
Define fallback conditions explicitly: timeout, rate limit, invalid structured output, low classifier confidence, or a failed evaluation check. Track fallback volume because an apparently inexpensive primary route can become costly if it frequently invokes a second model.
Quality thresholds should come from a representative test set, not a few impressive examples. The LLM evaluation guide provides a practical framework for scoring responses before changing the production route.
Worked examples
Example 1: Reducing repeated context
Suppose a support assistant receives 100,000 monthly requests. Each request includes a repeated instruction block and product reference text, followed by a variable customer question. The team records input and output tokens separately and finds that the repeated portion represents most of the input.
The team can model three scenarios:
- Baseline: send the full instruction and reference text on every request.
- Compressed prompt: remove duplicated wording and retrieve only the relevant reference sections.
- Cached context: use the provider’s supported caching mechanism for stable content, while keeping customer-specific information outside the shared cache.
For each scenario, calculate input tokens, output tokens, cache hits, cache misses, retries, and estimated cost. Then compare answer-grounding scores and response latency. The winning design is not automatically the one with the lowest token count; it is the one that meets the support team’s quality and response requirements at an acceptable cost.
Example 2: Routing a document workflow
A document workflow performs extraction, validation, and exception handling. Instead of sending every document to the strongest model, the team uses a smaller model for normal extraction and escalates documents that contain ambiguous fields or fail schema validation.
To estimate the route, calculate:
Total cost = normal documents × primary cost + escalated documents × (primary cost + secondary cost) + retry cost
Measure the escalation rate from a representative sample. If the primary model produces invalid JSON too often, its apparent price advantage may disappear through repair calls and manual review. Pair routing with strict schemas and automated checks; the prompt injection defense guide is also relevant when retrieved or uploaded documents can contain untrusted instructions.
Example 3: Batchable background work
Consider nightly tagging of a content library. Because users do not need an immediate response, the team can compare synchronous processing with a supported batch workflow. The estimate should include the number of items, tokens per item, expected completion window, retry behavior, and the cost of reprocessing failed items. If the batch option does not satisfy freshness or operational requirements, keep the synchronous route for time-sensitive jobs and batch only the backlog.
When to recalculate
Recalculate your estimate whenever a provider changes input or output rates, tokenization, context-window rules, caching terms, batch options, or model availability. Because these inputs change independently, review the provider documentation before relying on an old worksheet. Avoid hard-coding prices in application logic; store them in versioned configuration so historical reports remain interpretable.
Perform a monthly usage review that answers five questions:
- Which endpoints consumed the most tokens and money?
- Did average or high-percentile input and output size change?
- How often did retries, fallbacks, and repair calls occur?
- Did cache-hit rates match the assumption in the forecast?
- Did quality, latency, error rates, or user satisfaction change after optimization?
Run a deeper review after a major prompt change, new feature, traffic increase, model migration, or RAG index update. Compare a fixed evaluation set before and after the change, then inspect production samples for failure modes that benchmarks miss. For provider changes, the LLM provider migration guide can help organize compatibility checks, while the API error-handling guide covers failures that can create hidden retry costs.
As a practical starting checklist, export the last billing period, group usage by endpoint and model, record token percentiles, calculate retry and fallback rates, update pricing inputs, and rerun baseline, optimized, and high-volume scenarios. Keep the worksheet with its assumptions and date. That record makes the next billing-cycle review faster and shows whether a cost reduction came from a durable design improvement or simply a temporary change in traffic.