LLM observability turns an opaque AI feature into a system you can inspect. This guide shows how to trace prompts, model calls, retrieval, tools, latency, token usage, cost, errors, and user feedback, then use repeatable calculations to find the likely cause of slow, expensive, or unreliable responses.
Overview
Production monitoring for an LLM application needs more than an uptime check. A request can return a successful HTTP response while still being too slow, factually wrong, expensive, or unsafe for its intended use. LLM observability connects the user’s request to every important step in the response path.
A useful trace usually contains a request or conversation identifier, timestamps, model and version identifiers, prompt-template version, input and output token counts, latency, status, error category, and a redacted record of the result. For a retrieval-augmented generation (RAG) workflow, add embedding time, search latency, retrieved document identifiers, scores or ranks, and the context supplied to the model. For an AI agent, record tool names, arguments, results, retries, approval events, and the reason the agent stopped.
The goal is not to collect everything indiscriminately. The goal is to answer practical questions:
- Did the request fail before reaching the model, during generation, or in application post-processing?
- Did a prompt, model, retrieval setting, tool call, or traffic pattern cause the change?
- Is the application slow because of queueing, network time, retrieval, generation, or repeated calls?
- Is spending rising because requests are longer, outputs are larger, retries increased, or routing changed?
- Did response quality decline for all users or only for a specific task, tenant, language, or document set?
Think of an observability setup as three connected layers. Traces explain individual requests. Metrics reveal trends across many requests. Evaluations and user feedback reveal whether technically successful responses are actually useful. Comparing these layers is more informative than relying on a single dashboard.
How to estimate
Start with a simple request-level cost and latency model. The exact billing formula depends on the provider and pricing plan, so keep rates configurable rather than embedding them permanently in application code.
Estimated request cost can be represented as:
(input tokens ÷ 1,000,000 × input rate) + (output tokens ÷ 1,000,000 × output rate) + tool, retrieval, or infrastructure charges
For a batch of requests:
estimated batch cost = request count × average request cost
Use percentiles for performance rather than averages alone. The median, or p50, describes a typical request. P95 shows the experience of slower requests, while p99 helps expose a long tail caused by large contexts, retries, overloaded dependencies, or complex agent paths.
A basic end-to-end latency breakdown is:
total latency = queue time + application time + retrieval time + model time + tool time + post-processing time
Some stages overlap, so this should be treated as a diagnostic model rather than a guaranteed accounting identity. Instrument timestamps at the start and end of each stage, then compare the sum with the observed request duration.
For reliability, calculate rates by error category:
error rate = failed requests ÷ total requests × 100
Separate timeouts, rate limits, authentication failures, malformed outputs, tool errors, content-policy blocks, validation failures, and application exceptions. A single “AI error” number hides the action required to fix the problem.
For quality, combine automated checks with sampled review. Track structured-output validation, citation or source checks, task-specific graders, user ratings, escalation rates, and repeat attempts. A response that passes schema validation may still be unhelpful, while a low rating may reflect a product issue rather than a model issue. Link each quality signal to the trace and the prompt or model version that produced it.
Inputs and assumptions
Before comparing observability tools or building a dashboard, define the fields your team actually needs. A small application may need request traces, token usage, latency, errors, and feedback. A RAG or agent system needs more detailed spans and relationships between parent and child operations.
Core trace fields
- Request ID, session ID, tenant or workspace identifier, and environment.
- Model provider, model identifier, API route, temperature or other relevant settings, and prompt version.
- Input and output token counts, estimated cost, start time, end time, and time-to-first-token where applicable.
- Final status, retry count, timeout reason, exception type, and application response code.
- Redacted input, output, tool arguments, and retrieved context when retention is appropriate.
What to compare in an observability tool
Compare tools by workflow fit rather than by the number of dashboard features. Check whether tracing works with your model providers, application framework, queue, vector database, and tool layer. Look for support for nested spans, searchable metadata, prompt-version comparison, token and cost fields, latency percentiles, sampling controls, exports, alerts, and role-based access.
Also inspect operational details. Can sensitive fields be masked before export? Can retention be configured? Can engineers retrieve a trace during an incident without exposing an entire customer conversation? Is the instrumentation vendor-specific, open, or easy to replace? These questions matter when your application changes providers or frameworks; a provider migration guide can help you identify fields that should remain provider-neutral.
Do not assume that storing complete prompts and responses is always appropriate. Classify data, redact secrets and personal information, restrict access, and establish retention rules before enabling production capture. Use deterministic test inputs or sampled traces for debugging where full payload retention is unnecessary. For security-sensitive applications, pair observability with the controls described in the prompt injection defense guide.
Worked examples
Example 1: estimating a request budget
Assume an internal assistant processes 10,000 requests in a planning exercise. Each request is estimated at 2,000 input tokens and 500 output tokens. Use placeholder rates from your provider’s current pricing page; the numbers below are intentionally illustrative, not a current price claim.
If the configured input rate is 1 cost unit per million tokens and the output rate is 4 cost units per million tokens:
- Input volume: 10,000 × 2,000 = 20,000,000 tokens.
- Output volume: 10,000 × 500 = 5,000,000 tokens.
- Input estimate: 20 × 1 = 20 cost units.
- Output estimate: 5 × 4 = 20 cost units.
- Illustrative model total: 40 cost units, before tools, retrieval, retries, and infrastructure.
Now compare that estimate with observed usage. If the dashboard shows 3,000 input tokens per request instead of 2,000, investigate conversation history, retrieved context, repeated system instructions, or an unexpectedly verbose tool result. The AI API cost optimization guide provides a useful next step for token budgets, caching, and routing decisions.
Example 2: finding a latency regression
Suppose p50 latency remains stable but p95 rises after a prompt release. Trace comparison shows that the slower requests retrieve twice as many passages and make a second model call when JSON validation fails. The prompt itself may not be the only cause. Compare retrieved-context size, validation-failure rate, retry count, and output length between the old and new versions.
A practical incident hypothesis would be: “The new prompt increases the probability of a repair call for requests with long context.” Test that hypothesis against a representative evaluation set and production traces. The prompt-to-JSON workflow guide can help separate schema design problems from model-generation problems.
Example 3: diagnosing quality complaints
Suppose users report incorrect answers, but uptime and error rates look normal. Filter traces by task type and compare retrieved documents, citation checks, model version, prompt version, and user feedback. If failures cluster around one document collection, the problem may be ingestion or retrieval. If the right context is present but ignored, review prompt structure and model behavior. Use a fixed test set to compare changes rather than judging isolated examples; see the LLM evaluation guide for a repeatable evaluation process.
When to recalculate
Recalculate cost, latency, and reliability baselines after any change that can alter request volume or execution paths. This includes a model or provider change, prompt-template release, context-window change, retrieval-index update, new tool, retry-policy change, traffic increase, or modification to streaming and batching behavior.
Review the dashboard after each deployment, then inspect it on a regular billing or operations cycle. Pricing inputs should be refreshed whenever a provider changes rates, token definitions, caching treatment, or included usage. Benchmark assumptions should be revisited when model versions change or your workload gains new languages, longer documents, or more complex agent tasks.
Use this incident checklist:
- Confirm scope: which users, routes, models, versions, and time window are affected?
- Compare p50, p95, and p99 latency with the previous stable period.
- Check token volume, output length, retry count, cache behavior, and estimated cost.
- Follow the trace through retrieval, model calls, validation, tools, and post-processing.
- Group failures by actionable category instead of treating them as one error rate.
- Sample successful and failed responses, applying redaction and access controls.
- Record the suspected cause, mitigation, owner, and the metric that should confirm recovery.
Finally, keep observability proportional to risk. Begin with a clear trace schema and a few dependable alerts, then add detail where incidents show a blind spot. A well-maintained record of prompts, costs, latency, tool calls, retrieval, and outcomes makes model and tool comparisons more credible—and gives developers a faster path from an alarming metric to a specific fix.