LLM Evaluation Guide: Build a Practical Test Set and Score Model Responses

Overview

LLM applications are difficult to evaluate with a single metric. A response can be factually correct but poorly formatted, helpful but too verbose, or fluent while relying on unsupported information. Model quality also varies by task: a model that performs well on extraction may not be the best choice for nuanced classification or retrieval-grounded answers.

A practical LLM evaluation guide therefore starts with the decision you need to make. You might be asking whether a new system prompt is better, whether a smaller model is adequate, whether a RAG pipeline retrieves the right evidence, or whether a code change caused a regression. Each question requires a test set and rubric suited to that use case.

Use this four-part evaluation loop:

1. Collect representative cases. Include normal requests, edge cases, ambiguous inputs, and previously failed production examples.
2. Define observable criteria. Separate correctness, completeness, format, safety, tone, latency, and cost instead of hiding them in one vague score.
3. Run a controlled comparison. Keep the input cases, generation settings, tool access, and scoring process consistent.
4. Review failures and update the set. A failed response is often more valuable than another easy example.

This approach is more actionable than relying only on a general AI model benchmark. Public benchmarks can provide context, but your own evaluation set should reflect the users, documents, constraints, and failure costs of your application.

How to estimate

Begin by creating a test-set table. A useful minimum schema is:

• Case ID: A stable identifier such as support-014 or invoice-027.
• Input: The user request and any permitted context.
• Expected behavior: The facts, fields, actions, or constraints the response must satisfy.
• Reference answer or labels: A human-approved target where one exists.
• Risk level: Low, medium, or high based on the effect of an error.
• Evaluation method: Exact match, rule-based validation, human review, or an evaluator model.
• Observed output and scores: Stored separately for each prompt or model version.

For each candidate, calculate a pass rate:

Pass rate = passed cases ÷ evaluated cases × 100

If cases have different importance, use a weighted score instead:

Weighted score = sum of (case weight × case score) ÷ sum of case weights

For example, assign a weight of 3 to a high-risk case, 2 to a normal case, and 1 to a low-risk case. This prevents a large number of easy cases from masking poor performance on the situations that matter most.

Track more than quality. A simple model-selection worksheet can include:

• Quality score or pass rate
• High-risk case pass rate
• Average and worst-case latency
• Input and output token counts
• Estimated cost per request using your current pricing inputs
• Structured-output validation rate
• Tool-call or retrieval success rate, when applicable
• Human review time per case

Do not treat these measures as interchangeable. A candidate with a slightly higher quality score may be unsuitable if it exceeds your latency budget. Conversely, the lowest estimated cost may not be economical if it creates additional review work or retries.

Inputs and assumptions

The reliability of an LLM testing framework depends on the quality of its inputs. Build the test set from several sources rather than writing every case from scratch:

• Frequently requested user tasks
• Known failures from logs, support tickets, or manual review
• Boundary cases, such as empty fields, long documents, conflicting instructions, or unusual formatting
• Adversarial cases involving prompt injection, irrelevant context, or misleading wording
• Cases that test language, domain terminology, and expected output formats

Keep a distinction between the development set and the holdout set. Use the development set while refining a prompt or pipeline. Reserve the holdout set for a final comparison. If you repeatedly optimize against every case, the system can appear to improve while simply memorizing the evaluation patterns.

Define the scoring rubric before comparing candidates. For a customer-support answer, a five-point rubric might assess factual accuracy, policy alignment, completeness, tone, and escalation behavior. For extraction, use field-level checks: required fields present, values correct, no unsupported fields, and valid JSON. For RAG, evaluate both retrieval and generation: did the system retrieve the relevant passage, and did the final answer stay supported by it?

Use deterministic checks whenever possible. JSON schema validation, required-field checks, string normalization, allowed-label checks, and citation-presence checks are easier to reproduce than subjective judgments. Human or evaluator-model scoring can supplement these checks for qualities such as helpfulness or reasoning clarity, but document the rubric and review a sample of scores.

For a structured-output workflow, a test record might look like this:

{
  "case_id": "order-008",
  "expected_fields": ["order_id", "status", "needs_escalation"],
  "risk": "high",
  "checks": ["valid_json", "required_fields", "status_allowed", "escalation_correct"]
}

Keep model name, prompt version, retrieval settings, temperature or equivalent controls, tool configuration, timestamp, and software version with every run. Without this metadata, a score is difficult to reproduce or explain.

Worked examples

Example 1: Comparing two prompts

Suppose a classification application has 40 test cases. Prompt A passes 34 cases, while Prompt B passes 36. Their raw pass rates are 85% and 90%. That suggests an improvement, but inspect the six and four failures before changing production. If Prompt B fails two high-risk cases that Prompt A handled correctly, the weighted result may favor Prompt A.

Record the difference by category: billing, account access, technical issue, and ambiguous request. A category-level table can reveal that an overall improvement comes entirely from easy billing cases while technical cases remain unchanged.

Example 2: Comparing models with quality and cost

Imagine three candidate models evaluated on the same 50 cases. Model X passes 44, Model Y passes 42, and Model Z passes 40. You then add latency and cost columns using the current values from your provider accounts or pricing documentation. If Model X is substantially more expensive but only improves low-risk cases, Model Y may be the better operational choice. If Model X prevents failures in high-risk cases, the extra cost may be justified.

Do not publish a universal “best model” conclusion from this table. The result is specific to the test set, prompt, context window, tools, and scoring policy used. Save the worksheet so it can be rerun when those inputs change.

Example 3: Adding a regression test

A production user reports that an assistant invents a return deadline when the policy document contains no deadline. Add the original request, relevant context, and expected behavior to the test set. The expected result might be: state that the deadline is not found, avoid inventing a date, and offer the approved next step. Every future prompt, model, or retrieval change must pass this case before release.

When to recalculate

Run the full evaluation before releasing a new model, system prompt, retrieval configuration, tool integration, or output schema. Also rerun it after changing chunking, reranking, temperature settings, fallback logic, or post-processing. These changes can alter outputs even when the user-facing prompt remains the same.

Revisit the test set on a regular maintenance cycle and whenever production reveals a new failure pattern. Add cases when users encounter unsupported answers, incorrect classifications, malformed JSON, missed tool calls, excessive latency, or unexpected refusals. Retire a case only when its business purpose no longer exists; otherwise, preserve it as historical coverage.

Refresh cost and latency inputs when provider pricing, model availability, infrastructure, traffic volume, or token usage changes. Recalculate monthly or quarterly if the application is actively evolving, and before making a model-selection decision. Keep historical runs rather than overwriting them so you can distinguish genuine improvement from a change in test composition.

For a practical release gate, require every high-risk case to pass, set a minimum overall weighted score, and define acceptable latency and cost limits. Investigate any regression instead of averaging it away. Finally, link each failed case to an action: improve the prompt, revise retrieval, add validation, change the model, require human approval, or document an intentional limitation. That turns LLM evaluation from a one-time benchmark into an operating process for safer, more predictable AI development.