Prompt Evaluation Framework: How to Test, Score, and Improve LLM Prompts

Overview

Prompt evaluation is the practice of testing an LLM prompt systematically instead of judging it from a few impressive responses. The goal is not to find one universally perfect prompt. It is to understand which version performs best for a defined task, under defined conditions, and at an acceptable cost and level of risk.

A useful evaluation process has five parts:

1. Define the task: State what the model must produce and who will use the result.
2. Build a test set: Collect representative inputs, including ordinary, difficult, ambiguous, and invalid cases.
3. Set evaluation criteria: Decide what quality means before reviewing outputs.
4. Compare versions: Run the same inputs against each prompt, model, and relevant configuration.
5. Record decisions: Keep the results, failures, and rationale so future changes can be evaluated consistently.

This workflow applies to classification, extraction, summarization, customer support, coding assistance, RAG applications, and agentic workflows. It also complements prompt version control. For a related process, see Prompt Versioning for Teams: How to Track Changes, Tests, and Rollbacks.

Template structure

Start with a small evaluation record that can live in a spreadsheet, JSON file, database, or test repository. The format matters less than consistency. Each test case should contain the following fields:

{
  "case_id": "support-001",
  "input": "The customer wants to cancel an order placed yesterday.",
  "context": "Cancellation is allowed before shipment.",
  "expected_behavior": "Explain the next step and do not invent an order status.",
  "reference_output": "Optional approved answer or required facts",
  "risk_level": "medium",
  "tags": ["cancellation", "policy", "customer-support"]
}

The input is the user message or source document. Add context when the application supplies retrieved text, account data, tool results, or other variables. Expected behavior describes the outcome in plain language. It is often more useful than a single reference answer because several responses may be acceptable.

Use a scoring sheet for each prompt run. A practical structure includes:

• Correctness: Does the response contain the required facts and reach the right conclusion?
• Instruction following: Does it follow formatting, tone, length, and workflow requirements?
• Grounding: Is it supported by the supplied context, with uncertainty stated when evidence is missing?
• Completeness: Does it address all important parts of the request?
• Safety and policy fit: Does it avoid disallowed disclosures, unsupported claims, or inappropriate actions?
• Usability: Can a person or downstream system use the result without unnecessary editing?

Choose a simple scale, such as zero to two: zero for a failure, one for a partial result, and two for an acceptable result. Add binary checks for hard requirements, such as valid JSON, a required field, or a citation to supplied context. A weighted score can help when correctness matters more than style:

Total score = correctness × 0.4 + grounding × 0.25 + instruction following × 0.2 + completeness × 0.15.

The exact weights are assumptions, not universal standards. Document them and change them only when the task’s priorities change.

How to customize

Customize the framework around the real failure modes of your application. A content summarizer may prioritize factual coverage, compression, and preservation of names or numbers. An extraction prompt may prioritize field accuracy, missing-value handling, and schema validity. A coding assistant may need checks for compilation, test behavior, security-sensitive patterns, and whether the answer explains assumptions.

Build the test set from several sources rather than ideal examples alone:

• Typical inputs from the normal workflow.
• Previously reported failures and user corrections.
• Long, short, incomplete, and poorly formatted inputs.
• Ambiguous requests that should trigger clarification.
• Inputs containing irrelevant or conflicting information.
• Adversarial or instruction-conflicting text, where relevant.

Keep a small “golden set” for fast checks and a larger set for scheduled evaluations. The golden set should contain cases that represent the most important behaviors, including at least a few known failures. Do not remove a difficult case simply because a prompt now passes it; preserve it as a regression test.

Separate prompt quality from model and application quality. When comparing two prompts, hold the model, temperature or equivalent sampling settings, input data, retrieved context, and output parser constant where possible. If you change several variables at once, record the combination clearly and avoid attributing the result to the prompt alone.

For RAG applications, evaluate retrieval and generation separately. First check whether the correct source passages were retrieved. Then assess whether the model used those passages accurately. A prompt cannot reliably compensate for missing or irrelevant context. The RAG testing and improvement tutorial provides a useful companion workflow.

Examples

Example 1: Structured extraction

Suppose a prompt extracts invoice fields into JSON. Test cases should include complete invoices, missing totals, multiple dates, currency symbols, inconsistent labels, and scanned text with errors. Score exact field accuracy separately from JSON validity. A response with correct values but invalid syntax may be unusable by the application, while valid JSON with incorrect values is also a failure.

A useful acceptance rule might be: all required fields must be correct, optional fields may be null when unsupported, and the output must parse without repair. This rule is more actionable than a general instruction to “be accurate.”

Example 2: Grounded summarization

For a document summarizer, create a reference checklist of facts that must appear and claims that must not appear. Include documents with unresolved questions and contradictory passages. Score whether the summary distinguishes stated facts from inference. If your application depends on this workflow, compare models and task patterns using a focused evaluation rather than relying on broad claims about the best model; see the model comparison guide for additional context.

Example 3: Support response drafting

Test routine requests, policy edge cases, angry messages, requests for confidential information, and cases requiring escalation. Score the response for correct policy use, appropriate tone, clear next steps, and refusal or escalation behavior. A polished tone should not compensate for an incorrect policy answer.

Example 4: Prompt comparison

Run prompt version A and prompt version B on the identical test set. Review aggregate scores, but also inspect individual cases. A higher average can hide a serious regression in a high-risk category. Record the prompt version, model identifier, date, input variables, output, scores, reviewer notes, latency, and token usage when those operational measures matter.

When to update

Revisit the evaluation whenever the prompt, model, retrieved content, tools, parser, or surrounding application changes. Also update it after a user reports a failure, a new input category appears, or the business rules change. New model releases and changes in model defaults are additional reasons to rerun the suite, even when your prompt has not changed.

Use a practical update cycle:

1. Run the golden set before merging a prompt change.
2. Run the broader suite before a production release.
3. Review failures by category instead of looking only at the average score.
4. Add important failures as permanent test cases.
5. Keep the previous prompt and results for rollback and comparison.

Start today with 20 to 50 representative cases, five clearly written criteria, and two prompt versions. Review every failure manually at first. Once the rubric is stable, you can introduce automated checks or model-assisted grading, but retain human review for high-risk cases and for validating whether the grader itself is behaving consistently. Treat the evaluation set as a living part of the application: small, traceable updates make prompt improvements easier to trust and regressions easier to find.