LLM Structured Outputs: JSON Schema Prompting, Validation, and Error Handling

Overview

An LLM produces text unless the surrounding API and application impose a reliable structure. Even when a model follows instructions well, responses can contain missing fields, incorrect types, extra commentary, invalid escape sequences, or values that technically parse as JSON but fail your business rules.

A dependable structured-output workflow has five layers:

1. Define the contract: specify the fields, types, required values, allowed options, and rules for empty or unknown data.
2. Constrain generation: use structured outputs, JSON mode, function calling, or carefully written prompt instructions when supported.
3. Validate the result: parse the response and validate it against the same contract used by your application.
4. Recover deliberately: retry only when the failure is likely to be temporary or correctable.
5. Observe and review: log failures safely, measure field-level errors, and revisit the schema when requirements or model behavior change.

This distinction matters: JSON mode may help produce syntactically valid JSON, while schema-constrained output can provide stronger guarantees about the shape. Neither approach automatically proves that the values are accurate, complete, or appropriate for your product.

What to track

1. The schema contract

Start with the smallest useful schema. Every field should have a clear purpose and a defined failure behavior. For example, a support-ticket extractor might use:

{
  "type": "object",
  "properties": {
    "summary": { "type": "string" },
    "priority": { "type": "string", "enum": ["low", "medium", "high"] },
    "customer_sentiment": { "type": "string", "enum": ["positive", "neutral", "negative", "unknown"] },
    "needs_human_review": { "type": "boolean" },
    "missing_information": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["summary", "priority", "customer_sentiment", "needs_human_review", "missing_information"],
  "additionalProperties": false
}

Use an explicit enum when downstream code expects a finite set of values. If information may be unavailable, define a value such as unknown or use a documented nullable representation rather than leaving the behavior implicit. Check your provider's supported JSON Schema subset before relying on advanced features such as unions, complex nesting, or conditional rules.

2. Field-level instructions

The schema describes shape; the prompt should explain meaning. Tell the model what evidence to use, how to handle ambiguity, and what not to infer. A useful instruction might be:

Extract only information supported by the ticket text.
- summary: one concise sentence describing the customer's issue.
- priority: choose high only when the text indicates a severe impact, urgent deadline, or broad outage.
- customer_sentiment: use unknown when the tone is unclear.
- needs_human_review: true when required information is missing or the request is outside the supported workflow.
- missing_information: list specific facts needed to continue; return [] when nothing is missing.

Keep formatting instructions separate from decision rules. This makes prompt changes easier to test and helps you identify whether a failure came from the schema, interpretation, or source text.

3. Validation signals

Track more than whether parsing succeeded. Useful measures include JSON parse failures, schema-validation failures, missing required information, enum violations, unexpected extra fields, empty outputs, refusal or safety interruptions, and business-rule failures. A response can pass schema validation while still assigning the wrong priority or inventing a value, so include task-specific checks and human review samples.

4. Model and request metadata

For each request, retain enough metadata to reproduce a problem without storing sensitive input unnecessarily: schema version, model identifier, application version, prompt version, latency, token or usage data when available, validation result, retry count, and a redacted failure category. Treat logs as production data and apply your normal access controls and retention rules.

Cadence and checkpoints

Structured-output reliability should be reviewed on a regular cadence, not only after an outage. A monthly or quarterly checkpoint is usually enough for stable applications, with additional review after any schema, prompt, model, SDK, or input-source change.

Before release

• Test valid, incomplete, ambiguous, adversarial, and unusually long inputs.
• Verify that every required field is consumed correctly by downstream code.
• Test malformed provider responses and network failures in the retry path.
• Confirm that sensitive content is not written to ordinary application logs.
• Record a schema version and maintain a small regression set of representative examples.

During routine review

• Compare parse, validation, retry, and human-review rates with the previous period.
• Group failures by field instead of treating every failure as a generic JSON error.
• Inspect new enum values, recurring nulls, unexpectedly long strings, and frequent empty arrays.
• Re-run the regression set after changing prompts, schemas, models, or SDK versions.

Do not automatically retry every failure. A temporary transport error, truncated response, or correctable formatting issue may justify a retry. A schema mismatch caused by an unsupported feature or a prompt that requests contradictory values needs code or configuration changes instead.

How to interpret changes

A rising parsing-failure rate often points to an integration problem: a changed response path, incorrect extraction of the provider's tool arguments, truncation, or a model/API configuration change. A stable parse rate with more schema failures suggests that the model is producing structured data but not conforming to the contract. Investigate field-level patterns before rewriting the entire prompt.

Frequent failures in one optional-looking field may mean the field should be nullable, have an explicit unknown value, or be removed from the first extraction step. Repeated enum violations may indicate that the categories overlap or do not match the language used in real inputs. If reviewers reject otherwise valid records, the issue is likely semantic rather than syntactic; improve examples, decision rules, or evaluation criteria instead of adding more formatting instructions.

When using few-shot examples, choose examples that clarify boundary cases rather than simply adding volume. For related guidance, see these few-shot prompting examples for classification and extraction. If the structured result feeds a retrieval workflow, validate the extracted fields before indexing or using them in generation; the production RAG tutorial provides broader guidance on evaluating retrieval applications.

A reusable retry pattern

Keep the first request simple and make a retry informative without exposing internal implementation details to the model:

async function extractWithRetry(input, schema) {
  const first = await callModel({ input, schema });
  const checked = validate(first, schema);
  if (checked.ok) return checked.value;

  const repairRequest = {
    input,
    schema,
    previousOutput: redact(first.raw),
    errors: checked.errors,
    instruction: "Return a new response that satisfies the schema. Do not add commentary."
  };

  const second = await callModel(repairRequest);
  const repaired = validate(second, schema);
  if (repaired.ok) return repaired.value;

  throw new StructuredOutputError({
    category: "validation_failed_after_retry",
    errors: repaired.errors
  });
}

In production, add a maximum retry count, timeouts, backoff for transient errors, and a controlled fallback such as a review queue. Never execute a tool call or persist important data solely because the model returned parseable arguments. Apply authorization and business validation in application code.

When to revisit

Revisit this guide and your implementation when a model or API provider changes structured-output support, when an SDK changes response objects, when your application adds or renames fields, or when parsing and validation errors begin to trend upward. Also review the workflow when the input distribution changes—for example, a new document type, language, customer segment, or much longer context.

At each review, confirm four things: the schema still represents the product requirement; the provider still supports the features you use; the prompt explains ambiguous fields; and your tests cover recent failures. Keep schema changes versioned. Adding a truly optional field may be backward-compatible, while renaming a required field or changing an enum can break consumers and stored records.

For a practical next step, create a small evaluation set of representative inputs, implement schema validation before business logic, and record failures by category. Run that set whenever you change the prompt, schema, model, or SDK. Structured outputs are most reliable when treated as a maintained interface—not as a formatting trick—and this review loop gives your team a clear way to improve that interface over time.