RAG Tutorial: Build, Test, and Improve a Retrieval-Augmented Generation App

Overview

Retrieval-augmented generation, or RAG, combines information retrieval with text generation. Instead of asking a language model to answer from its general training alone, your application first finds relevant passages in a collection of documents. Those passages are then added to the model input as context.

A typical RAG pipeline has six stages:

1. Ingestion: collect source files, web pages, records, or other approved content.
2. Preparation: extract text, remove unwanted elements, preserve useful metadata, and normalize the content.
3. Chunking: divide documents into searchable sections without destroying their meaning.
4. Indexing: create embeddings and store them with the original text and metadata in a vector database or search system.
5. Retrieval: use a user question to find relevant chunks, optionally applying filters or reranking.
6. Generation: provide the retrieved context to a language model and require an answer that is appropriately grounded.

RAG is not a single model feature. It is an application workflow with several independent failure points. A strong model cannot reliably compensate for missing documents, poor extraction, irrelevant retrieval, or an unclear prompt. Treat each stage as a testable component.

For a broader production perspective, compare this workflow with the guidance in how to build an internal AI knowledge base with RAG, permissions, and auditability. If you are using a framework, the LangChain production tutorial can help you assess which abstractions are useful and which should remain explicit in your application code.

Checklist by scenario

Scenario 1: Building a first document question-answering app

• Define the document collection and the questions the application must answer.
• Choose a small, representative test set before selecting a vector database or framework.
• Inspect extracted text manually. Confirm that headings, tables, lists, page boundaries, and code blocks are not being lost.
• Store source identifiers, titles, section names, URLs, timestamps, and access metadata alongside each chunk.
• Start with moderate chunks and test several overlap values rather than assuming one universal setting.
• Keep the original chunk text available so you can inspect exactly what the model received.
• Make the answer prompt explicit: use the supplied context, distinguish missing information, and cite the supporting source.
• Log the question, retrieved chunks, filters, prompt version, model identifier, response, latency, and failure status.

A simple prompt assembly pattern looks like this:

System: Answer using only the provided context when the question depends on the source collection. If the context is insufficient, say what is missing. Cite the source identifiers attached to relevant passages.

Context:
[Source: handbook-07, Section: Access]
...

Question:
{user_question}

The exact wording can change, but the application should make the boundary between instructions, retrieved data, and the user question clear. This also makes prompt testing and debugging easier.

Scenario 2: Improving retrieval quality

• Create questions that represent direct lookups, paraphrases, multi-part requests, and questions with no valid answer in the collection.
• Measure whether the relevant passage appears in the top results before judging the generated answer.
• Compare keyword, semantic, and hybrid retrieval where the content contains exact names, identifiers, product codes, or technical terms.
• Use metadata filters for attributes such as product, department, version, tenant, region, or publication status.
• Test whether chunks contain enough surrounding context to answer the question without adding unrelated material.
• Consider reranking only after you can observe that initial retrieval is returning plausible candidates.
• Review failed searches by category: missing source, extraction error, poor chunk boundary, vocabulary mismatch, incorrect filter, or ranking problem.

Embedding choice is part of the retrieval design, not a permanent branding decision. Re-evaluate it against your language, document types, query style, and operational constraints. See how to choose an embedding model for search, RAG, and classification for a focused comparison checklist.

Scenario 3: Adding citations and trustworthy answers

• Give every retrieved chunk a stable source identifier.
• Keep citation metadata separate from the text so the model cannot accidentally rewrite the source location.
• Require citations only when the answer uses retrieved content; do not create citations for unsupported statements.
• Display enough source context for a user to verify the answer.
• Test contradictory documents and outdated versions.
• Define what happens when retrieval returns no acceptable result: ask a clarifying question, provide a limited response, or decline to answer.
• Apply authorization before retrieval, not merely when displaying the response.

Citations improve reviewability, but they do not prove that an answer is correct. A citation can be relevant while the generated conclusion is too broad. Evaluate both retrieval relevance and claim-level grounding.

Scenario 4: Preparing for production

• Set limits for retrieved chunks, input length, output length, retries, and tool calls.
• Record usage and latency metrics without storing sensitive content unnecessarily.
• Build an evaluation dataset from real questions, reviewed examples, and known edge cases.
• Version ingestion code, chunking settings, embedding models, retrieval settings, prompts, and answer models.
• Use staged indexing when documents change so a new index can be compared with the current one.
• Test access controls, prompt injection attempts, malformed documents, and unavailable dependencies.
• Provide a feedback path that captures whether the problem was missing information, poor retrieval, an incorrect answer, or an unusable response.

Production observability should cover more than model latency. The guide to monitoring LLM applications provides a useful framework for tracking failures, cost signals, user feedback, and the stages of an AI workflow.

What to double-check

Document preparation

Confirm that the source content is complete, authorized for use, and represented in a format the parser can handle. PDFs, scanned pages, spreadsheets, presentations, and web pages often require different extraction paths. Keep a sample of original documents next to extracted output so regressions are visible.

Chunk boundaries

Chunks should usually preserve a coherent idea: a procedure, subsection, policy rule, or group of related rows. Splitting every fixed number of characters can separate a heading from its explanation or divide a condition from its exception. Use structure-aware splitting when the document format provides headings or sections, then validate the result with real questions.

Retrieval behavior

Inspect the top results for every evaluation question. Ask whether the result is relevant, sufficiently complete, and permitted for the requesting user. A high similarity score is only a ranking signal; it is not a guarantee that the passage answers the question.

Prompt assembly

Keep instructions stable and context clearly delimited. Tell the model how to handle uncertainty, conflicting sources, and requests outside the collection. Avoid filling the context window with every vaguely related result. More text can make the answer harder to verify when the additional passages are noisy or contradictory.

Evaluation

Separate retrieval evaluation from generation evaluation. For retrieval, check whether the expected evidence is present and ranked sufficiently high. For generation, check factual support, completeness, citation accuracy, refusal or uncertainty behavior, format compliance, and whether the response actually answers the question. Store examples of failures, not just aggregate scores, because examples reveal which component needs attention.

For related guidance on reducing unsupported answers without making the system unhelpfully rigid, read how to reduce hallucinations in RAG systems.

Common mistakes

• Indexing without a test set: Teams often tune settings by intuition. A small, reviewed question set gives every change a consistent reference point.
• Changing several components at once: If chunk size, embeddings, retrieval count, prompt, and model all change together, you cannot identify the cause of an improvement or regression.
• Assuming semantic search solves exact-match queries: IDs, error codes, legal terms, and product names may benefit from keyword or hybrid retrieval.
• Ignoring metadata: Without source, version, date, and permission metadata, filtering, citations, and updates become difficult.
• Returning an answer for every question: A well-designed RAG app needs a clear behavior for insufficient or conflicting evidence.
• Trusting citations automatically: Verify that each cited passage supports the specific claim, not merely the general subject.
• Rebuilding indexes manually: Make ingestion repeatable and record the configuration used to create each index.
• Treating prompt changes as harmless: A small instruction change can affect refusal behavior, citation format, and answer scope. Track prompts as versioned application assets. The prompt versioning checklist can help establish a practical review process.
• Skipping security tests: Retrieved documents may contain instructions that attempt to influence the model. Separate trusted application instructions from untrusted content and review the prompt injection prevention checklist.

When to revisit

Revisit this RAG checklist whenever the underlying inputs or workflow change. At minimum, review it before a major planning or release cycle, after adding a new document type, and whenever users report that answers are incomplete, outdated, incorrectly cited, or unexpectedly confident.

Run a focused evaluation when you change:

• the embedding model, vector database, search algorithm, reranker, or retrieval count;
• document parsers, chunking rules, metadata fields, or indexing schedules;
• the system prompt, context format, citation rules, or fallback behavior;
• the language model, model settings, context limits, or structured output schema;
• permissions, tenant boundaries, source freshness requirements, or retention practices.

Use a repeatable review sequence: freeze a representative evaluation set, record the current configuration, make one controlled change, compare retrieval and answer results, inspect regressions manually, and keep or roll back the change based on evidence. Update the evaluation set when the product gains new use cases, but retain older cases so improvements do not hide regressions.

Before acting on a new RAG implementation, complete this short final pass:

1. Can you identify the exact source passage behind a typical answer?
2. Does the system behave sensibly when no relevant passage is retrieved?
3. Are document versions, permissions, and citations preserved end to end?
4. Can you tell whether a failure came from ingestion, retrieval, prompting, or generation?
5. Can you reproduce the current index and prompt configuration?
6. Do your tests include ambiguous, unsupported, adversarial, and multi-part questions?

If the answer to these questions is yes, your RAG application has a foundation that can be tested and improved as models, documents, and retrieval tools change. If not, improve observability and evaluation before adding more complexity.