This RAG tutorial explains how to design a retrieval-augmented generation application that can be tested, updated, and operated reliably. You will learn how to compare ingestion, chunking, embedding, vector search, prompt assembly, citation, evaluation, and deployment choices without tying the architecture to one vendor or framework.
Overview
Retrieval-augmented generation, or RAG, connects an LLM to an external collection of documents. Instead of asking a model to answer only from its training or conversation context, the application searches a controlled knowledge source, selects relevant passages, and includes them in the model request.
A practical RAG system usually contains six stages:
- Ingestion: collect documents and extract usable text and metadata.
- Chunking: divide content into searchable sections while preserving meaning.
- Embedding: represent each chunk as a numerical vector.
- Retrieval: find passages related to a user’s question.
- Generation: provide the retrieved context and instructions to an LLM.
- Evaluation and operations: measure answer quality, latency, cost, failures, and feedback.
RAG is not simply a vector database followed by a prompt. Most production issues originate earlier or later in the pipeline: poor document extraction, chunks without enough context, weak metadata filters, ambiguous questions, unsupported answers, or missing evaluation cases. Treat the system as an application architecture rather than a single model feature.
For a deeper implementation sequence, see the RAG tutorial on building, testing, and improving a retrieval-augmented generation app. This guide focuses on comparing design choices and establishing checkpoints that remain useful as tools change.
How to compare options
Begin with the information need, not the database or framework. A support assistant, internal policy search tool, research workspace, and document question-answering API may all use RAG, but they have different requirements for permissions, freshness, response format, and tolerance for uncertainty.
1. Define the retrieval unit
Decide what a result should represent. It might be a paragraph, a section, a product record, a ticket, or a page with structured fields. The best unit is large enough to answer a question and small enough to avoid burying relevant text in unrelated material. Record this decision as a testable assumption rather than treating chunk size as a permanent setting.
2. Separate retrieval quality from answer quality
An answer can be wrong because the correct passage was never retrieved, or because the model misread an adequate passage. Test these failure modes separately. Retrieval tests can check whether the expected document appears in the top results. Generation tests can check factual support, completeness, format, and whether the response appropriately says that the evidence is insufficient.
3. Compare systems against the same test set
When comparing embedding models, vector databases, rerankers, or LLMs, keep the documents, questions, filters, and evaluation criteria consistent. A small curated set of representative questions is more useful than unstructured impressions. Include easy lookups, multi-part questions, terminology variations, outdated documents, and questions with no answer in the corpus.
4. Include operational constraints
Quality is only one selection criterion. Also assess integration effort, data controls, observability, backup and deletion workflows, scaling behavior, latency, and the team’s ability to maintain the system. Pricing and feature availability change, so record the date and assumptions behind any commercial comparison rather than presenting them as permanent facts.
Feature-by-feature breakdown
Document ingestion and metadata
Ingestion should preserve the source’s structure wherever possible. Capture headings, page or section identifiers, document titles, publication or revision dates, product areas, and access-control attributes. Keep the original source reference with every chunk so the application can show citations and support later re-indexing.
Extracted text should be inspected before embedding. Tables, scanned pages, repeated headers, footers, broken line wraps, and hidden markup can reduce retrieval quality. A useful checkpoint is a sample review of raw text, normalized text, and final chunks before the full corpus is processed.
Chunking strategies
Fixed-size chunks are simple and predictable, but they can split definitions or procedures at inconvenient points. Structure-aware chunking follows headings, paragraphs, lists, or records. Overlap can preserve context across boundaries, although excessive overlap increases storage and may return duplicate evidence.
Start with a modest, explainable strategy and test alternatives. When a chunk is retrieved, ask whether a developer or end user could understand it without opening the entire document. If not, improve the chunk’s context or attach parent-section information during prompt assembly.
Embeddings and vector search
Embedding models are useful for semantic similarity, but they do not replace exact matching. Product codes, error identifiers, names, and version numbers may benefit from keyword or metadata search. Hybrid retrieval combines semantic and lexical signals when both meaning and exact terms matter.
Compare embedding options using your own domain questions. A general model may work well for ordinary prose while struggling with technical abbreviations or specialized terminology. Keep the embedding model and preprocessing configuration recorded with the index; changing either generally requires a deliberate re-indexing plan. The guide on choosing an embedding model for search, RAG, and classification provides a useful decision framework.
Filtering, reranking, and query transformation
Metadata filters are essential when users should search only within a team, customer, region, product version, or permitted document set. Apply authorization filters before results reach the model, not merely in the user interface. A reranker can reorder an initial candidate set when simple similarity scores are not sufficiently precise.
Query transformation can help with vague or conversational questions. Examples include resolving references from earlier turns, expanding an abbreviation, or generating several search formulations. Use it carefully: an incorrect rewrite can remove important terms. Log the original question, transformed query, filters, and retrieved identifiers so failures are diagnosable.
Prompt assembly and citations
A RAG prompt should state the task, provide the retrieved context, define how evidence should be used, and specify what to do when the context does not support an answer. A durable instruction might be: “Answer using the supplied sources. Distinguish direct evidence from reasonable interpretation. If the sources do not answer the question, say so and identify what information is missing.”
Do not assume that an instruction to avoid hallucinations is sufficient. Include source identifiers beside passages, ask for citations in a predictable format, and validate that cited sources were actually retrieved. For practical guidance on balancing useful answers with evidence boundaries, read how to reduce hallucinations in RAG systems without overconstraining answers.
Evaluation and monitoring
Evaluation should cover retrieval recall, relevance, groundedness, completeness, citation accuracy, refusal behavior, latency, and cost. Human review remains valuable for ambiguous questions and nuanced business requirements, while automated checks can catch regressions across a larger test set.
Monitor production behavior separately from offline tests. Track retrieval failures, empty results, model errors, slow requests, prompt length, user corrections, and repeated searches. Avoid logging sensitive document content unnecessarily; retain identifiers and diagnostic metadata according to the application’s data requirements. See how to monitor LLM applications in production for a broader operational checklist.
Best fit by scenario
Small, stable document collections
A lightweight application can use a managed search service or a simple vector store with scheduled indexing. Prioritize clear metadata, source links, and a compact evaluation set. Avoid adding agents or multiple retrieval stages until basic search and answer quality are understood.
Large or frequently changing knowledge bases
Use an ingestion pipeline that supports incremental updates, deletion, revision tracking, and failed-document retries. Store a document identifier and content hash so unchanged files are not processed unnecessarily. Separate the source of truth from the search index, which makes rebuilding or changing embedding models safer.
Permission-sensitive internal systems
Represent access attributes in the indexed metadata and enforce them at retrieval time. Test cross-user and cross-group access explicitly, including documents whose permissions have recently changed. An internal knowledge base also needs auditability, ownership, and a process for correcting outdated content. The guide to building an internal AI knowledge base with permissions and auditability covers these concerns in more detail.
Technical support and troubleshooting
Combine semantic retrieval with exact matching for error codes, versions, commands, and configuration names. Preserve code formatting during ingestion and return source locations that let a user verify the procedure. Include questions with incomplete symptoms in evaluation so the system learns to request clarification rather than guess.
Local or open-source deployments
Local models and self-managed components can be appropriate when infrastructure control, offline operation, or predictable data handling is more important than minimizing maintenance. Compare the complete system, including hardware, model quality, embedding generation, indexing, upgrades, and monitoring. A model comparison without the surrounding operational work is incomplete; see the overview of open-source LLMs for local development for related trade-offs.
When to revisit
RAG architecture should be reviewed whenever its inputs or operating assumptions change. Revisit the design after a major document-format change, a new permission model, a substantial corpus expansion, or a shift in user questions. Also review it when the embedding model, vector search service, reranking option, LLM, framework, or deployment environment changes.
Set a repeatable maintenance cycle even when no obvious change occurs. Re-run the evaluation set after prompt edits, chunking changes, index rebuilds, model updates, and retrieval-parameter changes. Keep prompt and configuration versions so a regression can be traced and rolled back; the guide on prompt versioning for teams explains a practical approach.
Before expanding the system, complete these actions:
- Write down the supported questions and the questions the system should decline.
- Create a representative evaluation set with expected sources and answer requirements.
- Inspect a sample of extracted documents and chunks.
- Test semantic, keyword, metadata, and hybrid retrieval where appropriate.
- Verify citations, authorization filters, deletion behavior, and no-answer responses.
- Measure latency, token usage, failures, and user corrections in a privacy-conscious way.
- Record component versions, configuration assumptions, and the next review trigger.
Finally, keep the architecture understandable. Frameworks can accelerate experimentation, but they should not hide the retrieval query, filters, prompt, or evaluation logic. Whether you use a framework such as LangChain or a smaller custom implementation, make each stage inspectable. A RAG application that can show what it retrieved, why it retrieved it, and how the final answer was assembled is easier to improve when models, prices, features, and project requirements change. For framework-specific trade-offs, consult this LangChain tutorial for production applications.
