This RAG tutorial explains how to plan, estimate, and improve a production-ready retrieval-augmented generation app. You will learn how to model ingestion and storage costs, measure retrieval and answer quality, tune chunking and search, add citations and guardrails, and decide when your assumptions need to be recalculated.
Overview
Retrieval-augmented generation, or RAG, connects a language model to a collection of external documents. Instead of asking the model to rely only on its trained knowledge, the application retrieves relevant passages at request time and places them in the model’s context. The model then uses those passages to produce an answer.
A useful RAG system has several distinct stages:
- Ingestion: collect documents, extract text, preserve metadata, and identify access rules.
- Chunking: divide documents into searchable passages without separating important context.
- Embedding: convert chunks and user queries into numerical representations.
- Indexing: store embeddings and metadata in a vector search system or a hybrid search index.
- Retrieval: select candidate passages for a query, optionally using keyword search, filters, reranking, or all three.
- Generation: provide the selected context to an LLM with instructions for answering, citing, and handling uncertainty.
- Evaluation: test retrieval and answer quality against a representative question set.
The central planning mistake is treating RAG as one model call. Its behavior and cost are influenced by document volume, update frequency, query traffic, token counts, search configuration, model selection, latency targets, and operational requirements. Separating these variables makes the system easier to estimate and improve.
For a broader implementation workflow, see the RAG application tutorial on building a reliable retrieval workflow. If your project is an internal knowledge base, permissions and auditability should be designed alongside retrieval rather than added after launch.
How to estimate
Start with a simple monthly model. Use variables first, then replace them with values from your own application or provider pricing pages.
One-time ingestion work can be estimated as:
Ingestion workload = documents × average pages per document × processing time per page
This is not necessarily a direct bill. It helps estimate engineering and infrastructure effort. Add separate stages for text extraction, cleaning, chunking, embedding, indexing, and validation. Reprocessing the entire corpus whenever one document changes is usually a different workload from incremental updates, so track both.
Embedding workload can be represented as:
Embedding volume = total indexed tokens ÷ tokens per embedding request
In practice, providers may price embeddings by input volume, request count, or another unit. Record the provider’s current unit and use the formula that matches it. Keep a margin for metadata, repeated processing, failed requests, and documents that are updated more frequently than expected.
Query workload starts with:
Monthly queries = active users × queries per user per period
Then estimate the model input for each request:
Input tokens per query = system instructions + user question + retrieved context + conversation history
Generation tokens should be estimated separately. A concise answer with citations generally has a different output profile from a long report. If the application supports follow-up questions, include the portion of conversation history that is actually sent to the model, not the entire chat by default.
A basic recurring cost model is:
Total monthly cost = model input cost + model output cost + embedding cost + search cost + storage cost + observability cost + retry overhead
Use a low, expected, and high scenario for each major input. For example, query volume may vary more than document storage, while retrieved-context size may change after a prompt or chunking adjustment. This range is more useful than a single precise-looking estimate.
Quality also needs a repeatable estimate. Create a test set containing real or carefully anonymized questions and label each question with the expected source document, relevant passage, answer requirements, and whether the correct response should be “not enough information.” Run the same set after every meaningful change.
Inputs and assumptions
Document assumptions before selecting a framework or vector database. The following inputs have the greatest effect on architecture and results.
Corpus shape and freshness
Record document types, average length, language, tables, images, duplicate content, and update frequency. A small set of clean text files may need only basic indexing. Scanned PDFs, changing product catalogs, or policy documents with tables require more careful extraction and validation.
Chunking and overlap
Chunk size should be large enough to preserve a useful idea but small enough to avoid filling the context with unrelated material. Test several sizes rather than assuming one default is best. Overlap can help preserve continuity, but excessive overlap increases index size and may return near-duplicate passages. Store document ID, title, section, page or location, version, timestamp, and permission metadata with every chunk.
Retrieval configuration
Define how many candidates are retrieved, whether metadata filters are mandatory, and whether keyword search or reranking is needed. A larger candidate set can improve recall but may increase latency and context length. Measure retrieval separately from generation so a strong answer does not hide weak search performance.
Prompt and answer policy
Your generation prompt should tell the model what the context represents, how to cite it, what to do when evidence is missing, and which instructions have priority. Avoid telling the model to answer every question. A reliable fallback is to state that the available sources do not establish an answer and suggest what information is needed.
For practical guidance on reducing unsupported answers, read how to reduce hallucinations in RAG systems without overconstraining responses. Prompt changes should be versioned and evaluated like code; a small wording change can affect refusal behavior, citation style, and token usage.
Access control and observability
Permissions must be applied during retrieval, not only in the user interface. Filter results using the user’s authorization context before passages reach the model. Log request IDs, retrieved document IDs, model and prompt versions, latency by stage, token counts, errors, and user feedback. Avoid storing sensitive content in logs unless the system has an explicit retention and access policy.
Worked examples
Example 1: Estimating a document refresh
Assume a team has a corpus of D documents, each producing C chunks, and expects U updates during a refresh period. The approximate number of new embedding inputs is:
New embedding inputs = D × C × U
If updates are incremental, replace D with the number of changed documents. If the pipeline reprocesses the entire corpus, use the full document count. Compare both estimates before choosing a refresh strategy. Incremental processing may reduce work, but it requires reliable document identity, version tracking, deletion handling, and protection against stale chunks.
Example 2: Estimating request cost and latency
Suppose an application receives Q monthly questions. Each request sends S system and instruction tokens, H history tokens, and R retrieved tokens, then generates G output tokens. Total monthly token volume is approximately:
Input tokens = Q × (S + H + R)Output tokens = Q × G
Apply the selected provider’s current input and output rates only after these volumes are known. If retrieval returns too much context, reducing R may lower cost and latency, but it can also remove evidence. Test a smaller candidate set, better metadata filters, improved chunking, or reranking before simply truncating context.
Example 3: Evaluating retrieval before answer quality
Create questions that represent easy lookups, cross-section questions, ambiguous wording, outdated documents, and unanswerable requests. For each question, inspect whether the retrieved passages contain the evidence needed for a correct response. Then score the generated answer for factual support, completeness, citation accuracy, refusal behavior, and format compliance.
This two-stage review identifies the source of failure. If the evidence is absent from the retrieved set, revise chunking, indexing, filters, or query rewriting. If the evidence is present but the answer is wrong, revise the prompt, output schema, model choice, or context ordering. A prompt evaluation framework can help teams turn these checks into a repeatable regression process.
When to recalculate
Revisit your RAG estimates whenever an input that affects workload, quality, or risk changes. At minimum, recalculate after:
- Changing the embedding model, generation model, or provider pricing unit.
- Adding a large document collection or a new document type.
- Changing chunk size, overlap, metadata filters, top-k retrieval, reranking, or context limits.
- Increasing conversation history or introducing multi-step retrieval.
- Changing expected users, query volume, answer length, or peak traffic.
- Adding citations, structured output, tool calls, retries, or fallback models.
- Changing retention, logging, permission, or audit requirements.
- Observing a meaningful shift in latency, token usage, retrieval recall, citation accuracy, or user feedback.
Keep a small assumptions table in the project repository with the date, corpus size, average chunks per document, query volume, token estimates, retrieval settings, model versions, and evaluation results. When the system changes, update the table and rerun the test set rather than relying on memory.
The most useful production RAG workflow is not the one with the most components. It is the one whose costs, evidence quality, permissions, and failure modes are visible. Start with a narrow corpus and a representative evaluation set, measure retrieval separately from generation, and expand only when the results justify it. Recalculate after every material change so your architecture remains aligned with both user needs and current operating assumptions.
