Retrieval-Augmented Generation (RAG) is a way to build AI applications that answer questions using information from a collection of documents. Instead of relying only on what a language model learned during training, a RAG system searches a knowledge source at question time and supplies relevant passages to the model. The model then uses those passages to prepare an answer.
RAG is useful for internal knowledge assistants, product documentation search, support tools, policy lookup, research workflows and learning applications. It is not a guarantee of correctness: the quality of the answer depends on the quality of the documents, retrieval process, access controls and evaluation.
1. Why use RAG?
A language model's training data may be incomplete, outdated or unrelated to your private documents. Fine-tuning can change model behaviour, but it is not usually the best way to keep a frequently changing knowledge base current. RAG addresses a different problem: it retrieves external evidence for each request.
For example, imagine a company has hundreds of product manuals. A user asks, “Does the mobile app support offline exports?” The application searches the manuals for relevant passages, provides the strongest matches to the model, and asks it to answer using those sources. If no manual mentions offline exports, the system should say that the available documentation does not establish an answer.
RAG compared with a normal prompt
A normal prompt includes the question and whatever context a developer manually adds. RAG adds a search stage that selects context dynamically. This is especially useful when the collection is too large to fit into every request.
2. The main stages of a RAG pipeline
A production pipeline usually has an ingestion path and a question-answering path.
Stage A: Ingest and prepare documents
Collect permitted source files, extract text, preserve headings and metadata, and normalize the content. Keep useful information such as document title, URL, owner, version, publication date and access permissions.
Do not discard source identity. A retrieved passage should be traceable to the document and section from which it came. If the source is updated or deleted, your index needs a way to reflect that change.
Stage B: Split documents into chunks
A chunk is a smaller section of a document that can be embedded and retrieved independently. Very large chunks may contain several unrelated topics. Very small chunks may lose the context needed to interpret a sentence.
A practical starting point is to split around headings or paragraphs, then apply a token-based size limit. A small overlap between neighbouring chunks can preserve context across boundaries, but excessive overlap creates duplicate results and wastes storage.
There is no universal best chunk size. Test it with the questions users actually ask. A code reference, a legal policy, and a tutorial may need different chunking strategies.
Stage C: Create embeddings
An embedding model converts a piece of text into a numerical vector. Texts with related meaning can have vectors that are close under a chosen similarity measure. The application stores each vector alongside the original text and metadata.
Embeddings support semantic retrieval: a question can find a passage even when it uses different wording. They do not understand the document in the same way a human does, and similarity alone does not prove that a passage answers the question.
Stage D: Retrieve relevant passages
At question time, the application embeds the query and searches the vector index. It may also use keyword search. Hybrid retrieval can help when exact product names, error codes, identifiers or technical terms matter.
The initial search may return more passages than the model needs. A reranker can score the candidates against the question and move the most relevant passages to the top. Keep the final context concise enough to leave room for instructions and the answer.
Stage E: Generate an evidence-based answer
The application sends the question, selected passages and clear instructions to the model. Tell it to distinguish evidence from inference, cite the supplied source identifiers, and say when the sources do not contain the answer.
The model should not be allowed to treat instructions embedded inside retrieved documents as trusted application instructions. Retrieved text is data. This is an important prompt-injection boundary.
3. A simple architecture
A small system can be represented as:
- Ingestion: read documents and extract text.
- Preparation: clean, split and attach metadata.
- Indexing: create embeddings and store vectors.
- Query: accept a user's question.
- Retrieval: find candidate passages using semantic and/or keyword search.
- Reranking: prioritize the most useful candidates.
- Generation: ask the language model to answer from the selected evidence.
- Validation: check citations, permissions, format and answer quality.
- Monitoring: record retrieval quality, latency, cost and failure patterns.
Keep ingestion separate from answering. This makes it possible to update the index when documents change without rebuilding the whole application for every question.
4. Build a small course-notes assistant
Start with five to ten documents that you are allowed to use. Choose a narrow subject, such as a course module or a product's public documentation.
Step 1: Prepare the source set
Give each document a stable identifier. Extract text and preserve headings. Remove repeated navigation text and boilerplate, but do not remove warnings, dates or qualifications that change the meaning.
Step 2: Create chunks
Split by section first and then by a token budget. Store metadata with every chunk:
- document ID and title
- section heading
- source URL or file name
- last updated date
- access-control label
- chunk sequence number
Step 3: Index and retrieve
Create embeddings for each chunk and store them in a vector index. When a question arrives, retrieve a small candidate set. Inspect the results manually before connecting a language model. If the right source is not retrieved, generation cannot reliably fix that failure.
Step 4: Add grounded generation
Pass the retrieved passages with explicit source labels. Ask for a concise answer and citations that refer to those labels. If the evidence is missing or contradictory, require the assistant to explain the limitation rather than invent a resolution.
Step 5: Test with a fixed question set
Create questions whose answers are present, questions that require combining two passages, questions with ambiguous wording, and questions that are not answered by the documents. Check both the retrieved evidence and the final response.
5. How to evaluate RAG properly
RAG has at least two separate quality problems: retrieval and generation.
Retrieval evaluation asks whether the system found the right evidence. Useful measures include recall at K (whether a relevant passage appears in the first K results), precision at K (how many of those results are relevant), and ranking quality.
Answer evaluation asks whether the generated response is supported by the retrieved material, complete enough for the question, readable, and appropriately cautious. Check whether citations actually support the statements beside them.
Track additional operational measures: response latency, embedding and model cost, index freshness, failure rate, and the number of requests where no useful source was found. Review real failure examples; a single average score can hide important problems.
6. Common RAG problems and fixes
| Problem | Likely cause | What to try |
|---|---|---|
| Correct document is never retrieved | Weak chunking, metadata or query match | Inspect chunks; add keyword or hybrid search |
| Relevant result is ranked too low | Similar passages dominate | Test a reranker and tune candidate count |
| Answer includes unsupported details | Prompt allows speculation or context is weak | Require evidence and explicit uncertainty |
| Answer misses a key qualification | Chunk split removed surrounding context | Use heading-aware chunks and limited overlap |
| Results contain outdated facts | Index was not refreshed | Add update and deletion workflows |
| Private content appears to the wrong user | Permissions were not enforced during retrieval | Filter by access rights before generation |
| Costs rise unexpectedly | Too many large chunks are sent | Rerank, deduplicate and limit final context |
7. Security and privacy considerations
RAG does not remove the need for application security. Apply document permissions before a passage reaches the model. Do not retrieve every document and rely on the model to decide which user may see it.
Treat source documents as untrusted input. A document could contain text that attempts to manipulate the assistant or request secret data. Keep system instructions separate, restrict tools, validate actions in application code, and never put credentials in retrieved content.
Consider retention, deletion, data residency and provider policies before indexing sensitive information. When a source document is removed or a user's access changes, update the index and caches accordingly.
8. When RAG is not the right answer
RAG is not required for every AI feature. If the task is a small transformation that needs no external knowledge, a normal prompt may be simpler. If the task requires exact database records or transactional updates, a structured API or database query may be more appropriate than semantic search.
If the desired behaviour is a consistent style or a specialized task pattern, prompt design or fine-tuning may be relevant. RAG and fine-tuning solve different problems and can also be combined.
Frequently asked questions
Does RAG eliminate hallucinations?
No. It can provide relevant evidence and make answers easier to verify, but retrieval can fail and models can misinterpret sources. Evaluate grounding and provide a safe “not found” behaviour.
Do I need a vector database?
Not always. A small prototype can use a local index or a search engine with vector support. Choose infrastructure based on data size, filtering, operational needs and team experience.
Should I use only vector search?
Not necessarily. Keyword search is valuable for exact names, IDs and error messages. Hybrid retrieval is often worth testing.
How often should the index be updated?
Update it whenever source freshness matters. For frequently changing documents, use an incremental pipeline that handles additions, edits and deletions.
Final checklist
- Use documents you are authorized to index.
- Preserve source identifiers, dates and permissions.
- Test chunking with realistic questions.
- Inspect retrieval results before tuning generation.
- Keep the final context focused and within the model's token budget.
- Require grounded answers and traceable citations.
- Evaluate retrieval and generation separately.
- Monitor freshness, access control, cost and latency.
The best RAG system is not the one with the most complicated vector database. It is the smallest system that reliably finds the right evidence, respects permissions, and clearly communicates what the sources do and do not say.