Retrieval-Augmented Generation (RAG) finds relevant information from an external collection and gives it to a language model before the model answers. It connects generated text to sources that the model did not memorize during training.

A typical system searches company documents, product manuals, or other approved data for the user's question. It places the best passages in the prompt and asks the model to answer from them.

RAG helps with private, detailed, and changing knowledge. It can also support citations and easier updates. It does not guarantee truth, because retrieval can miss the right passage and the model can still misunderstand or ignore good evidence.

RAG supplies information that model weights may not contain, such as internal policies, customer records, or recent documentation. The team can update the knowledge source without retraining or waiting for a new model.

Retrieved passages provide evidence that can be cited and inspected. Access filters can also limit which documents each user may see. These controls are difficult when facts exist only inside a model's weights.

Built-in knowledge is still useful for general language and reasoning. Use RAG when the answer depends on private, current, or source-specific facts. Evaluate retrieval carefully, since adding irrelevant documents can make the final answer worse.

A RAG pipeline has an ingestion path and a query path. During ingestion, it parses documents, splits them into useful chunks, creates embeddings, and stores text, vectors, metadata, and access rules.

During a query, it:

1. Rewrites or expands the user's question when needed.
2. Retrieves candidate chunks with vector, keyword, or hybrid search.
3. Filters and reranks the candidates.
4. Builds a prompt from the best evidence.
5. Generates an answer with source references.

The application then validates citations, logs the stages, and collects feedback from real users. Each stage needs separate evaluation so a retrieval miss is not mistaken for a generation failure.

An embedding is a numeric vector that represents the meaning of text. Similar ideas tend to produce nearby vectors, even when they use different words.

For RAG, the system embeds document chunks during ingestion and stores their vectors. It embeds the user's question with the same model, compares that query vector with stored vectors, and returns nearby chunks. This enables semantic search across vocabulary differences.

Embeddings are central but not sufficient. They can miss exact names, numbers, or rare terms, so keyword search and reranking often help. The embedding model, chunking method, metadata, and evaluation data all affect retrieval quality.

A vector database stores embeddings with their source text and metadata, then supports fast similarity search. It also provides indexing, filtering, updates, deletion, replication, and operational controls.

Comparing a query with every vector gives exact nearest neighbors, but the work grows linearly with the collection. That can be too slow and expensive for millions of vectors or high query traffic.

Vector databases use approximate nearest neighbor indexes to examine a much smaller candidate set. This trades a little recall for large speed gains. Exact search is still reasonable for small collections, offline evaluation, or a final reranking step after filtering.

Whole documents often contain several topics, so one embedding blurs their meaning. Retrieving the entire document also wastes context tokens and can hide the exact passage needed for an answer.

Chunking creates smaller units that are easier to match and fit into a prompt. Each chunk should carry metadata that connects it to the source, section, page, and access policy.

Chunks that are too small lose context; chunks that are too large reduce retrieval precision. Start with natural sections or paragraphs, then tune size and overlap on real questions. Preserve a way to expand around a winning chunk when neighboring text changes its meaning.

Chunk overlap repeats some text at the boundary between neighboring chunks. It reduces the chance that a sentence, definition, or important relationship is split so neither chunk contains enough context.

Overlap can improve retrieval, but it increases embedding cost, storage, and duplicate results. Too much overlap may fill the final prompt with repeated evidence and reduce source diversity.

Use the smallest overlap that fixes measured boundary failures. Structural chunking may need less overlap because headings and paragraphs already create meaningful boundaries. Deduplicate or merge adjacent retrieved chunks before generation, and evaluate overlap with real queries rather than selecting a percentage by habit.