RAG (Retrieval-Augmented Generation).

Retrieval-Augmented Generation (RAG) fetches relevant text from an outside knowledge source at query time, then puts that text in the prompt. The model answers from the supplied evidence, not only from memory.

A trained model's knowledge is frozen at its cutoff, the date its training data stops. It has never seen your company's wiki, last week's ticket, or a private contract. Asking it anyway produces confident guesses.

RAG solves this without retraining. You update the documents, and the next answer changes. Because each answer traces back to a retrieved passage, you can show the source.

The tradeoff is that answer quality now depends on retrieval quality. If the search misses the right passage, the model is back to guessing.

A RAG pipeline has two halves. An offline indexing pass runs whenever documents change. An online query pass runs on every request.

The offline pass runs these stages in order:

• Load and parse the documents.
• Chunk them.
• Embed each chunk.
• Write the vectors, plus metadata, to the index.

The online pass runs these stages in order:

• Rewrite the query.
• Embed it.
• Retrieve the top-k chunks.
• Rerank them.
• Build the prompt.
• Generate the answer and return it with citations.

Two points beginners miss. The same embedding model must serve both halves, or the query vector and the stored vectors do not share one space. And the pipeline is a chain: if any stage before generation drops the right chunk, the model cannot recover it.

Reranking is optional. Retrieval is tuned for speed over a huge index, reranking for precision over a short list.

An embedding is a list of numbers, called a vector, that a model produces from a piece of text. Text with similar meaning gets vectors that sit close together in vector space.

RAG depends on embeddings because retrieval must match on meaning, not exact words. You embed each chunk at index time and the question at query time, then hand the nearest chunks to the LLM that writes the answer. A question about getting money back can reach the refund policy with no word in common.

q = embed("can I get my money back")      # one vector, e.g. 1536 floats
hits = index.search(q, k=5)               # chunks whose vectors are nearest

The query and the chunks must go through the same embedding model, or the numbers are not comparable. And the vector only carries what the model learned, so rare product names or IDs can be missed.

A vector database stores each chunk's embedding next to its text and metadata, and returns the chunks nearest to a query vector. Its core trick is an approximate nearest neighbor (ANN) index, which finds close matches without scanning every vector. In RAG, the chunks it returns become the LLM's context.

Comparing the query against every stored vector is a brute-force or flat scan. It is exact, and it is fine for a few thousand vectors. But the cost grows linearly with the corpus. Five million vectors at 1536 dimensions is about 7.7 billion multiply-adds per query.

An ANN index keeps the cost per query nearly flat as the corpus grows. It usually finds 95 percent or more of the true nearest neighbors. The database also handles filters, updates and persistence, which an in-memory array leaves to you. So you trade a little recall for a large speed win.

Chunking exists because an embedding is one fixed-size vector, no matter how long the input. A whole document collapses into a single point that averages all its topics, so the one relevant paragraph loses its signal.

Three forces make this concrete:

• Dilution. A 40-page manual covering billing, refunds, and shipping matches a refund question only weakly. Most of its text is about something else.
• Model limits. Embedding models cap input tokens. Text past the cap is truncated, so it never gets indexed at all.
• Prompt budget. What you retrieve is what you paste into the prompt. Whole documents burn tokens and bury the answer among unrelated text.

So retrieve the smallest unit that still stands on its own.

Chunk overlap means neighboring chunks share some text at their edges. A 500-token chunk with 50-token overlap repeats the last 50 tokens of one chunk as the start of the next.

The reason is boundary loss. A hard cut can split a definition from its example, or leave a pronoun with nothing to point at. Overlap gives each idea a second chance to sit whole inside some chunk.

step = chunk_size - overlap
chunks = [tokens[i:i + chunk_size] for i in range(0, len(tokens), step)]

Typical overlap is 10 to 20 percent of chunk size. The cost is duplication: the index grows, and retrieval can return near-copies, so dedupe neighboring chunks before building the prompt.